Summary of Statistical Learning (Machine Learning) Course for Econometrics Students at UvA taught by Yi HE
6 views 0 purchase
Course
6013B0357Y
Institution
Universiteit Van Amsterdam (UvA)
Book
An Introduction to Statistical Learning
Dive into the world of Statistical Learning with this comprehensive guide, meticulously crafted to bridge the gap between theoretical concepts and practical applications in data science and machine learning. Whether you're a student eager to master the fundamentals, a practitioner aiming to sharpen...
Statistical Learning
Made by Maxmillian Forman, please do not sell this document. Good Luck!
1. Bias-Variance Trade-off
1.1. Regression Function
Predictor Function Notation: is a predictor function, a prediction rule, it is an element of a set of all
possible predictor functions, denoted . We can subset with , hence .
Squared Loss Function: Given an independent test observation and a independent training sample
for a candidate prediction rule . The squared loss function is given by:
Population Risk. Also known as the expected loss, for a predictor function that is an element of all
possible predictor functions.
Regression Function: By minimizing the population risk we obtain the regression function which is
unknown in practice.
Statistical learning aims to estimate the fixed but unknown
Prediction Model: The regression function showcases the systematic information provides about
.
1.2. Empirical Risk
Empirical Risk Function: For a random sample and and sample prediction model
the empirical risk function for a candidate prediction rule is:
The empirical risk function is an estimator of the population risk .
A estimator for would be the minimum of the empirical risk.
Approximation Error: The approximation error of the risk function is:
For we want to find the smallest approximation error for , so we look at the greatest lower bound
of the approximation error.
A flexible model has a large set .
A large set corresponds to high variance of
, A high variance of results in high approximation error.
Overfitting: Refers to fitting to the training set too much, causing to predict empirical
risk and not the population risk. Occurring when a large set is present.
Overfitting occurs if we have a low training MSE and a high test MSE.
1.3. Regularization
Regularization: Refers to restricting all predictor functions to a smaller set , to control the bias
variance trade-off.
A large sample size corresponds to a low variance.
For :
As we increase :
Flexible model, risk of overfitting
Flexibility increases.
The variance of increases
The bias of decreases
As we decrease :
Inflexible model, risk of underfitting.
Flexibility decreases.
The variance of decreases
The bias of increases.
Bias: Bias refers to how far is from the true regression function .
Variance: Variance refers to the extent of fluctuations in .
Regularized Estimators: There are two estimators when we minimize on a smaller domain , regularized
and oracle estimators.
The regularized estimator estimates the oracle estimator, we are looking for the best prediction
rule in the subdomain .
1. Regularized Estimator:
Depends on the empirical risk function.
2. Oracle Estimator
Depends on the population risk function.
Population Mean Squared Error:
Sample Mean Squared Error:
Reducible and Irreducible Errors: Suppose is a possibly regularized estimate of , fitted using a
training set. For and from the test set we can rewrite the population MSE into a reducible error and
a irreducible error.
Bias-Variance Trade-off: We use the law of iterated expectations to decompose the reducible error into
the bias and variance of .
, Bias Variance for an Estimator: For an estimator of some parameter we can decompose the
mean squared error as:
1.4. Cross Validation
Hyper Parameter Optimization: We want to estimate the reducible error too find the optimal
subdomain for the estimator . So we introduce a hyperparameter where
which assists in finding the appropriate .
Validation Set Approach: A method to estimate . Outputs the validation set error rate which is an
estimate for test set error rate.
Advantages:
Relatively simple to understand
Disadvantages:
Validation set error rate, MSE, is variable due to random split.
Validation set error may overestimate test set error.
We use a subset of the initial dataset to train , so we loose some information.
1. Randomly splits the dataset into a training and validation dataset.
2. Trains on the training dataset.
3. The fitted estimator then predicts responses for the validation dataset.
4. We calculate the validation set error, MSE, which will serve as an approximation for test error
rate.
K-Fold Cross Validation: A method used to estimate . Outputs the validation set error rate which is an
estimate for test set error rate.
Advantages:
MSE variability is smaller then in validation set approach and LOOCV.
Disadvantages:
Has higher bias then LOOCV.
1. We divide the initial dataset into equally sized partitions.
2. We select partitions to be the training set and the remaining -th partition will be the
validation set.
3. We fit using the training set.
4. The fitted estimator then predicts responses for the -th validation dataset.
5. We calculate the validation set error and call it .
6. We repeat so for all partitions exclusively and compute the average
Leave One Out Cross Validation (LOOCV): A method to estimate . Outputs the validation set error rate
which is an estimate for test set error rate.
Advantages:
We use all the observations in the dataset, so LOOCV is less biased and does not
overestimate the test error rate.
Disadvantages:
Computationally Intensive
Has higher variance then -fold, but a lower bias.
1. We divide the dataset into a training and validation set where the validation set is of size
.
The benefits of buying summaries with Stuvia:
Guaranteed quality through customer reviews
Stuvia customers have reviewed more than 700,000 summaries. This how you know that you are buying the best documents.
Quick and easy check-out
You can quickly pay through credit card or Stuvia-credit for the summaries. There is no membership needed.
Focus on what matters
Your fellow students write the study notes themselves, which is why the documents are always reliable and up-to-date. This ensures you quickly get to the core!
Frequently asked questions
What do I get when I buy this document?
You get a PDF, available immediately after your purchase. The purchased document is accessible anytime, anywhere and indefinitely through your profile.
Satisfaction guarantee: how does it work?
Our satisfaction guarantee ensures that you always find a study document that suits you well. You fill out a form, and our customer service team takes care of the rest.
Who am I buying these notes from?
Stuvia is a marketplace, so you are not buying this document from us, but from seller formanmaximilian. Stuvia facilitates payment to the seller.
Will I be stuck with a subscription?
No, you only buy these notes for $27.42. You're not tied to anything after your purchase.
