Description

• Submit hard copy of your homework before class:
– Your answers to the problem set
– Please also attach the code
• The abbreviation ISL refers to the book http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh% 20Printing.pdf. The abbreviation ESL refers to the book https://web.stanford.edu/~hastie/ ElemStatLearn/printings/ESLII_print12.pdf.
1. Theoretical Problems
1. Methods between Ridge and LASSO. Consider the elstic-net optimization problem
. (1.1)
Show how one can turn this into a lasso problem, using an augmented version of X and y. (This is Ex. 3.30 in ESL.)
2. Closed forms of LASSO, Ridge and Best Subset Selection. Consider a special case of linear regression

where the columns of X are orthonormal, that is XTX = I. Let denote the least square estimators and βˆ(M) denote the M-th largest absolute value of βˆ, that is . Show the corresponding formulas for Best subset with size M, Ridge and Lasso estimator,
Estimator Formula
Best subset (size M) βˆj · I(|βˆj| ≥ |βˆ(M)|)
Ridge βˆj/(1 + λ)
Lasso sign(βˆj)(|βˆj| − λ)+
1
2. Applied Problems
3. Bootstrap with Least squares, Ridge and Lasso. Let β = (β1,β2,··· ,βp) and let x,y be random variables such that the entries of x are i.i.d. standard normal random variables (i.e., with mean zero and variance
one) and where
(a) Simulate a dataset (x1,y1),…(xn,yn) as n i.i.d. copies of the random variables x,y defined above, with n = 800,p = 200 and βj = j−1.
(b) The goal of this problem is to construct confidence intervals for β1 using Bootstrap method.
(a) Construct confidence intervals for β1 by bootstrapping the data and applying Least Squares to the bootstrapped data set.
(b) Construct confidence intervals for β1 by bootstrapping the data and applying Ridge to the bootstrapped data set.
(c) Construct confidence intervals for β1 by bootstrapping the data and applying Lasso to the bootstrapped data set.
(c) Comment on the obtained results.
4. Shrinkage Method and Dimension Reduction. Problem 9 at page 263 of ISL
2