Description

Model selection Performance
Best param log2 𝜆 Mean of MSE Std of MSE MSE on train MSE on test
Least square – – –
𝒘
(show your estimated w)
𝑙1(𝑤) = 𝑙2(𝑤) = Spars=
LASSO
𝒘 (show your estimated w)
𝑙1(𝑤) = 𝑙2(𝑤) = Spars=
Ridge
𝒘 (show your estimated w)
𝑙1(𝑤) = 𝑙2(𝑤) = Spars=
Caption for statistics in the table:
• Best param 𝜆: the regularization coefficient you choose using cross validation.
• Mean of MSE: the averaged MSE of the 5-fold cross validation process for your chosen 𝜆.
• Std of MSE: the standard deviation of MSE of the 5-fold cross validation process for your chosen
𝜆.
• 𝑙1(𝑤): 𝑙1 norm of 𝒘
• 𝑙2(𝑤): 𝑙2 norm of 𝒘
• Spars: Sparsity, i.e., the number of zeros in the augmented weight vector

(b) Bob learned that 𝑙1 regularization could lead to more sparsity, and he really wants to visualize this. So he collects another bunch of datasets for 2-dimensional (before augmentation) features:
number of training samples (𝑁𝑡𝑟) number of testing samples
Dataset4 10 1000
Dataset5 30 1000
Dataset6 100 1000
Dataset7 10 1000
Dataset8 30 1000
Dataset9 100 1000
He tries them out and find that the last three datasets (7,8,9) are “special cases” where the 𝑙1 norm might not provide the intended result.
i. Repeat (a)(i) for all new datasets. (You’ll have 6 tables)
ii. For each dataset, draw the following plot in the 2D space 𝑤2 vs. 𝑤1 with 𝑤0 = your estimated 𝑤0: (1) draw the curve of ‘MSE = training_MSE of your estimated 𝒘 and
‘MSE=10+training_MSE of your estimated 𝒘; (2) draw the curve for ‖𝑤‖𝑙1 = the 𝑙1 norm of your estimated 𝒘. Repeat this plot drawing for ridge regression results, except for (2) draw the curve for ‖𝑤‖𝑙2 = the 𝑙2 norm of your estimated 𝒘. (therefore you have 2 plots for each dataset. An example is shown below.)
iii. Based on the statistics and plots, answer the following questions:
1. Observe and explain how the plots relate to sparsity.
2. Can you explain how much effect the regularizer has, from looking at the plots (i.e., how different the regularized performance (MSE) is from the unregularized performance)
3. Observe and explain how Lasso has a different effect with the “special case” datasets than the other datasets

Hint: please refer to the example code file in the homework folder on how to generate such plots.