Description

For example, consider x ∈ Rn. Further, consider W ∈ Rm×n where m < n. Then Wx is of lower dimensionality than x. One way to design W so that Wx still contains key features of x is to minimize the following expression
Wx
with respect to W. (To be complete, autoencoders also have a nonlinearity in each layer, i.e., the loss is (Wx)) . However, we’ll work with the linear example.)
(a) (3 points) In words, describe why this minimization finds a W that ought to preserve information about x.
(b) (3 points) Draw the computational graph for L.
(c) (3 points) In the computational graph, there should be two paths to W. How do we account for these two paths when calculating ∇WL? Your answer should include a mathematical argument.
(d) (6 points) Calculate the gradient: ∇WL.
2. (20 points) Backpropagation for Gaussian-process latent variable model. An important component of unsupervised learning is visualizing high-dimensional data in lowdimensional spaces. One such nonlinear algorithm to do so is from Lawrence, NIPS 2004, called GP-LVM. GP-LVM optimizes the maximum-likelihood of a probabilistic model. We won’t get into the details here, but rather to the bottom line: in this paper, a log-likelihood has to be differentiated with respect to a matrix to derive the optimal parameters.
To do so, we will use apply the chain rule for multivariate derivatives via backpropagation. The log-likelihood is:
tr(K−1YYT)
where K = αXXT + β−1I and c is a constant. To solve this, we’ll take the derivatives with respect to the two terms with dependencies on X:

YYT
1
∂L −T ∂L −T
∂ K = −K ∂ K−1K
(a) (3 points) Draw a computational graph for L1.
(b) (6 points) Compute .
(c) (3 points) Draw a computational graph for L2.
(d) (6 points) Compute .
(e) (2 points) Compute .
3. (40 points) 2-layer neural network. Compete the two-layer neural network Jupyter notebook. Print out the entire workbook and relevant code and submit it as a pdf to gradescope. Download the CIFAR-10 dataset, as you did in HW #2.
4. (25 points) General FC neural network. Compete the FC Net Jupyter notebook. Print out the entire workbook and relevant code and submit it as a pdf to gradescope.
2