Description

1. [2 points] KL Divergence
(a) [1 point] What is the expression of the KL divergence DKL(q(x)||p(x)) given two continuous distributions p(x) and q(x) defined on the domain of R1?

(b) [1 point] Show that the KL divergence is non-negative. You can use Jensen’s inequality here without proving it.

2. [3 points] In the class, we derive the following equality:

Instead of maximizing the log likelihood logpθ(x) w.r.t. θ, we find a lower bound for logpθ(x) and maximize the lower bound.
(a) [1 point] Use the above equation and your result in 1(b) to give a lower bound for logpθ(x).

(b) [1 point] What do people usually call the bound?
Your answer: L(pθ,qφ) is often referred to as empirical lower bound (ELBO).
(c) [1 point] In what condition will the bound be tight?
Your answer: It holds with equality if and only if qφ = pθ for all x.
3. [2 points] Given z ∈ R1, p(z) ∼ N(0,1) and q(z|x) ∼ N(µz,σz2), write DKL(q(z|x)||p(z)) in terms of σz and µz.

4. [1 points] In VAEs, the encoder computes the mean µz and the variance ) assuming qφ(z|x) is Gaussian. Explain why we usually model σz2 in log space, i.e., modeling logσz2 instead of σz2 when implementing it using neural nets?
Your answer:
It is more numerically stable to take exponent compared to computing log.

Furthermore, if we model just σz2, it always outputs a non-negative number, but if we model σz2 in log space, we can get all the number in R1, which will give us more options.
5. [1 points] Why do we need the reparameterization trick when training VAEs instead of directly sampling from the latent distribution )?
Your answer: In a nutshell, reparametrization trick make sure we can backpropagate. We are given z that is drawn from a distribution qφ(z|x), and we want to take derivatives of a function of z with respect to φ, The reparametrization trick lets us backpropagate (take derivatives using the chain rule) with respect to φ through the objective (the ELBO) which is a function of samples of the latent variables z.
2