Description

1 Definitions 1. A function g is said to be convex if
g(λx1 + (1 − λ) x2) ≤ λg(x1) + (1 − λ)g(x2)
2. g is convex if and only if (1.1.1)
g′′(x) ≥ 0
3. The information associated with an event E is defined as (1.2.1)
I (E) = −log2 (p(E))
4. The entropy of a random variable X is given by (1.3.1)
H (X) = −E log2 (p(X))
5. For a bernoulli random variable Y ∈ {0,1}, the p.m.f is given by (1.4.1)

 p k = 0 pY(k) =
1 − p k = 1
2 Problems 1. Show that log 1x is convex.
2. Show that (1.5.1)
H(Y) = −plog2 p − (1 − p)log2 (1 − p)
and find the maximum value of H(Y).
3. Let X ∈ {x1, x2} and (2.2.1)
λ = q = pX(X = x1)
in (1.1.1). Show that (2.3.1)
E g . (2.3.2)
4. Using (2.3.2), show that
H(Y) ≤ 1 (2.4.1)