Description

0 0 3 0 22.0 1 0 7.2500
1 1 1 1 38.0 1 0 71.2833
2 1 3 1 26.0 0 0 7.9250
3 1 1 1 35.0 1 0 53.1000
4 0 3 0 35.0 0 0 8.0500

Out[5]:
Survived Pclass Sex Age Siblings/Spouses Aboard Parents/Children Aboard Fare

0 0 3 0 22.0 1 0 7.2500
1 1 1 1 38.0 1 0 71.2833
2 1 3 1 26.0 0 0 7.9250
3 1 1 1 35.0 1 0 53.1000
4 0 3 0 35.0 0 0 8.0500
3.1
Consider log-likelihood function:
ℓ(𝜽) =

Out[10]: Pclass -0.157296 Sex 0.691840
Age -0.022838
Siblings/Spouses Aboard -0.128911
Parents/Children Aboard -0.040576 Fare 0.013129 dtype: float64
(a)
I choose constant stepsize as 10−7, the optimization process went pretty well. I’ve chosen some other stategies at the beginning, such as exact lone search/ learning rate decay҅ momentum method, or some other methods involving second gradient. The method involving learn search didn’t get good result and usually requires more iterations. The likelihood will fluctuate in some iterations. The method methods involving second gradient cost more time and computation complexity.
In the end, the constant stepsize showed the desired result.
(b)
In [11]: import time start_time = time.time() theta, vec = GD(20000,1e-7, 1e-10) print(“Total %s seconds elapsed” % (time.time() – start_time))
Total 17155 iteration
Total 28.299755096435547 seconds elapsed
It took 28.30 seconds to complete the iteration. It took about 3 seconds to converge according to the loglikelihood plot.
(c)
In [12]: theta
Out[12]: Pclass -0.157296 Sex 0.691840
Age -0.022838
Siblings/Spouses Aboard -0.128911
Parents/Children Aboard -0.040576 Fare 0.013129 dtype: float64
(d)
In [13]: vec[-1]
Out[13]: -463.53153422224244
(e)
Using 𝜃⋆ to denote ture parameter.
Consider Fisher information:
𝐈𝜽⋆ := −𝔼[ d2dℓ𝜽(2𝜽)∣∣∣∣𝜽=𝜽⋆] = 𝔼⎢⎢⎣⎡∑𝐢=N1 (1 +𝑒−𝑒𝜽−⊤𝜽𝐱⊤𝐢 𝐱𝐢 )2 𝐱𝐢𝐱𝐢⊤∣∣∣∣∣𝜽=𝜽⋆⎤⎥⎥⎦ = ∑𝐢=N1 (1 +𝑒−𝑒𝜽−⋆𝜽⊤⋆𝐱⊤𝐢 𝐱𝐢 )2 𝐱𝐢𝐱𝐢⊤
Let
𝜽̂ := arg maxℓ(𝜽)
𝜽∈ℝD+1
Suppose d2ℓ(2𝜽) . Then we have: d𝜽
𝜽̂ ⟶𝑑 (𝜽⋆, 𝐈𝜽−⋆1)
According to computation, we have:

Then we have:
𝜽̂ ⟶𝑑  ̂
where 𝐈−𝜽̂1 is
In [16]: var
Out[16]: array([[ 3.21359720e-03, -1.96270908e-03, -1.94319179e-04,
-1.48733570e-03, -1.01101582e-03, 5.15191289e-05],
[-1.96270908e-03, 2.37288881e-02, -3.72060099e-05,
1.08457571e-04, -2.95630615e-03, -7.20737802e-05], [-1.94319179e-04, -3.72060099e-05, 2.19491616e-05,
1.01031536e-04, 6.98258066e-05, -7.11340387e-06], [-1.48733570e-03, 1.08457571e-04, 1.01031536e-04,
5.81895119e-03, -2.23402533e-03, -6.34861528e-05], [-1.01101582e-03, -2.95630615e-03, 6.98258066e-05,
-2.23402533e-03, 1.05884034e-02, -5.48940130e-05],
[ 5.15191289e-05, -7.20737802e-05, -7.11340387e-06,
-6.34861528e-05, -5.48940130e-05, 6.73377556e-06]])
In [ ]:
3.2
Consider the the log-odds are defined as
𝜔⋆ : 𝜽⋆⊤𝐱
By the invariance property of the MLE, we know that the MLE of 𝜔⋆ is given by
𝜔̂ := 𝜽̂⊤𝐱
We further have:
𝜔̂ ⟶𝑑 (𝜔⋆, 𝐱⊤𝐈𝜽−⋆1𝐱)
In [ ]:
3.3
My own feature would be Plass:3, Gender:0, Age:23, Siblings:0, Parents/Children:0, Fare: 7.25.

Out[18]: 0.7113562614023329
(a)
since 0.66>0.5. According to (6.7) in lecture notes, I will survive the Titanic sinking.
(b)
According to:
ℙ(∣∣𝜔̂ − 𝜔⋆∣∣ > 𝜏) = 2Φ(𝜏 ∣ 0, 𝐱⊤𝐈𝜽−⋆1𝐱)

Out[19]: 0.013940140637597352
𝜏 = Φ−1 (𝛼/2 ∣ 0, 𝐱⊤𝐈𝜽−⋆1𝐱)
When 𝛼 = 0.05, the 𝜏 I compute based on my estimated variance is 0.2245956.

Out[20]: 0.4867606614023329

Out[21]: 0.9359518614023329
Then the 95% confidence interval would be [0.487,0.936]
(c)
I think my answer from (a) is fairly certain. In (a) I compute estimated probablity larger than 1/2. In (c) I got my 95% confidence interval. Most of my interval lies in [0.5,1], so I think the interval I got coincides with the answer I got from (a).
In [ ]:
3.4
(a)
Consider LRT test:
̂
̂
̂
𝜏
The quantile I calulated for chi-square distribution with df 1 is 3.841459.

Out[22]: Pclass True Sex True
Age True
Siblings/Spouses Aboard True
Parents/Children Aboard True Fare True dtype: bool
(b)
According the result in (a), all the features are significant.
(c)
Yes, the survival prediction is based on the feature prediction. If I change most significant feature in my feature vector, my survival prediction will change.
In [ ]:
In [ ]: