Description

Bayesian Statistics and Data Analysis

Assignment 8

General information
The recommended tool in this course is R (with the IDE R-Studio). You can download R here and R-Studio here. There are many tutorials, videos and introductions to R and R-Studio online. You can nd some initial hints from RStudio Education pages.
When working with R, we recommend writing the report using R markdown and the provided R markdown template. The template includes the formatting instructions and how to include code and gures.
Instead of R markdown, you can use other software to make the PDF report, but you should use the same instructions for formatting. These instructions are also available in the PDF produced from the R markdown template.
We supply a Google Colab notebook that you can also use for the assignments. We have included the installation of all necessary R packages; hence, this can be an alternative to using your own local computer. You can nd the notebook here. You can also open the notebook in Colab here.
Report all results in a single and anonymous pdf. Note that no other formats are allowed.
The course has its own R package bsda with data and functionality to simplify coding. To install the package, just run the following (upgrade=”never” skips question about updating other packages):
1. install.packages(“remotes”)
2. remotes::install_github(“MansMeg/BSDA”, subdir = “rpackage”, upgrade=”never”)
Many of the exercises can be checked automatically using the R package markmyassignment. you can nd information on how to install and use the package here. There is no need to include markmyassignment results in the report.
You can nd common questions and answers regarding the installation and technical problems in Frequently Asked Questions (FAQ).
You can nd deadlines and information on how to turn in the assignments in Studium.
If you have any suggestions or improvements to the course material, please post in the course chat feedback channel, create an issue, or submit a pull request to the public repository here.
It is mandatory to include the following parts in all assignments (these are included already in the template):
1. Time used for reading: How long time took the reading assignment (in hours)
2. Time used for the assignment: How long time took the basic assignment (in hours)
3. Good with assignment: Write one-two sentences of what you liked with the assignment/what we should keep for next year.
4. Things to improve in the assignment: Write one-two sentences of what you think can be improved in the assignment. Can something be clari ed further? Did you get stuck on stu unrelated to the content of the assignment etc.
To pass (G) the assignment, you need 70% of the total points. To pass with distinction (VG), you need 90% of the total points. See the grading information on the point allocations for each assignment.
You are not allowed to show your assignments (text or code) to anyone. Only discuss the assignments with your fellow students. The student that show their assignment to anyone else could also be considered to cheat. Similarly, on zoom labs, only screen share when you are in a separate zoom room with teaching assistants.

Information on this assignment
This assignment is related to Chapter 7.
Note! This assignment build upon assignment 7, so be sure that assignment 7 is correct before you start with this assignment. You can ask TAs for help to check that your assignment 7 answer is good.
Reading instructions: Chapter 7 in BDA3, see reading instructions. The [‘loo‘ package vignette on the basics of LOO](https://mc-stan.org/loo/articles/loo2-with-rstan.html) shows an example of how to modify Stan code and use the package with Stan models. Also read about the PSIS-LOO that in this article.
Reporting accuracy: For posterior statistics of interest, only report digits for which the Monte Carlo standard error (MCSE) is zero. Example: If you estimate E(µ) = 1.234 with MCSE(E(µ)) = 0.01, you should report E(µ) = 1.2.
When computing the Rˆ diagnostics, you only need to include two decimals.
Installing and using stan: To install Stan on your laptop, https://github.com/ stan-dev/rstan/wiki/RStan-Getting-Started. If you encounter problems, see additional answers in the FAQ. Recently there have been reports of installation problems with Windows and R 4.0 (see Stan discourse for more).
Installing and using CmdStanR: If you want to use Stan in R on local computer, it can be easier to install CmdStanR interface mc-stan.org/cmdstanr/.
General information on using stan: Additional useful packages are loo, bayesplot and shinystan. For Python users, PyStan, CmdStanPy, and Arviz packages are useful.
Stan manual can be found at https://mc-stan.org/users/documentation/. From this website, you can also nd a lot of other useful material about Stan.
R-studio enables you to automatically check the Stan syntax. Just open a stan le (with le extension .stan in R-studio). Then you can use the button “Check” to check the Stan syntax.
Model assessment: LOO-CV for factory data with Stan
Use leave-one-out cross-validation (LOO-CV) to assess the predictive performance of the pooled, separate and hierarchical Gaussian models for the factory dataset (see the second exercise in Assignment 7). To read in the data, just use:
library(bsda) data(“factory”)
PSIS-LOO is a recently developed method for approximating the exact LOO and is thus not in BDA3. For more information, see the original paper here or here.
Use Stan for tting the models, and the loo R package for computing the approximate LOO-CV given the posterior samples provided by Stan. You can install the package as
install.packages(“loo”)
Python users can use PSIS-LOO implementation in ArviZ library. The report should include the following parts.
1. Fit the models with Stan as instructed in Assignment 7. With the N(100,100) prior for the µ and θ parameters and the N(0,50) prior σ and τ parameters. To use the loo or psisloo functions, you need to compute the log-likelihood values of each observation for every posterior draw (i.e. an S-by-N matrix, where S is the number of posterior draws and N = 30 is the total number of observations). This can be done in the generated quantities block in the Stan code.
2. Compute the PSIS-LOO elpd values and the kˆ-values for each of the three models.
Hint 1! The elpd values for all models should be between -100 and -150.
3. Compute the e ective number of parameters pe for each of the three models. Interpret the pe and compare it with the number of estimated parameters. Does pe make sense? Why or why not? Hint! The estimated e ective number of parameters in the model can be computed from equation (7.15) in the book, where elpdloo-cv is the PSIS-LOO value (sum of the LOO log densities) and lpd is given by equation (7.5) in the book.
4. Assess how reliable the PSIS-LOO estimates are for the three models based on the kˆ-values.
5. An assessment of whether there are di erences between the models with regard to the elpdloo-cv, and if so, which model should be selected according to PSIS-LOO.
6. Both the Stan and R code should be included in your report.