Description

General information
• The recommended tool in this course is R (with the IDE R-Studio). You can download R here and R-Studio here. There are tons of tutorials, videos and introductions to R and R-Studio online. You can find some initial hints here.
• You can write the report with your preferred software, but the outline of the report should follow the instruction in the R markdown template that can be found here.
• Report all results in a single, anonymous *.pdf -file and return it to peergrade.io. • The course has its own R package with data and functionality to simplify coding. To install the package just run the following:
1. install.packages(“remotes”)
2. remotes::install_github(“avehtari/BDA_course_Aalto”, subdir = “rpackage”)
• Many of the exercises can be checked automatically using the R package markmyassignment. Information on how to install and use the package can be found here. • Additional self study exercises and solutions for each chapter in BDA3 can be found here.
• We collect common questions regarding installation and technical problems in a course Frequently Asked Questions (FAQ). This can be found here. • If you have any suggestions or improvements to the course material, please feel free to create an issue or submit a pull request to the public repository!!
Information on this assignment
This exercise is related to Chapter 7. The maximum amount of points from this assignment is 6.
Note! This assignment build upon assignment 7, so be sure that assignment 7 is correct before you start with this assignment.
Reading instructions: Chapter 7 in BDA3, see reading instructions here. Also read the paper on PSIS-LOO that can be found here or here.
Grading instructions: The grading will be done in peergrade. All grading questions and evaluations for assignment 8 can be found here
Reporting accuracy: As many significant digits as justified by the Monte Carlo error and posterior accuracy.
Stan manual can be found at http://mc-stan.org/documentation/. From this website, you can also find a lot of other useful material about Stan.
Model assessment: LOO-CV for factory data with Stan (6p)
Use leave-one-out cross-validation (LOO-CV) to assess the predictive performance of the pooled, separate and hierarchical Gaussian models for the factory dataset (see the second exercise in Assignment 7). To read in the data, just use:
> library(aaltobda) > data(“factory”)
PSIS-LOO is a recently developed method for approximating the exact LOO and is thus not in BDA3. For more information, see the lecture slides and the original paper here or here.
Use Stan for fitting the models, and the loo R package for computing the approximate
LOO-CV given the posterior samples provided by Stan. You can install the package as
> install.packages(“loo”)
Python users can use PSIS-LOO implementation in ArviZ library.
The report should include the following parts.
1. Fit the models with Stan as instructed in Assignment 7. To use the loo or psisloo functions, you need to compute the log-likelihood values of each observation for every posterior draw (i.e. an S-by-N matrix, where S is the number of posterior draws and N = 30 is the total number of observations). This can be done in the generated quantities block in the Stan code; for a demonstration, see the Gaussian linear model lin.stan in the R Stan examples that can be found here.
2. Compute the PSIS-LOO elpd values and the kˆ-values for each of the three models.
3. Compute the effective number of parameters peff for each of the three models.
Hint! The estimated effective number of parameters in the model can be computed from equation (7.15) in the book, where elpdloo-cv is the PSIS-LOO value (sum of the LOO log densities) and lpd is given by equation (7.5) in the book.
4. Assess how reliable the PSIS-LOO estimates are for the three models based on the kˆ-values.
5. An assessment of whether there are differences between the models with regard to the elpdloo-cv, and if so, which model should be selected according to PSIS-LOO.
6. Both the Stan and R code used should be included in the your report.