Description

Project 1 – Digit Recognition
Marks: 10% of final assessment
Introduction
In this assignment, you will be implementing a single layer network, a two layer network and a convolutional network to classify handwritten digits. We will work with the MNIST dataset, a common dataset used to evaluate Machine Learning models.
Preliminaries
Before commencing this assignment, you should download and install TensorFlow, and the appropriate python version. It is also helpful to complete the ‘MNIST for beginners’ tutorial located on the TensorFlow
website https://www.tensorflow.org/get_started/mnist/beginners
TensorFlow (TF) is an opensource library primarily used to construct, train and evaluate machine learning models. TF allows rapid development and supports automatic differentiation – meaning backprop is able to be done automatically for any model adequately defined. TF also abstracts away much of the low-level code required to set up training on GPU’s; in many cases TF will automatically detect and utilize your computer’s GPU if it has one. Central to the design of TF is the concept of a ‘graph’ – a low level representation of a model consisting of nodes and tensors. Broadly, implementing a TF model can be broken down into two sections; creating the graph, and training/testing it. This assignment is mainly concerned with graph creation. You can read more about the general structure of TensorFlow here.
Getting Started
Copy the archive src.zip into your own filespace and unzip it. Then type cd src
You will see two files: train.py and hw1.py Now run train.py by typing python3 train.py
When run for the first time, train.py should create a new folder called data and download a copy of the MNIST dataset into this folder. All subsequent runs of train.py will use this local data. (Don’t worry about the ValueError at this stage.) The file train.py contains the TensorFlow code required to create a session, build the graph, and run training and test iterations. It has been provided to assist you with the testing and evaluation of your model. While it is not required for this assignment to have a detailed understanding of this code, it will be useful when implementing your own models, and for later assignments.
The file train.py calls functions defined in hw1.py and should not be modified during the course of the assignment. A submission that does not run correctly when train.py is called will lose marks. The only situation where you should modify train.py is when you need to switch between different network architectures. This can be done by setting the global variable on line 7: network = “none” to any of the following values:
network = “onelayer” network = “twolayer” network = “conv”
Stage 0: Provided Code
The functions input_placeholder() and target_placeholder() specify the inputs and outpus of your networks in the TensorFlow graph. They have been implemented for you.
In addition, there is a function train_step() that passes batches of images to the constructed TensorFlow Graph during training. It’s implementation should help you understand the shape and structure of the actual data that is being provided to the model.
Unless otherwise specified, the underlying type (dtype) for each TF object should be float32. INPUT_SIZE, where it appears in comments, refers to the length of a flattened single image; in this case 784. OUTPUT_SIZE, where it appears in comments, refers to the length of a one-hot output vector; in this case 10.
In the provided file hw1.py, detailed specifications are provided in the comments for each function.
Stage 1: Single-Layer Network (3 marks)
Write a function onelayer(X, Y, layersize=10) which creates a TensorFlow model for a one layer neural network (sometimes also called logistic regression). Your model should consist of one fully connected layer with weights w and biases b, using softmax activation.
Your function should take two parameters X and Y that are TensorFlow placeholders as defined in input_placeholder() and target_placeholder(). It should return varibles w, b, logits, preds, batch_xentropy and batch_loss, where:
• w and b are TensorFlow variables representing the weights and biases, respectively
• logits and preds are the input to the activation function and its output
• xentropy_loss is the cross-entropy loss for each image in the batch
• batch_loss is the average of the cross-entropy loss for all images in the batch
Change line 7 of train.py to network = “onelayer” and test your network on the
MNIST dataset by typing python3 train.py
It should achieve about 92% accuracy after 5 epochs of training.
It is a good idea to submit your code after completing Stage 1, because the submit script will run some simple tests and give you some feedback on whether your model is correctly structured.
Stage 2: Two-Layer Network (3 marks)
Create a TensorFlow model for a Neural Network with two fully connected layers of weights w1, w2 and biases b1, b2, with ReLU activation functions on the first layer, and softmax on the second. Your function should take two parameters X and Y that are TensorFlow placeholders as defined in input_placeholder() and target_placeholder(). It should return varibles w1, b1, w2, b2, logits, preds, batch_xentropy and batch_loss, where:
• w1 and b1 are TensorFlow variables representing the weights and biases of the first layer
• w2 and b2 are TensorFlow variables representing the weights and biases of the second layer
• logits and preds are the inputs to the final activation functions and their output
• xentropy_loss is the cross-entropy loss for each image in the batch
• batch_loss is the average of the cross-entropy loss for all images in the batch
Change line 7 of train.py to network = “twolayer” and test your network on the
MNIST dataset by typing python3 train.py
Stage 4: Convolutional Network (4 marks)
Create a TensorFlow model for a Convolutional Neural Network. This network should consist of two convolutional layers followed by a fully connected layer of the form:
conv_layer1 → conv_layer2 → fully-connected → output
Your function should take two parameters X and Y that are TensorFlow placeholders as defined in input_placeholder() and target_placeholder(). It should return varibles conv1, conv2, w, b, logits, preds, batch_xentropy and batch_loss, where:
• conv1 is a convolutional layer of convlayer_sizes[0] filters of shape filter_shape
• conv2 is a convolutional layer of convlayer_sizes[1] filters of shape filter_shape
• w and b are TensorFlow variables representing the weights and biases of the final fully connected layer
• logits and preds are the inputs to the final activation functions and their output
• xentropy_loss is the cross-entropy loss for each image in the batch
• batch_loss is the average of the cross-entropy loss for all images in the batch Hints:
1. use tf.layer.conv2d
2. the final layer is very similar to the onelayer network, except that the input will be from the conv2 layer. If you reshape the conv2 output using tf.reshape, you should be able to call onelayer() to get the final layer of your network
Change line 7 of train.py to network = “conv” and test your network on the MNIST dataset by typing python3 train.py
Notes
All TensorFlow objects, if not otherwise specified, should be explicity created with tf.float32 datatypes. Not specifying this datatype for variables and placeholders will cause your code to fail some tests.
Visualizing Your Models
In addition to the output of train.py, you can view the progress of your models and the created TensorFlow graph using the TensorFlow visualization platform, TensorBoard. After beginning training, run the following command from the src directory: python3 -m tensorflow.tensorboard –logdir=./summaries
Depending on your installation, the following command might also work: tensorboard –logdir=./summaries
1. open a Web browser and navigate to http://localhost:6006
2. you should be able to see a plot of the train and test accuracies in TensorBoard
3. if you click on the histogram tab you’ll also see some histograms of your weights, biases and the pre-activation inputs to the softmax in the final layer
Make sure you are in the same directory from which train.py is running. Don’t worry if you are unable to get TensorBoard working; it is not required to complete the assignment, but it can be a useful tool to monitor training, so it is probably worth your while becoming familiar with it. Click here for more information:
Submission
You can test your code by typing python3 train.py
Once submissions are open, you should submit by typing
give cs9444 hw1 myfile.py
(Your file can have any name, but must end in .py)
When you submit, you will see some feedback for Stage 1, which you can use to check that you are structuring your code correctly.
You can submit as many times as you like – later submissions will overwrite earlier ones. You can check that your submission has been received by using the following command:
9444 classrun -check
Questions relating to the project can also be posted to the Forums on the course Web page.
If you have a question that has not already been answered on the FAQ or the Forums, you can email it to alex.long@student.unsw.edu.au
You should always adhere to good coding practices and style. In general, a program that attempts a substantial part of the job but does that part correctly will receive more marks than one attempting to do the entire job but with many errors.
DO NOT COPY FROM OTHERS; DO NOT ALLOW ANYONE TO SEE YOUR CODE
Good luck!