Description

Created by Ankur Mali Course Policy: Carefully read all the instructions, before you start working on the assignment
• All Problems should be coded in Tensorflow 1.14 or above
• No single line answers are accepted in the submission.
• Late assignments will suffer 50 percent loss after the first day and all loss after the second.
• We will be creating Canvas submission page for this.You have to submit python file[no ipython allowed] given in gitrepo along with your response pdf.
• All queries related to Assignment should have a subject line IST597:Assignment10000 Queries
Problem 1:Implementing various Recurrent neural network cells using basic tensorflow ops (3+3+2=8
points)
One step of shallow RNN st−1 is previous hidden state, xt is current input and st+1 is future hidden state. RNNs are stateful models, the process sequence overt time. In the given diagram , RNN takes input at current time step(xt) and also previous hidden state (st−1) and then computes output(yt) and updates the current hidden state(st). Our assignment is focused on exploring the effectiveness of various recurrent units in
…

GRU/LSTM
Unit
st−1 st+1
solving temporal credit assignment problem.
In this assignment we will be implementing and comparing Gated Recurrent Unit with minimal Gated Unit. I have provided the scripts to create LSTM cell in my github repo, please follow it. Current code base can work with keras/Tensorflow eager mode as well as graph mode.
You have to design the forward as well as backward call for your custom RNN cells.[Note:- Feel free to use keras backend with your custom RNN]. People are free to use gradientTape or kerasbackend for backward call. Below we are providing equations for two Recurrent unit.
1
– Assignment #10000 2
1 Gated Recurrent Unit (GRU)
Update equation for Gated Recurrent Unit(GRU). GRU is the simplified version of LSTM and is also as powerful as LSTM in terms of generalization performance. We encourage you to read [1] for in-depth understanding.
zt = σ(Wz[st−1,xt] + bz) rt = σ(Wr[st−1,xt] + br) s˜t s
2 Minimal Gated Unit (MGU)
ft = σ(Wf[st−1,xt] + bf) s˜t
s
Things to do
• You will be writing your own training procedure to train RNNCell on notmnist dataset [data]. Report classification error over 3 trials.
• Implement GRU and MGU using custom code provided.
• Compare GRU and MGU over 3 trials. Number of hidden units for the network can vary between 50 to 512. Number of hidden layers can vary between 1 to 4. You have flexibility to choose parameters based on resources available.
• Plot the training vs test curve for all the models. Report your findings.
References
[1] Cho, K., Van Merrienboer, B., Bahdanau, D., and Bengio, Y.¨ On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259 (2014).