Title: Quiz 9
Course: F19 Introduction to Deep Learning

Q: (Select all that apply.) For the purpose of predicting stock values, a well-trained finite response system that looks 7-time steps into the past can be expected to be able to predict:
- Weekly Trends
- Daily Trends
- Bi-Weekly Trends (every 2 weeks)
- Annual Trends
A: Weekly Trends; Daily Trends

Q: (Select all that apply.) Which of the following are true for the Elman and Jordan networks:
- The training processes for neither of the types network captures true recurrence
- Elman networks maintain a running average of outputs from previous time instants within a memory unit, while Jordan networks store hidden unit values for one time instant in a “context” unit.
- Both networks are not actually recurrent during inference because the history is not fully represented in the networks.
- Jordan networks have infinite memory since they factor in all the past outputs during inference
A: The training processes for neither of the types network captures true recurrence; Jordan networks have infinite memory since they factor in all the past outputs during inference

Q: (Select all that apply.) You are training an RNN using ReLU activation for hidden units, which of the following are possible.
- The outputs of the RNN may blow up
- The network is not guaranteed BIBO stable
- Saturation will occur
- Since the input-output relationship of the Relu is piecewise-linear, the network is BIBO stable
A: The outputs of the RNN may blow up; The network is not guaranteed BIBO stable

Q: (Select all that apply.) Which of the following statements are true about the RNN and LSTM models.
- LSTMs are more complex units than RNNs, and hence retain memory longer, but in the end both of them are similar in memory retention behavior in that memories of inputs are eventually forgotten (i.e individual inputs only have infinitesimal influence on the hidden and output activations of the network after a sufficient amount of time, e.g. t >1000, has elapsed).
- In RNN, memory behavior is primarily a function of the parameters of the network, while in an LSTM memory behavior is controlled by the input.
- In primary advantage of an LSTM over an RNN is that while in the latter the recurrent weight matrix may have singular values greater than or less than 1, all the singular values of the recurrent weights matrix in the memory line of the LSTM (the constant error carousel) are 1.0.
- The main advantage of LSTM over RNN is that it uses multiplicative (gated) combinations of inputs instead of the additive combinations used by the RNN.
A: LSTMs are more complex units than RNNs, and hence retain memory longer, but in the end both of them are similar in memory retention behavior in that memories of inputs are eventually forgotten (i.e individual inputs only have infinitesimal influence on the hidden and output activations of the network after a sufficient amount of time, e.g. t >1000, has elapsed).; In RNN, memory behavior is primarily a function of the parameters of the network, while in an LSTM memory behavior is controlled by the input.; In primary advantage of an LSTM over an RNN is that while in the latter the recurrent weight matrix may have singular values greater than or less than 1, all the singular values of the recurrent weights matrix in the memory line of the LSTM (the constant error carousel) are 1.0.
If right: In LSTMs too the memory content has recurrence through the input, input gate, and forget gate. Recurrence is both direct, through peephole connections, and indirect, through the output h(t) = outputgate(t) * tanh(c(t)). Depending on the weights of the various connections, the recurrence will modify the memory. Thus, even for well-trained LSTMs there is a tendency for the memory to fade or grow with time, even in the absence of additional input. The CEC effectively has an identity recurrent weights matrix. This is why the memory by itself doesn’t expand or contract without external additive or multiplicative modifications. On the other hand, in a regular RNN, the recurrent connections have learned weight matrices that have non-unity singular values and can expand or contract the stored hidden value over time, even without external influence.

Q: Let us assume that you are looking to construct word embeddings for two different languages and you would like for your embeddings to align approximately (as shown in the below chart) so you can learn a linear transformation from the english space into the spanish space (this projection is obtained using the first 2 dimensions of a PCA).

You can achieve this by doing training word embeddings in what way?

To answer this question, you may want to refer to Additional Reading:

https://arxiv.org/pdf/1309.4168.pdf
- Data: 2 Monolingual Corpora with no alignment constraints, Model: Word2Vec
- Data: 2 Monolingual Corpora that are content aligned, Model: Word2Vec
- Data: Bilingual Word Aligned Translations, Model: Word2Vec
- Data: Bilingual Document Aligned Translations, Model: MIT GloVe
- Data: Bilingual Sentence Aligned Translations, Model: Word2Vec
A: Data: 2 Monolingual Corpora with no alignment constraints, Model: Word2Vec

Q: (Select all that apply.) Which of the following is true?
- Language (text) can be modeled using a TDNN.
- Word embeddings can learned using a TDNN.
- Word embeddings can be learned by predicting a word in a sentence from its word context.
- Word embeddings can be learned by using each word to predict its own context.
A: Language (text) can be modeled using a TDNN.; Word embeddings can learned using a TDNN.; Word embeddings can be learned by predicting a word in a sentence from its word context.; Word embeddings can be learned by using each word to predict its own context.

Q: Under the CTC framework, which of the following would be a possible transcript of the utterance "deep learning" (assuming each alphabet corresponds to a phoneme label. Spaces are also valid characters)
- dee-eeee-p-l-ee-aa-r-n-i-nn-g
- d-e-e-p l-e-a-r-n-i-n-g
- d-ee-p l-e-a-r-n-i-n-g
- dd-d--e-eep ll-ee-a-rn-i-nng
A: d-e-e-p l-e-a-r-n-i-n-g

Q: (Select all that apply.) Which of the following is true about the derivative of the divergence between the output of the network and the aligned desired output sequence, for CTC learning?
- It is the sum of the divergences at each time step
- It cannot be decomposed into the sum of divergences at individual time steps
- It depends only on the probabilities assigned to the symbols in the target output sequence
- It depends on the probabilities assigned to all symbols at all time steps
A: It is the sum of the divergences at each time step; It depends only on the probabilities assigned to the symbols in the target output sequence

Q: The original Cascade-Correlation architecture is an example of:
- Backprop-like uni-directional neural network.
- Boltzmann-Machine-like generative model.
- Convolutional neural network.
- Detailed model of interacting neurons.
A: Backprop-like uni-directional neural network.

Q: A recurrent neural network with a single hidden layer has 3 neurons in its recurrent hidden layer (i.e. the hidden representation is a 3-dimensional vector).  The recurrent weights matrix of the hidden layer is given by

\(W = \begin{bmatrix}0.75 & 0.25 & 0 \\0.2 & -0.1 & 0.7 \\-0.2 & 0.65 & 0.15\end{bmatrix}\)

The hidden units have \(\tanh()\) as their activation function.  All bias values are 0.

Let us define the “memory duration” of the network as the number of time steps \(N\) such that for an isolated input at time \(t\) (with no previous or subsequent inputs) the length of the hidden activation vector at time \(t+N\) almost certainly falls to less than 1/100th of its value at time \(t\).  This is effectively the amount of time in which the influence of the input at time \(t\) all but vanishes from the network.

What is the memory duration of the above network (choose the closest answer). Assume \(h(t-1) = 0\) (the zero vector).  You may want to simulate the network to determine this value (an analytical solution is difficult to get).
- 20
- 25
- 30
- 35
A: 20