Title: Quiz 7
Course: F19 Introduction to Deep Learning

Q: As explained in class, the following pseudo-code performs the forward computation for one layer (layer \(\texttt{l}\)) of a CNN.

\(\texttt{for j = 1:D}_\texttt{l} \\ \quad\texttt{for x = 1:W-K+1} \\ \quad\quad\texttt{for y = 1:H-K+1} \\ \quad\quad\quad\texttt{z(l,j,x,y) = 0} \\ \quad\quad\quad\texttt{for i = 1:D}_\texttt{l-1} \\ \quad\quad\quad\quad\texttt{for x’ = 1:K}_\texttt{l} \\ \quad\quad\quad\quad\quad\texttt{for y’ = 1:K}_\texttt{l} \\ \quad\quad\quad\quad\quad\quad\texttt{z(l,j,x,y) += w(l,j,i,x’,y’)Y(l-1,i,x+x’-1,y+y’-1)} \\ \quad\quad\quad\texttt{Y(l,j,x,y) = activation(z(l,j,x,y))}\)

Which of the following is true?
- \(\texttt{K}_\texttt{l}\)K l is the width of the filter and \(\texttt{D}_\texttt{l-1}\)D l-1 is the number of channels in the \(\texttt{l-1}\)l-1 layer and \(\texttt{D}_\texttt{l}\)D l is the number of channels in the \(\texttt{l}\)lth layer.
- \(\texttt{K}_\texttt{l}\)K l is the height of the input map and \(\texttt{D}_\texttt{l-1}\)D l-1 is the number of channels in the \(\texttt{l-1}\)l-1 layer and \(\texttt{D}_\texttt{l}\)D l is the number of channels in the \(\texttt{l}\)lth layer.
- \(\texttt{K}_\texttt{l}\)K l is the height of the filter and \(\texttt{D}_\texttt{l-1}\)D l-1 is the filter width in the \(\texttt{l-1}\)l-1 layer and \(\texttt{D}_\texttt{l}\)D l is the filter width in the \(\texttt{l}\)lth layer.
- \(\texttt{K}_\texttt{l}\)K l is the height of the filter and \(\texttt{D}_\texttt{l-1}\)D l-1 is the filter width in the \(\texttt{l-1}\)l-1 layer and \(\texttt{D}_\texttt{l}\)D l is the filter width in the \(\texttt{l}\)lth layer.
- \(\texttt{K}_\texttt{l}\)K l is the total number of layers and \(\texttt{D}_\texttt{l-1}\)D l-1 is the number of channels in the \(\texttt{l-1}\)l-1 layer and \(\texttt{D}_\texttt{l}\)D l is the number of channels in the \(\texttt{l}\)lth layer.
A: \(\texttt{K}_\texttt{l}\)K l is the width of the filter and \(\texttt{D}_\texttt{l-1}\)D l-1 is the number of channels in the \(\texttt{l-1}\)l-1 layer and \(\texttt{D}_\texttt{l}\)D l is the number of channels in the \(\texttt{l}\)lth layer.

Q: The following is the equation for the derivative of the divergence with respect to the output at location \(t\) of the \(m\)th channel in the \((l-1)\)th layer for a TDNN. Assume all filters have kernel width \(K\).

\(\frac{\partial Div}{\partial y(l-1, m, t)}=\sum\limits_{n} \sum\limits_{t'} \hat{w}(l,n,m,t')\frac{\partial Div}{\partial z_{\text{shift}}(l,n,t+t'-1)}\)

What do \(\hat{w}(l,n,m,t)\) and \(z_\text{shift}(l,n,t)\) represent according to the equation?
- w represents the filters of the \(l\)lth layer after they have been flipped left to right and \(z_\text{shift}\)z shift are the affine maps of the \(l\)lth layer after they have been shifted left by \(K-1\)K − 1 indices such that the index \(K\)K now becomes the index \(1\)1.
- \(\hat{w}\)w ^ represents the filters of the \((l-1)\)( l − 1 )th layer after they have been flipped left to right and \(z_\text{shift}\)z shift are the affine maps of the \(l\)lth layer after they have been shifted left and up by \(K-1\)K − 1 indices such that the index \(K\)K now become the index \(0\)0.
- \(\hat{w}\)w ^ represents the filters of the \((l-1)\)( l − 1 )th layer after they have been flipped left to right and \(z_\text{shift}\)z shift are the affine maps of the \(l\)lth layer after they have been shifted right by \(K-1\)K − 1 indices such that the index \(1\)1 now become the index \(K\)K.
- \(\hat{w}\)w ^ represents the filters of the \(l\)lth layer after they have been flipped left to right and \(z_\text{shift}\)z shift are the affine maps of the \(l\)lth layer after they have been shifted left by \(K\)K indices such that the index \(K\)K now become the index \(0\)0.
A: \(\hat{w}\)w ^ represents the filters of the \((l-1)\)( l − 1 )th layer after they have been flipped left to right and \(z_\text{shift}\)z shift are the affine maps of the \(l\)lth layer after they have been shifted right by \(K-1\)K − 1 indices such that the index \(1\)1 now become the index \(K\)K.

Q: While backpropagating through a max pooling layer, the divergence derivative at the output of a max pool filter is
- equally distributed over the input pool
- assigned to the input location of the maximum value, when there is a unique maximum value in the pool
- randomly assigned to one of the input locations
- assigned such that more weightage is given to the location of the maximum value and the remaining locations get equal parts of the derivative
A: assigned to the input location of the maximum value, when there is a unique maximum value in the pool

Q: While backpropagating through a max pooling layer, the divergence derivative at the input locations of the non maximum values is
- 0
- Identical to the derivative at the location of the maximum value
- NaN (not a number)
- Given a small, nonzero value
A: 0

Q: The innermost loop in an explicitly indexed (non-vectorized) implementation of a 2D CNN, is written as:

\(\texttt{z(l,j,x,y) += w(l,j,i,x',y')Y(l-1,i,x+x'-1,y+y'-1)}\)

where \(\texttt{z(l,j,x,y)}\)is the value at location \(\texttt{x}\), \(\texttt{y}\) of the \(\texttt{j}\)th affine map at the \(\texttt{l}\)th layer,
\(\texttt{w(l,j,i,x',y')}\) is the value of the \(\texttt{j}\)th filter of the \(\texttt{l}\)th layer,  that is assigned to a shift \(\texttt{x'}\), \(\texttt{y'}\) within the \(\texttt{i}\)th channel of the \(\texttt{(l-1)}\)th layer, and \(\texttt{Y(l-1,i,x, y)}\) is the value of the output of the \(\texttt{i}\)th channel of the \(\texttt{(l-1)}\)th layer at location \(\texttt{x}\), \(\texttt{y}\).

If we use the same loops, loop variables and loop limits as the original CNN code, which of the following are correct backpropagation derivative updates
- \(\texttt{dY(l-1,i,x+x'-1,y+y'-1) += w(l,j,i,x',y')dz(l,j,x,y)} \\ \texttt{dw(l,j,i,x',y') += dz(l,j,x,y)Y(l-1,i,x+x'-1,y+y'-1)}\)
- \(\texttt{dY(l-1,i,x+x'-1,y+y'-1) += w(l,j,i,x',y')z(l,j,x,y)} \\ \texttt{dw(l,j,i,x',y') += dz(l-1,i,x+x'-1,y+y'-1)}\)
- \(\texttt{dY(l-1,i,x+x'-1,y+y'-1) += z(l,j,x,y)dw(l,j,i,x',y')} \\ \texttt{dw(l,j,i,x',y') += z(l,j,x,y)}\)
- \(\texttt{dY(l-1,i,x+x'-1,y+y'-1) = w(l,j,i,x',y')dz(l,j,x,y)} \\ \texttt{dw(l,j,i,x',y') = dz(l,j,x,y)Y(l-1,i,x+x'-1,y+y'-1)}\)
A: \(\texttt{dY(l-1,i,x+x'-1,y+y'-1) += w(l,j,i,x',y')dz(l,j,x,y)} \\ \texttt{dw(l,j,i,x',y') += dz(l,j,x,y)Y(l-1,i,x+x'-1,y+y'-1)}\)

Q: (Select all that apply.) In a 2D CNN the filters of the \(l\)th layer are convolved with the outputs of the \((l-1)\)th layer with a stride greater than \(1\).  Select all of the following statements that are true.
- The outputs of the \((l-1)\)th layer are never recoverable from the filters and affine combination maps at the \(l\)th layer
- The outputs of the \((l-1)\)th layer are recoverable from the filters and affine combination maps at the \(l\)th layer if there are a sufficiently large number of filters
- If all information must be retained even after convolution with stride greater than \(1\), then a sufficiently large number of filters must be used.
- The output of such a convolution will be always the same size as the input
A: The outputs of the \((l-1)\)th layer are recoverable from the filters and affine combination maps at the \(l\)th layer if there are a sufficiently large number of filters; If all information must be retained even after convolution with stride greater than \(1\), then a sufficiently large number of filters must be used.

Q: (Select all that apply.) Which of the following are true statements?
- In a CNN the relationship between the output maps of the \((l-1)\)th layer and the affine maps of the \(l\)th layer is given by a set of simultaneous equations.
- The set of affine maps at the \(l\)th layer can always be used to recover the output maps of the \((l-1)\)th layer exactly given the filters.
- As the stride of the convolution increases, the number of filters must also be increased, if we want the output of the convolution to retain all information about the input (such that the input maps can be recovered from the output affine maps).
- In general, the information loss from downsampling can be compensated by increasing the number of filters in subsequent layers.
A: In a CNN the relationship between the output maps of the \((l-1)\)th layer and the affine maps of the \(l\)th layer is given by a set of simultaneous equations.; As the stride of the convolution increases, the number of filters must also be increased, if we want the output of the convolution to retain all information about the input (such that the input maps can be recovered from the output affine maps).

Q: (True/False) Having a convolution layer with stride equal to 1 and linear activation followed by a mean pooling layer with the stride of 2 can also be achieved with a convolutional layer with a larger filter and a stride of 2.
- True
- False
A: True

Q: (Select all that apply.) Which of the following is true when the size of the output map in the \(l\)th layer of a 2D CNN is greater than the size of the maps in the \((l-1)\)th layer.
- This is equivalent to scanning the input with an MLP where the number of neurons in the \(l\)th layer is greater than the number of neurons in the \((l-1)\)th layer.
- Every location in the output map is computed from the same number of location in the input map
- This can be equivalently interpreted as one layer of an MLP where the weight matrix is taller than it is broad.
- All input maps contribute to each location of the output map.
A: This is equivalent to scanning the input with an MLP where the number of neurons in the \(l\)th layer is greater than the number of neurons in the \((l-1)\)th layer.; This can be equivalently interpreted as one layer of an MLP where the weight matrix is taller than it is broad.; All input maps contribute to each location of the output map.

Q: The figure below shows one residual block in a ResNet. Here \(W\) represents the filters of the convolution and ⨂ represents convolution.

Let the activation function be a scalar activation function \(\sigma(\cdot)\) and its derivative be \(\sigma '\).

What would be the gradient of the final output of the above residual unit with respect to the input \(x\).  In the equations below, we are generically using \(\nabla_uV\) as the derivative of \(V\) with respect to \(u\) and doesn’t specifically refer to gradients, Jacobeans, etc.

\(fud\) is short for flip up and down. \(flr\) is the flip left to right.  The symbol "\(\circ\)" represents component-wise multiplication
- \(\nabla_x Div=flr(W) \otimes zeropad(\nabla_y Div)\)∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v )∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v )∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v )∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v )
- \(\nabla_x Div = flr(W) \otimes zeropad(\nabla_y Div)\)∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v )∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v )∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v )∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v )
- \(\nabla_x Div = flr(W) \otimes zeropad(\nabla_y Div) + \nabla_{out} Div\)∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v
- \(\nabla_x Div = fud(W) \otimes zeropad(\nabla_y Div) + \nabla_{out} Div\)∇ x D i v = f u d ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f u d ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f u d ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f u d ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v
- \(\nabla_x Div=flr(fud(W)) \otimes zeropad(\nabla_y Div) + \nabla_{out} Div\)∇ x D i v = f l r ( f u d ( W ) ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f l r ( f u d ( W ) ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f l r ( f u d ( W ) ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f l r ( f u d ( W ) ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v
- \(\nabla_x Div = flr(W) \otimes zeropad ( \nabla_{out} Div \circ \sigma ' (y) + \nabla_{out} Div)\)∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ o u t D i v ∘ σ ′ ( y ) + ∇ o u t D i v )∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ o u t D i v ∘ σ ′ ( y ) + ∇ o u t D i v )∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ o u t D i v ∘ σ ′ ( y ) + ∇ o u t D i v )∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ o u t D i v ∘ σ ′ ( y ) + ∇ o u t D i v )
A: \(\nabla_x Div = flr(W) \otimes zeropad(\nabla_y Div) + \nabla_{out} Div\)∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v∇ x D i v = f l r ( W ) ⊗ z e r o p a d ( ∇ y D i v ) + ∇ o u t D i v