Title: Quiz 6
Course: F19 Introduction to Deep Learning

Q: (Select all that apply.) A time-delay neural network has three convolutional layers followed by a softmax unit. The convolutional layers have the following structure:

- The first hidden layer has 4 filters of kernel-width 2;
- The second layer has 3 filters of kernel-width 2;
- The third layer has 2 filters of kernel-width 2.

As explained in class, the convolution layers of this TDNN are exactly equivalent to scanning the input with a (shared-parameter) MLP (and passing the set of outputs of the MLP at the individual time instants through a final softmax). What would be the architecture of this scanning MLP? Assume that the stride of the convolution is 2 in every layer.
- A three-layer MLP with 16 neurons in the first layer, 6 in the second layer and 2 in the third layer
- A three layer MLP with 8 neurons in the first layer, 6 neurons in the second layer and 4 neurons in the third layer
- A three layer MLP with 4 neurons in the first layer, 3 in the second layer and 2 in the third layer
- A one-layer MLP with a layer of size 24
- Classification with this TDNN cannot be modeled as scanning with an MLP
A: A three-layer MLP with 16 neurons in the first layer, 6 in the second layer and 2 in the third layer
Feedback: The basic MLP is shown below. The sequence of little black bars shows the time sequence of input vectors. All blocks with the same background color are identical and share parameters (weights and biases). Neurons represented using the same color are identical and share parameters. The full scan would have the structure shown below

Q: (Select all that apply.) A 1-D convolutional neural network has three convolutional layers followed by a softmax unit. The convolutional layers have the following structure:

- The first hidden layer has 2 filters of kernel-width 2;
- The second layer has 3 filters of kernel-width 3;
- The third layer has 2 filters of kernel-width 4.

As explained in class, the convolution layers of this TDNN are exactly equivalent to scanning the input with a (shared-parameter) MLP (and passing the set of outputs of the MLP at the individual time instants through a final softmax). What would be the architecture of this scanning MLP? Assume that the stride of the convolution is 1 in every layer.
- A three-layer MLP with 12 neurons in the first layer, 12 in the second layer and 2 in the third layer
- A three layer MLP with 4 neurons in the first layer, 9 neurons in the second layer and 8 neurons in the third layer
- A three layer MLP with 2 neurons in the first layer, 3 in the second layer and 2 in the third layer
- A one-layer MLP with a layer of size 8
A: A three-layer MLP with 12 neurons in the first layer, 12 in the second layer and 2 in the third layer
If right: The basic MLP is shown below. The sequence of little black bars shows the time sequence of input vectors. All blocks with the same background color are identical and share parameters (weights and biases). Neurons represented using the same color are identical and share parameters. The full scan would have the structure shown below

Q: (Select all that apply.) A time-series input is scanned for a pattern using the following MLP.  While scanning, the entire network strides two-time steps at a time. Subsequently, the outputs of the MLP at each time step are combined through a softmax.  In the figure, neurons of identical color within each layer have identical responses.

The same operation can be performed using a time-delay neural network.  What will the architecture of this TDNN be?
- Three convolutional layers. 3 filters of kernel-width 2 in the first layer, with stride 2, 3 filters of kernel-width 2 in the second layer, with stride 1, and 2 filters of kernel-width 2 in the third layer, with a stride of 1.
- Three convolutional layers. 9 filters of kernel-width 2 in the first layer, with stride 2, 6 filters of kernel-width 2 in the second layer, with stride 2, and 2 filters of kernel-width 2 in the final layer with stride 1.
- Three convolutional layers. 3 filters of kernel-width 2 in the first layer, with stride of 2, 3 filters of kernel-width 6 with stride 3 in the second layer, and 2 filters of kernel-width 6 with stride 3 in layer 3.
- Two convolutional layers and a flat layer. 9 filters of kernel-width 2 in the first layer, with stride 2. 6 filters of kernel-width 2 with stride 3 in the second layer. The final flat layer has two neurons with all the outputs of the second layer going to them.
A: Three convolutional layers. 3 filters of kernel-width 2 in the first layer, with stride 2, 3 filters of kernel-width 2 in the second layer, with stride 1, and 2 filters of kernel-width 2 in the third layer, with a stride of 1.

Q: Suppose a convolutional neural network for detecting flowers is trained on images that tend to have blue flowers in the upper left corner and pink flowers in the lower right corner. It is then tested on two different sets of test instances, test set 1 with blue flowers in the upper left corner, and pink flowers in the lower right corner, test set 2 with pink flowers in the upper left corner and blue flowers in the lower right corner. If the final layer of the network is fully connected then:
- The error rate of the network will be higher on the testset 2 than on testset 1
- The error rate of the network will be about the same on both test sets
- The error rate of the network will be lower on testset 2 than on testset 1
- The expected value of the error rate will be worse than the empirical test error rate
A: The error rate of the network will be higher on the testset 2 than on testset 1

Q: Let G be a group of transformations used by a transform-invariant CNN. Depending on your choice of G, which of the following are true (the question compares operations based on G to those in a standard CNN)?
- The sliding inner product in a standard translational convolution could generalize to a sliding-and-rotating inner-product.
- The sliding inner product in a standard translational convolution could generalize to a sliding-and-flipping inner product.
- The sliding inner product in a standard translational convolution could generalize to a sliding-and-scaling inner product.
- The sliding inner product in a standard translational convolution could generalize to an outer product.
A: The sliding inner product in a standard translational convolution could generalize to a sliding-and-rotating inner-product.; The sliding inner product in a standard translational convolution could generalize to a sliding-and-flipping inner product.; The sliding inner product in a standard translational convolution could generalize to a sliding-and-scaling inner product.

Q: (Select all that apply.) Which of these pseudocodes perform the operations of a 2D CNN ?

Hint:

Recall that by “CNN” that we mean it scans a 2D image of size  \(U\times V\)  for patterns. Don’t worry about edge cases or shapes: assume that all these codes are syntactically correct and represent different networks, and that all variables  \(Y\)  are preallocated to size sufficiently greater than  \(U\times V\)  to ensure no exceptions happen in the code, and that these preallocated values of  \(Y\)  (other than the input to the net) are initialized to 0.

In the codes below, the index “l” represents the layer index.  The networks have  \(L\) layers.  \(N\) represents the number of filters in a layer and  \(K\) is the size of the filters.

Assume  \(Y(0,*)\) is the input to the network, where “*” represents the necessary set of variables required to represent the entire input. The number of indices represented must be inferred from the code.  In the notation below, in the products of w and   \(Y\) , when both are multi-dimensional arrays (tensors), the notation represents a scalar-valued tensor-inner-product, where corresponding components of the w and  \(Y\)  tensors are multiplied, and the set of pair-wise products is added to result in a scalar.
- for l = 1:L

for u = 1:U

for v = 1:V

Y(l,u,v) = activation(w(l,u,v)Y(l-1,u,v))

Y = softmax({Y(:,U,V)})
- for u = 1:U

for v = 1:V

for l = 1:L

for m = 1:N

Y(l,u,v) = activation(w(l,m)Y(l-1,u:u+2,v:v+2))

Y = softmax({Y(L,:,:)})
- for u = 1:U

for v = 1:V

for l = 1:L

for m = 1:N

Y(l,m,u,v) = \

activation(w(l,m,:,:,:)Y(l-1,:,u:u+K,v:v+K))

Y = softmax({Y(L,:,:,:)})
- for l = 1:L

for u = 1:U

for v = 1:V

for m = 1:N

Y(l,m,u,v) = \

activation(w(l,m,:,:,:)Y(l-1,:,u:u+K,v:v+K))

Y = softmax({Y(L,:,:,:)})
- for l = 1:L

for m = 1:N

for u = 1:U

for v = 1:V

Y(l,m,u,v) = \

activation(w(l,m,:,:,:)Y(l-1,:,u:u+K,v:v+K))

Y = softmax({Y(L,:,:,:)})
A: for u = 1:U

for v = 1:V

for l = 1:L

for m = 1:N

Y(l,m,u,v) = \

activation(w(l,m,:,:,:)Y(l-1,:,u:u+K,v:v+K))

Y = softmax({Y(L,:,:,:)}); for l = 1:L

for u = 1:U

for v = 1:V

for m = 1:N

Y(l,m,u,v) = \

activation(w(l,m,:,:,:)Y(l-1,:,u:u+K,v:v+K))

Y = softmax({Y(L,:,:,:)}); for l = 1:L

for m = 1:N

for u = 1:U

for v = 1:V

Y(l,m,u,v) = \

activation(w(l,m,:,:,:)Y(l-1,:,u:u+K,v:v+K))

Y = softmax({Y(L,:,:,:)})

Q: Which of the following best characterizes the relationship between a time-delay neural network and a 1D convolutional network
- The time-delay neural network is similar to a 1D-convolutional network, but the update rule is different
- The time-delay neural network is similar to a 1D-convolutional network, but the output at each timepoint is an exponential moving average of the outputs at all of the timepoints so far
- The networks are the same
- The time-delay neural network is similar to a 1D-convolutional network, but adds a recurrent inputs for k previous timesteps
A: The networks are the same

Q: You can implement EXACTLY the operations of a convolutional layer using a single fully connected layer.
- True
- False
A: True

Q: Read the paper "Visualizing and Understanding convolutional networks" and select the true statements from the following:

https://cs.nyu.edu/~fergus/papers/zeilerECCV2014.pdf
- The authors learn the deconvolutional network parameters using backpropagation.
- Given the max-pool outputs and the 'switches' defined in the paper, the authors are able to fully reconstruct the features in the layer before the max-pool operation.
- Using a linear SVM or softmax classifier saw improvements in performance when using features extracted from higher layers.
- None of the other answers is plausible.
A: Using a linear SVM or softmax classifier saw improvements in performance when using features extracted from higher layers.

Q: Hubel and Wiesel's 1959 experiment showed that:
- Cortical neurons involved in early visual processing do not depend upon the orientations of bars within their receptive fields
- Cortical neurons involved in early visual processing are driven more by certain orientations of bars within their receptive fields than by other orientation
- The retinal area that responds to a single stimulus is called a receptive field
- The retinal response does not depend on the frequency of light
A: Cortical neurons involved in early visual processing are driven more by certain orientations of bars within their receptive fields than by other orientation