Title: Quiz 2
Course: F19 Introduction to Deep Learning

Q: We want to build an MLP that composes the decision boundary shown in the figure below.  The output of the MLP must be 1 in the yellow regions and 0 otherwise.

Consider the following suboptimal MLP structure and given weights:

Each perceptron of this MLP computes the following function (bias is included in the sum, it is an implicit affine combination):

\(y=\left\{\begin{array}{ll}{1,} & {\sum_{i} \text {weight}_{i} \text {input}_{i} +bias \geq 0} \\ {0,} & {\text { otherwise }}\end{array}\right.\)

The weights of the connections are shown in blue against the corresponding black arrows. The biases are shown in red for all but two perceptrons.  What must the biases b1 and b2 be for the network to compute our target function perfectly? We require the biases to be integer values.  Please give the value of b1 first and b2 second in the spaces provided below.
- \(b_1=-2,b_2=-1\)b 1 = − 2 , b 2 = − 1b 1 = − 2 , b 2 = − 1
- \(b_1=-1,b_2=-2\)b 1 = − 1 , b 2 = − 2b 1 = − 1 , b 2 = − 2
- \(b_1=0,b_2=0\)b 1 = 0 , b 2 = 0b 1 = 0 , b 2 = 0
- \(b_1=2,b_2=1\)b 1 = 2 , b 2 = 1b 1 = 2 , b 2 = 1
- \(b_1\geq -2,b_{2}<0\)b 1 ≥ − 2 , b 2 < 0b 1 ≥ − 2 , b 2 < 0
- \(b_{1}\geq -2,b_{2}\leq0\)b 1 ≥ − 2 , b 2 ≤ 0b 1 ≥ − 2 , b 2 ≤ 0
A: \(b_1=-2,b_2=-1\)b 1 = − 2 , b 2 = − 1b 1 = − 2 , b 2 = − 1

Q: The true objective of learning network parameters, when we want it to represent a specific function, is to (pick all that apply)
- a. Minimize the average (expected) divergence between the output of the network and the actual function being approximated, over the entire domain of the input.
- b. Minimize the average error over all training points
- c. Minimize the expected  risk over the input.
- d. None of the above
A: a. Minimize the average (expected) divergence between the output of the network and the actual function being approximated, over the entire domain of the input.; c. Minimize the expected  risk over the input.

Q: Networks of perceptrons with threshold activations are hard to train because (select all that apply)
- The training data usually only provides labels for the entire network, and not for individual neurons in the network.
- The computational complexity of identifying the appropriate labels for each of the training instances for each of the hidden perceptrons may be exponential in the number of training instances.
- We cannot generally get any indication of whether increasing any particular parameter will increase or decrease the overall error.
- Threshold activations are inadequate to approximate most functions.
- All the other options are correct
A: The training data usually only provides labels for the entire network, and not for individual neurons in the network.; The computational complexity of identifying the appropriate labels for each of the training instances for each of the hidden perceptrons may be exponential in the number of training instances.; We cannot generally get any indication of whether increasing any particular parameter will increase or decrease the overall error.

Q: The function  \(f\left(x_{1}, x_{2}, x_{3}\right)=x_{1}^{2}-x_{2}^{2}+x_{3}^{2}+x_{1} x_{2}+x_{1} x_{3}+x_{2} x_{3}+x_{1}+x_{2}+x_{3}+3\) has (do not consider values at ±∞)
- A minimum
- A maximum
- A saddle point
- All other options are not correct
A: A saddle point

Q: How does ADALINE resolve the non-differentiability of the threshold activation?
- It tries to minimize the error between the desired binary output and the affine combination of inputs before the threshold activation is applied
- It uses a differentiable sigmoidal approximation to the threshold function during learning but uses the hard threshold activation subsequently when operating on test data.
- It computes the squared error between the output of the perceptron and the target output, instead of counting errors.
- It ignores the threshold activation during training and only applies it during testing.
A: It tries to minimize the error between the desired binary output and the affine combination of inputs before the threshold activation is applied; It ignores the threshold activation during training and only applies it during testing.

Q: How does ADALINE differ from MADALINE?
- ADALINE is used to train individual neurons, while MADALINE is used to  train the entire network
- ADALINE uses a crude approximation that ignores the threshold activation. MADALINE, on the other hand, is greedy but exact.
- MADALINE utilizes ADALINE to update neuron parameters
- MADALINE is simply ADALINE, when it utilizes parallel computation
A: ADALINE is used to train individual neurons, while MADALINE is used to  train the entire network; MADALINE utilizes ADALINE to update neuron parameters

Q: Choose all that are correct:
- A learning algorithm can scale the weights to change the output of a neuron with sigmoid activation to be more/less linear with respect to the input.
- The output of a neuron has a probabilistic interpretation when using sigmoid activations.
- Gradient descent cannot be used with threshold activations.
- A network with linear activation functions is equivalent to a single perceptron with linear activation
A: A learning algorithm can scale the weights to change the output of a neuron with sigmoid activation to be more/less linear with respect to the input.; The output of a neuron has a probabilistic interpretation when using sigmoid activations.; Gradient descent cannot be used with threshold activations.; A network with linear activation functions is equivalent to a single perceptron with linear activation

Q: We find the minimum point of a function f(x), that is twice differentiable and defined over the reals by:
- Computing the derivative and solving for zero
- Computing the second derivative and solving for zero
- Computing the second derivative f”(x) and find a positive x when f’(x) = 0
- Computing the second derivative f”(x) and find an x where f”(x) is positive and f’(x) = 0
A: Computing the derivative and solving for zero; Computing the second derivative f”(x) and find an x where f”(x) is positive and f’(x) = 0

Q: Gradient descent steps will always result in a decrease in the loss function we are minimizing (true or false)
- True
- False
A: False

Q: A 2-layer MLP operating on 2-dimensional inputs has 2 hidden neurons and one output neuron. All neurons use threshold activations as given below:

The first hidden neuron computes: \(z_{1}=w_{11} x_{1}+w_{12} x_{2}+b_{1}\),   where \(w_{11} = 1, w_{12} = 1, b_1 = -1.1\), and the perceptron uses the thresholding classification rule \(P_{1}=(z_{1}>=0)?\)

The second hidden neuron computes: \(z_{2}=w_{21}x_{1}+w_{22}x_{2}+b_{2}\), where  \(w_{21} = 1, w_{22} = -1, b_2 = 1.1\), and the perceptron uses the thresholding classification rule \(P_{2}=(z_{2}>=0)?\)

And the final output neuron computes: \(z_{3}=w_{31} P_{1}+w_{32} P_{2}+b 3\), where \(w_{31} = w_{32} = 1, b_3 = -2\), and the perceptron uses the thresholding classification rule \(Y=(z_{3}>=0)?\)

We are given a specific training input \(X = (1, 1), ~~d = 0,\) using which the weights must be updated. Using the MADALINE update rule,  the weights for which of the neurons will be updated first?
- The output neuron \(O\)OO
- The first hidden neuron \(P_1\)P 1P 1
- The second hidden neuron \(P_2\)P 2P 2
- None of the neurons would be updated simultaneously because this point is correctly classified
- All the neurons would be updated simultaneously because this point is incorrectly classified
A: The output neuron \(O\)OO