Title: Quiz 3
Course: F19 Introduction to Deep Learning

Q: (True/False) Backpropagation can be applied to any differentiable activation function.
- True
- False
A: True

Q: (True/False) A single perceptron will always learn the weights needed to correctly classify all data if:

1) it has data that are linearly separable,
2) it has sigmoid (logistic) activation,
3) it is trained with gradient descent and optimal learning rate, and
4) it is trained with L2 loss.
- True
- False
A: False

Q: Which of the following is true given the Hessian of a scalar function with multivariate inputs?
- The eigenvalues are all strictly positive at a local minimum
- The eigenvalues are all strictly negative at a local maximum
- The eigenvalues are all strictly positive at global minima, but not at local minima
- The eigenvalues are all non-negative at local minima
A: The eigenvalues are all non-negative at local minima

Q: (Select all that apply) At any point, the gradient of a scalar function with multivariate inputs…
- Is in the direction of steepest ascent
- Is in the direction of steepest descent
- Is the vector of local derivatives w.r.t. all the inputs
- Is parallel to equal-value contours of the function
A: Is in the direction of steepest ascent; Is the vector of local derivatives w.r.t. all the inputs

Q: (Select all that apply) When training a neural network…
- Its objective is to minimize the expected divergence between the true and desired outputs of the network
- It minimizes an empirical estimate of the expected divergence between the true and desired outputs of the network on a training set
- It minimizes the empirical risk to zero that results in an exact fit to the target function if and only if the network’s architecture has enough parameters
- We employ gradient descent to train the network
A: Its objective is to minimize the expected divergence between the true and desired outputs of the network; It minimizes an empirical estimate of the expected divergence between the true and desired outputs of the network on a training set; We employ gradient descent to train the network

Q: (Select all that apply) Which of the following is true of the vector and scalar versions of backpropagation?
- Scalar backpropagation is required for scalar activation functions, while vector backpropagation is essential for vector activation functions
- Scalar backpropagation uses the chain rule, while vector backpropagation uses vector arithmetic
- Both scalar backpropagation and vector backpropagation only differ in their arithmetic notation and the implementation of their underlying arithmetic
- Both scalar backpropagation and vector backpropagation are optimization algorithms that are used to find parameters that minimize a loss function
A: Scalar backpropagation uses the chain rule, while vector backpropagation uses vector arithmetic; Both scalar backpropagation and vector backpropagation only differ in their arithmetic notation and the implementation of their underlying arithmetic

Q: (Select all that apply.) Which of the following update rules explicitly computes second-order derivatives or their approximations?
- Gradient descent
- RProp
- Newton’s Method
- Quickprop
A: Newton’s Method; Quickprop

Q: (Select all that apply.) What are the challenges of using Newton’s method with neural networks?
- It cannot find the minimum in any quadratic function
- It is difficult to compute the inverse of the Hessian
- It has a very large Jacobian
- Its memory usage scales with the square of the number of weights
- It can produce unstable updates if the optimized function is not strictly convex.
A: It is difficult to compute the inverse of the Hessian; Its memory usage scales with the square of the number of weights; It can produce unstable updates if the optimized function is not strictly convex.

Q: (True/False) Gradient descent yields a solution that is not sensitive to how a network’s weights are initialized.
- True
- False
A: False

Q: (Select all that apply) Consider the class of twice differentiable functions unless otherwise specified. Which of the following are true when minimizing a function using gradient descent?
- It will converge quickly if the step size is twice the inverse of the second derivative of the function at the current point.
- It will diverge if the step size is more than twice the optimal step size for the quadratic approximation of the function at the current point.
- It has an optimal step size equal to the inverse of the second derivative of the function for the quadratic approximation of the function at the current point.
- It will converge to the optimum while oscillating if the step size is less than the inverse of the second derivative of the function
- It will converge to the optimum monotonically and without oscillating if the step size is less than the inverse of the second derivative of the function
A: It will diverge if the step size is more than twice the optimal step size for the quadratic approximation of the function at the current point.; It has an optimal step size equal to the inverse of the second derivative of the function for the quadratic approximation of the function at the current point.; It will converge to the optimum monotonically and without oscillating if the step size is less than the inverse of the second derivative of the function