Title: Quiz 5
Course: F19 Introduction to Deep Learning

Q: (Select all that apply.) Which of the following statements are true about Lipschitz continuous functions?

You can find useful pointers on Lipschitz functions in these nice notes by Anatoli Louditski:

https://ljk.imag.fr/membres/Anatoli.Iouditski/cours/convex/chapitre_25.pdf

You may also refer to other sources.
- Lipschitz functions are not necessarily differentiable everywhere
- Lipschitz functions are always differentiable.
- Lipschitz functions cannot have sharp angles
- Lipschitz functions can be discontinuous at a finite set of points
- Strongly convex Lipschitz functions can have unbounded derivative
A: Lipschitz functions are not necessarily differentiable everywhere

Q: (Select all that apply.) Which of the following images is Lipschitz continuous on the provided domain?
A: (a choice with no text)

Q: (Select all that apply.) A student applies Rprop to find the minimum of a function and finds, to his surprise, that the updates along a particular component (dimension) are continuously increasing in size, and never change sign. Which of the following are valid conclusions for the student to arrive at?
- Rprop may not be an appropriate algorithm for minimizing this particular function.
- The function is monotonically decreasing along that component.
- The function has no local or global minima.
- The derivative of the function never changes sign along that component.
- The minimum value of the function is minus infinity.
A: Rprop may not be an appropriate algorithm for minimizing this particular function.

Q: Tom, an aspiring young DL enthusiast, observes suddenly that RMSprop normalizes the learning rate by the second moment -- the RMS value -- of the derivatives, rather than the standard deviation. He knows from his statistics coursework that the fluctuations in a random variable’s value are characterized by its variance and not its RMS value.

Eager to publish a brilliant new result, Tom replaces the RMS value of the derivative employed in the RMSprop update rule with the standard deviation. He computes the standard deviation through a running estimate, which he initializes with a value 1.0. He calls his new algorithm STDEVprop.

He is dismayed to find that while RMSprop converges, STDEVprop does not.  In fact, the algorithm quickly takes larger and larger step sizes along the \(x\) direction and quickly diverges.

Which of the following statements are valid conclusions for Tom?
- The function possibly has a derivative of constant magnitude along x, as a result of which the variance goes to 0, resulting in exploding step sizes.
- In general, normalizing by the standard deviation as opposed to the RMS value is not a good idea, since it ignores the mean value of the derivatives, which could be small or even 0.
- It should work, but his code is buggy
- The average derivative along x is 0
- The function is non-convex with steep surfaces
A: The function possibly has a derivative of constant magnitude along x, as a result of which the variance goes to 0, resulting in exploding step sizes.; In general, normalizing by the standard deviation as opposed to the RMS value is not a good idea, since it ignores the mean value of the derivatives, which could be small or even 0.

Q: Which of the following is true in comparing batch descent, SGD, and Mini-batch descent?
- The variance of the estimate (variation from run to run) is greatest for SGD and lowest for batch descent
- Mini-batch descent converges to the optimum quickest (for the least computation) among the three methods (assuming sequential computation)
- In some cases SGD may provide the fastest convergence
- Mini-batch descent typically provides the fastest convergence, if we use parallel processing and vector operations
A: The variance of the estimate (variation from run to run) is greatest for SGD and lowest for batch descent

Q: Using the optimal learning rate with SGD, we will get an optimal solution...
- in the convex case.
- always.
- only in the strongly convex case.
- that is never guaranteed.
A: in the convex case.

Q: What could happen if incremental gradient descent was not stochastic (i.e. if we selected the training points in a constant order) ?
- A function that swings around instead of converging
- The loss value would converge very quickly
- Overfitting
- The behavior would not change
A: A function that swings around instead of converging

Q: Dropout on a network with \(N\) (non-output) neurons is often explained as bagging over all networks that may be composed by randomly selecting each of the non-output neurons with a probability.  By this explanation, which of the following statements must be true (for test data)?
- It theoretically averages the prediction of N separate networks.
- It theoretically averages the prediction of  \(N^2\)N 2N 2N 2  separate networks.
- It theoretically averages the prediction of  \(2^N\)2 N2 N2 N  separate networks.
- It randomly selects    \(\alpha N\)α Nα Nα N   sparate networks and averages their predictions .
A: It theoretically averages the prediction of  \(2^N\)2 N2 N2 N  separate networks.

Q: (Select all that apply.) Which of the following statements are true about Adam optimization.
- The hyperparameters of Adam (learning rate, exponential decay rates for the moment estimates, etc.) are usually set to predefined values (given in the paper), and do not need to be tuned
- Of the optimizers SGD, Momentum, and RMSprop, Adam uses the most memory for a given batch size
- Adam is often the default optimizer in training DL models
- Adam uses different learning rates for each parameter
- Adam uses an adaptive strategy to set the learning rate at each learning step and each parameter, making it very effective at learning the parameters
A: The hyperparameters of Adam (learning rate, exponential decay rates for the moment estimates, etc.) are usually set to predefined values (given in the paper), and do not need to be tuned; Of the optimizers SGD, Momentum, and RMSprop, Adam uses the most memory for a given batch size; Adam is often the default optimizer in training DL models; Adam uses different learning rates for each parameter; Adam uses an adaptive strategy to set the learning rate at each learning step and each parameter, making it very effective at learning the parameters

Q: Statement 1: One benefit of using L2 divergence for softmax is that it is convex with respect to the parameters (weights) of the softmax.

Statement 2: KL divergence bounds the L1-norm from above and  has a single global minimum with respect to the parameters of the softmax.
- Statement 1 is False; Statement 2 is True.
- Statement 1 is True; Statement 2 is False.
- Statement 1 is False; Statement 2 is False.
- Statement 1 is True; Statement 2 is True.
A: Statement 1 is False; Statement 2 is True.