Title: Quiz 4
Course: F19 Introduction to Deep Learning

Q: (Select all that apply) The following piecewise linear function with discontinuities at \(\left(-1,0\right)\) and \(\left(1,0.5\right)\) is used as an activation for a neuron. The slope of the last segment is \(40°\) with respect to the \(z\) axis (going anti-clockwise). Our objective is to find a \(z\) that minimizes the divergence \(div\left(y,d\right)\). Which of the following update rules is a valid subgradient descent update rule at \(z=1\)? Here is the step size and is a positive number. The superscript on \(z\) represents the step index in an iterative estimate. The derivative, \(\frac{\partial div(y,d)}{\partial y}\) is computed at \(z_k=1\). The value of \(\eta\) must not factor into your answer (i.e. remember that \(\eta\) has only been included in the equations for completeness sake and do not argue with us that you can always adjust to make any answer correct ☺).
- \(z^{k+1} = z^k+\eta \frac{\partial d(y,d)}{\partial y}\)
- \(z^{k+1} = z^k-\eta \frac{\partial d(y,d)}{\partial y}\)
- \(z^{k+1} = z^k-\eta 0.75 \frac{\partial d(y,d)}{\partial y}\)
- \(z^{k+1} = z^k-\eta 0.25 \frac{\partial d(y,d)}{\partial y}\)
- \(z^{k+1} = z^k-\eta 0.1 \frac{\partial d(y,d)}{\partial y}\)
A: \(z^{k+1} = z^k-\eta 0.75 \frac{\partial d(y,d)}{\partial y}\); \(z^{k+1} = z^k-\eta 0.25 \frac{\partial d(y,d)}{\partial y}\)

Q: (True/False) Batchnorm always has to be applied before the activations.
- True
- False
A: False

Q: We are given the following relationship

\(y=f\left(x,\:g\left(x,c\right)\right)\:g\left(x,\:f\left(x,\:d\right)\right)\)
where \(y\) is a scalar \(x\) is a vector, and \(g\left(.\right)\) outputs a column vector and \(f\left(.\right)\) outputs a row vector.  Which of the following figures is the influence diagram for \(y\) as a function of \(x\).

Option ( i ):

Option ( ii ):

Option ( iii ):

Option ( iv ):

Option ( v ):
- ( iii )
- ( i )
- ( ii )
- ( iv )
- ( e )
A: ( iii )

Q: (Select all that apply) We are given the relationship

\(y=f\left(x,\:g\left(x\right),\:h\left(x,\:g\left(x\right)\right)\right)\)
Which of the following correctly specify the formula for the derivative of \(y\) w.r.t \(x\)? You may find it useful to draw the influence diagram. Note that partial derivatives are denoted using \(\partial\) and full derivatives are denoted using \(d\). More than one answer may be correct.

( i ). \(\frac{dy}{dx}=\frac{\partial y}{\partial x}+\frac{\partial y}{\partial g}+\frac{\partial y}{\partial h}\)

( ii ). \(\frac{dy}{dx}\:=\:\frac{\partial y}{\partial x}+\frac{\partial y}{\partial g}\frac{dg}{dx}+\frac{\partial y}{\partial h}\left(\frac{\partial h}{\partial x}+\frac{\partial h}{\partial g}\frac{dg}{dx}\right)\)

( iii ). \(\frac{dy}{dx}=\frac{\partial y}{\partial x}+\frac{\partial y}{\partial g}\frac{\partial g}{\partial x}+\frac{\partial y}{\partial h}\frac{dh}{dx}\)

( iv ). \(\frac{dy}{dx}=\frac{\partial y}{\partial x}\frac{\partial h}{\partial g}\frac{dg}{dx}\)

( v ). \(\frac{dy}{dx}=\frac{\partial y}{\partial x}+\frac{\partial y}{\partial g}\frac{dg}{dx}+\frac{\partial y}{\partial h}\frac{dh}{dx}\)
- ( i )
- ( ii )
- ( iii )
- ( iv )
- ( v )
A: ( ii ); ( v )

Q: For a general convex function with a real-valued input, when applying gradient descent, the optimal learning rate:
- Is equal to the inverse of the second derivative.
- Is the same in every point when the function is quadratic
- Always leads you directly to the solution
- Leads you directly to the solution if the function is quadratic
A: Is equal to the inverse of the second derivative.; Is the same in every point when the function is quadratic; Leads you directly to the solution if the function is quadratic

Q: For multivariate functions, the convergence of gradient descent :
- Is linear for quadratic functions
- Is exponentially fast for quadratic functions
- Is linear for lipschitz-smooth functions
- Converges to the solution like the inverse of the number of iterations for lipschitz-smooth functions
A: Is linear for quadratic functions; Is exponentially fast for quadratic functions; Converges to the solution like the inverse of the number of iterations for lipschitz-smooth functions

Q: Several researchers separately decide to estimate an unknown function \(d(x)\), for a variable \(x\). Although they do not know the function, they do have access to an oracle who does know \(d(x)\). Upon demand the oracle will randomly draw a value \(x\) from a Gaussian probability distribution \(P\left(x\right)=Gaussian\left(\mu=0,\:\sigma^2=100\right)\)

and return \(\left(x,\:d\left(x\right)\right)\), i.e. a random value of \(x\) and the value of the function \(d\left(x\right)\) at that \(x\). Each of the researchers independently obtains 1000 training pairs from the oracle, and begins to estimate\(d(x)\) from them. They begin with an initial estimate of \(d(x)\) as \(y\left(x\right)=0\) (where \(y(x)\) is the estimate of \(d(x)\)). They plan to process their training data using stochastic gradient descent. So each of them computes the \(L2\) divergence (as defined in the last lecture) between the estimated output and the desired output for each of their training instances. They do not update \(y\left(x\right)\) during this exercise. In order to get a better handle on their problem, the researchers then get together and pool their divergences over all of their combined training instances, and compute the statistics of the collection of divergences. Unknown to them (but known to the oracle), \(d\left(x\right)=\sqrt{x}\). What would you expect the variance of the set of divergences to be?
- 25
- 0.250
- 0.025
- 2.500
- 0.225
- 2.250
- 0.003
A: 25

Q: You are trying to minimize the \(L2\) divergence (as defined in the last lecture) of the ReLU function \(y=\max\left(0,\:0.5w\right)\)with respect to parameter \(w\),  when the target output is \(1.0\).  Note that this corresponds to optimizing the RELU  function \(y=\max\left(0,\:wx\right)\) given only the training input \(\left(x,\:dx\right)=\left(0.5,\:1\right)\).   You are using Nestorov’s updates.  Your current estimate (in the k-th step) is w(k)=1. The last step you took was \(w^{\left(k\right)}=0.5\). Using the notation from class, you use \(\beta=0.9\) and \(\eta=0.1\). What is the value of \(w^{\left(k+1\right)}\) when using Nestorov’s update? Truncate the answer to three decimals (do not round up).
- 1.463
- 1.457
- 1.393
- 1.469
A: 1.463