Title: Quiz 13
Course: F19 Introduction to Deep Learning

Q: Given is the diagram of a Markov Decision Process. As shown on the diagram, there is only one choice to be made: left or right. The numbers show the rewards that are received deterministically after each action. γ = 0. We are using the same convention as in the slides in the diagram (actions are black circles and values white circles). Find the optimal policy
- Always choose left
- Always choose right
- Left and right are equally good actions, choose either with P = 0.5
- Choose left in 2 out of 3 times
A: Always choose left

Q: (True/False) GANs are pretty good at generating pictures, but it’s hard for a GAN to generate sentences. One of the reason is that image colors(RGB) are effectively continuous variables but words in language are discrete variables. One possible solution is that we can treat generate words as states and treat generating a real sentence as our goal, then we can use reinforcement learning to train the GAN.
- True
- False
A: True

Q: (True/False) The state value function and the action value function depend on the policy. But the optimal state value function and the optimal action value function do not depend on the policy.
- True
- False
A: True

Q: Which of the following statements are true about the discount factor in an MDP?
- If we set discount factor to 0, we only consider the reward at the current state.
- If we set discount factor to 0, we only consider the reward at the final state in the sequence.
- If we set discount factor to 1, we consider all of the future rewards.
- If we set discount factor to 1, we consider all of the past rewards.
A: If we set discount factor to 0, we only consider the reward at the current state.; If we set discount factor to 1, we consider all of the future rewards.

Q: Consider the spider-fly model, if the spider catches the fly, it will get a reward 1. There are no penalties for wandering around the corners. What are the expected returns at \(c_4\) and \(c_8\)?
- \(c_4=0;\:\:c_8=0\)
- \(c_4=0;\:\:c_8=1\)
- \(c_4=1;\:\:c_8=0\)
- \(c_4=1;\:\:c_8=1\)
A: \(c_4=1;\:\:c_8=1\)

Q: Which of the following is true about the benefits of using a discount factor γ in an MDP or MRP?
- The discounted return is well defined for infinite horizon processes.
- It assigns a greater weight to earlier rewards for infinite horizon processes.
- It is the only way of ensuring a finite (discounted) return at every state.
- It is required for us to be able to write the Bellman expectation equations (as given in the slides)
A: The discounted return is well defined for infinite horizon processes.; It assigns a greater weight to earlier rewards for infinite horizon processes.; It is required for us to be able to write the Bellman expectation equations (as given in the slides)

Q: Which of the following statements are true about the Bellman Optimality Equation?
- The solution to the Bellman Optimality Equation depends on the current policy.
- The solution to the Bellman Optimality Equation requires knowledge of state-action values.
- The solution to the Bellman Optimality Equation requires knowledge of the optimal state values.
- The solution to the Bellman Optimality Equation requires knowledge of the MDP.
A: The solution to the Bellman Optimality Equation requires knowledge of the MDP.

Q: Which of the following is true of returns and rewards.
- The reward at any time t is the immediate reward resulting from the action at that time.
- The return at any time t is the cumulative reward from that time into the entire future.
- The return at any time t is the cumulative reward from the start of the process until that time.
- Rewards can be computed without discounting, but discounting is always required to compute returns.
A: The reward at any time t is the immediate reward resulting from the action at that time.; The return at any time t is the cumulative reward from that time into the entire future.