To model nondeterministic motion by the defender, we set the defender's speed randomly within a range. For each attempt this speed is constant, but it varies from attempt to attempt. Since the agent observes only the defender's initial position, from the point of view of the agent, the defender's motion is nondeterministic.
First, observe that this nondeterminism makes the task more difficult. The following table shows that with either type of memory scheme, the agent performs significantly worse when the defender's speed varies than when it remains constant.
Even with the advantages of twice as many trials in which to learn and of an adaptive memory, the agent does not score as often when the defender's speed varies nondeterministically.
This set of experiments was designed to test the effectiveness of adaptive memory when the defender's speed was both nondeterministic and different from the speed used to train the existing memory. The memory was initialized in the same way as in Section 3.2 (for defender speed 50). We ran experiments using both types of memory in which the defender's speed varied between 10 and 50 or between 60 and 100. We compared agents with trained memories against agents with initially empty memories as shown in Figure 4.
Figure 4: A comparison of the effectiveness of starting with an empty
memory versus starting with a memory trained for a constant defender
speed (50) different from the defender speed during testing. The
performances of both adaptive and basic memories are plotted. Success
rate is measured as goal percentage thus far.
Notice in both cases, the agents with full initial memories outperformed the agents with initially empty memories in the short run. The agents learning from scratch did better over time since they didn't have any training examples from when the defender was moving at a fixed speed of 50 affecting their memories; but at first, the training examples for speed 50 were better than no training examples. Adaptive memory outperformed basic memory in both graphs.
Going back to the soccer scenario, practicing against a particular goalie is better than not practicing at all, even if the opposing goalie moves randomly and differently from the training goalie. When you would like to be successful immediately upon entering a novel setting, adaptive memory allows training in related situations to be effective without permanently reducing learning capacity.