Reinforcement Learning Methods for Military Applications

Malcolm Strens - Visiting CMU from the Defence Evaluation & Research Agency, U.K.

Abstract

My research aims to identify the military applicability of reinforcement learning (RL), and develop appropriate new algorithms. Potential applications include:

• Planning, control and decision-making in autonomous systems (e.g.  unmanned airborne vehicles, autonomous land vehicles, and guided weapons).
• Automation in manned systems. (e.g. control of imaging sensors for search and track, low level flight control, collision avoidance).
• Electronic warfare (e.g. when to transmit, which frequencies to choose, and when to stay silent).
• Logistics, scheduling, and tactical planning.
• Many more...

I will give a brief overview of this diverse range of applications, emphasising the role that high fidelity simulation has to play in making RL feasible. Then I will focus on a particular multi-pursuer evader problem. Various RL methods were applied, including Q-learning, model-based methods (certainty-equivalent and Bayesian), and direct policy search (Pegasus, and proposed alternatives). An important focus is the relationship between the RL agent and the simulation: the agent has complete control over the simulation, able to restart it in any state, observe its hidden state during learning, and control the random number sequence. This tight control can be used to accelerate learning.