The EA was evaluated with respect to the usefulness of its output, frequency of unwanted alerts, and real-time performance with realistic data streams. SAIM and the EA were tested against data produced by a high-fidelity military simulator on two scenarios. The simulator has detailed models of each type of vehicle and sensor. One scenario lasted 13.5 hours, but only the last 90 minutes were simulated at high fidelity. (The first 12 hours had a file of scripted events, with a few dozen tracks and reports.) The second scenario, on the same terrain, was simulated for 20 minutes. The 90-minute simulation had more than 45,000 events passed to the Watchman from SAIM, of which 13,000 were passed on to the EA Manager, which monitors only to the squad level (8 to 10 entities). During the simulator run, scripted events also simulate messages from any team members that are not running live (such as messages confirming mission starts and completions). The high fidelity of the simulation provides realistic data rates and inputs, thus providing some evidence indicating that the EA will perform as desired in the real world.
Our formal evaluation ran live with SAIM, the simulator, and several team members, each running their own copy of SAIM and the EA on different physical machines. For a shorter development cycle, we implemented an event generator that reproduces SAIM behavior, making the SAIM network unnecessary. The event generator creates messages from files of scripted events that include confirmations of mission starts and completions (that normally would come from a subordinate EA), orders for aborting the current plan or executing a new plan (that normally would come from a superior EA), and sensor tracks and location reports (that normally would come from the SAIM network). Our event scripts contain all messages captured from a run that included the simulator and SAIM.