To add another realistic problem to the comparison, we included one other planning application to the set of test domains: generating cases to test a software interface. Because of the similarities between software interface test cases and plans, we developed a system, several years ago, for automatically generating interface test cases using an AI planner. The system was designed to generate test cases for the user interface to Storage Technology's robot tape library [Howe et al.1997]. The interface (i.e., the commands in the interface) was coded as the domain theory. For example, the mount command/action's description required that a drive be empty and had the effect of changing the position of the tape being mounted and changing the status of the tape drive. Problems described initial states of the tape library (e.g., where tapes were resident, what was the status of the devices and software controller) and goal states that a human operator might wish to achieve.
At the time, we found that only the simplest problems could be generated using the planners available. We included this application in part because we knew it would be a challenge. As part of the test set, we include three domain theories (different ways of coding the application involving 8-11 operators) and twenty-four problems for each domain. We included only 24 because we wanted to include enough problems to see some effect, but not too many to overly bias the results. These problems were relatively simple, requiring the movement of no more than one tape coupled with some status changes, but they were still more difficult than could be solved in our original system.