Performance Metrics

Most comparisons emphasize the number of problems solved and the CPU time to completion as metrics. Often, the problems are organized in increasing difficulty to show scale-up. Comparing based on these metrics leaves a lot open to interpretation. For example, some planners are designed to find the optimal plan, as measured by number of steps in either a parallel or sequential plan. Consequently, these planners may require more computation. Thus, by ignoring plan quality, these planners may be unfairly judged. We also hypothesize that the hardware and software platform for the tests can vary the results. If a planner is developed for a machine with 1GB of memory, then likely its performance will degrade with less. A key issue is whether the effect is more or less uniform across the set of planners.

In this section, we examine these two issues: execution platform and effect of plan quality.