Results

The planning competition offers a great opportunity for assessing the relative performance of various techniques used in planning over a wide range of problems. Inevitably there will, however, be features that are not tested by the set of domains used in the competition. There will also be some domains in which many of the features of a planner collaborate to produce good results, rather than the results being directly attributable to one individual feature. Here we discuss the results from the competition and present further results to clarify which of the features of Marvin contribute to the performance in each particular case.

It is important to note that when we refer to macro-actions generated and used by Marvin these are all generated during the planning process for that specific problem. No additional learning time or knowledge gained from solving other problems was used by Marvin in the competition, or in producing the additional results presented in this paper. Although some planners can use additional `learning time' when solving a series of problems, a satisfactory way to incorporate this extra time into the time taken to solve each problem, as measured in the planning competition, has yet to be found. In the planning competition the planners are compared based on their performance on isolated problem instances, which is still an interesting comparison to make.

The results presented were produced on two machines: a machine at the University of Strathclyde (with a 3.4GHz Pentium 4 processor) and the IPC 4 competition machine (with a 3GHz Xeon processor). In both cases, the planner was subjected to a 30 minute time limit and a 1Gb memory usage limit. All results that are directly compared against each other (i.e. appear on the same graph) are produced on the same machine. The domains used for evaluation are taken from IPC 3 and IPC 4, and are described in detail in the papers giving an overview of each of the two competitions [Long & Fox 2003, Hoffmann & Edelkamp 2005].