Results of Analysis

The table in Figure 27 shows the significant scaling differences that we found between pairs of fully-automated planners at each of the levels. Figure 28 shows the relative scaling of the hand coded planners. In both sets of tests, two planners could only be compared at the levels at which both competed, and on domains in which they both agreed were either easy, hard, or neither easy nor hard. We report the results so that the planner indexed by row is the one showing the superior scaling behaviour. Where planners did not compete in the same tracks we indicate this with the symbol $\bigotimes$ denoting incomparable. Where no significant difference in scaling was found we indicate this with a zero correlation. Where no agreement was found to support a comparison we use the symbol $\bigodot$ denoting disagreement. To avoid duplication of data, we place entries as positive correlations only in the cell corresponding to the row for the planner favoured by the comparison and omit the corresponding negative correlation in the cell for which row and column planners are reversed.

Figure 27: Table showing correlation values, for fully-automated planners, between problem difficulty and difference in time performance, indicating scaling behaviour. $\bigotimes$ means that one of the pairs of planners did not produce data so no comparison may be drawn. $\bigodot$ means that there was insufficient agreement between the planners on the difficulty of domains or the ranking of problems in order to carry out a comparison.

Figure 28: Table showing correlation values, for hand-coded planners, between problem difficulty and difference in time performance, indicating scaling behaviour. $\bigotimes$ means that one of the pairs of planners did not produce data so no comparison may be drawn.