Analytic Framework

The t-tests are performed using the normalised performances of the two planners. They find the magnitude of difference between the performances of two planners, $p^{1}$ and $p^{2}$, on a collection of problem instances. For example, given a collection of $n$ problems, we find the pairs of results $p^{1}_{r_{i}}$ and $p^{2}_{r_{i}}$ obtained for instances $i=1$ to $i=n$. In each case we normalise these values by dividing each of them by the mean of the pair. This process establishes a range of performances between 0 and 2, with 1 standing for equal performance. The t-test results in a t-value representing the difference in the mean normalised performances of the two planners. We perform 2-tailed tests at $p < 0.05$ because we are interested in the individual results rather than in extrapolated partial orderings based on magnitude. We want to consider the magnitude information as it is relevant to individual consistency results.