Problem Assumption 3: Does Performance Depend on PDDL Requirements Features?

The planners were all intended to handle STRIPS problems. Some of the problems in the test set claim to require features other than STRIPS; one would expect that some of the planners would not be able to handle those problems. In addition, those planners that claim to be able to handle a given feature may not do as well as other planners. Table 6 shows the effects of feature requirements on the ability to solve problems. The data in this table are based on the features specified with the :requirements list in the PDDL definition of the domain.

Table 6: The number of problems claiming to require each PDDL feature solved by each planner.

Planner	Feature
	Axioms	Cond. Eff.	Dis. Pre.	Equality	Exists Pre.	Safety	Strips	Typing	Forall Pre.
A	0	0	0	35	0	0	255	8	0
B	0	0	0	8	0	0	268	0	0
C	5	169	165	216	163	2	561	197	160
D	3	164	166	196	139	0	279	180	139
E	1	162	152	199	157	0	384	168	149
F	0	157	145	185	150	0	376	165	145
G	0	0	0	8	0	0	276	0	0
H	0	0	0	46	0	0	285	17	0
I	0	138	138	169	138	0	441	139	138
J	0	130	130	160	130	0	502	130	130
K	0	0	0	8	0	0	240	0	0
L	0	19	13	24	16	0	421	13	13
M	0	168	169	212	149	2	372	180	151

We did not verify that the requirements were accurate or necessary; thus, the problem may be solvable by ignoring a part of the PDDL syntax that is not understood, or the problem may have been mislabeled by its designer. This is evident in cases where a planner that does not support a given feature still appears to be able to solve the corresponding problem. Some planners, e.g., older versions of STAN, will reject any problem that requires more than STRIPS without trying to solve it; an ADL problem that only makes use of STRIPS features would not be attempted.

As guidance on which planner to use when, these results must be viewed with some skepticism. For example, it would appear based on these results that planner I might be a good choice for problems with conditional effects as it was able to solve many of these problems. This would be a mistake, since that planner cannot actually handle these types of problems. In these cases, the problems claim to require ADL, but in fact, they only make use of the STRIPS subset.

Clearly, certain problems can only be solved by specific planners. For instance, C and M are the only planners that are able to handle safety constraints, while based on the data, only C, D and E appear to handle domain axioms. About half the planners had trouble with the typed problems. Some of the gaps appear to be due to problems in the translation to native representation.