Problem Assumption 2: How Do Syntactic Representation Differences Affect Performance?

Many studies, including this, have shown that planners may be sensitive to representational features. Just because representations can be translated automatically does not mean that performance will be unaffected. Just because an algorithm should theoretically be insensitive to a factor does not mean that in practice it is. All of the planners showed some sensitivity to permuted problems, and the degree of sensitivity varied. This outcome suggests that translators and even minor variations on problem descriptions impact outcome and should be used with care, especially when the sensitivity is not the focus of the study and some other planner is more vulnerable to the effect.

Recommendation 6: Representation translators should be avoided by using native versions of problems and testing multiple versions of problems if necessary.

With many planner developers participating in the AIPS competitions, this should become less of an issue.

More importantly, researchers should be explicitly testing the effect of alternative phrasings of planning problems to determine the sensitivity of performance and to separate the effects of advice/tuning from the essence of the problem.

Recommendation 7: Studies should consider the role of minor syntactic variations in performance and include permuted problems (i.e., initial conditions, goals, preconditions and actions) in their problem sets because they can demonstrate robustness, provide an opportunity for learning and protect developers from accidentally over-fitting their algorithm to the set of test problems.