Planners

Publicly available, general-purpose planners tend to be large programs developed over a period of years and enhanced to include additional features over time. Thus, several versions are likely to be available, and those versions are likely to have features that can be turned on/off via parameter settings.

When authors release later versions of their planning systems, the general assumption is that these newer versions will outperform their predecessors. However, this may not be the case in practice. For instance, a planner could be better optimized toward a specific class of problem which then in turn hurts its performance on other problems. Also, advanced capabilities, even when unused, may incur overhead in the solution of all problems.

So for comparison purposes, should one use the latest version? First, we tested this question in a study comparing multiple versions of four of the planners. Second, each planner relies on parameter settings to tune its performance. Some, such as blackbox, have many parameters. Others have none. Comparisons tend to use the default or published parameter settings because few people usually understand the effects of the parameters and tuning can be extremely time consuming. So does this practice undermine a fair comparison?