Evaluation Tracks

It was clear from the discussions leading up to the competition that different groups were prioritizing their efforts differently. We wanted to ensure that a diverse set of powerful approaches were recognized and decided to tabulate results in several different ways to acknowledge the value of these different approaches. The six tracks were:

• Overall. This track used a reward-based evaluation criterion for all domains (goal achievement counted as 500 for goal-based domains). Domains: Blocksworld (7 problems), Colored Blocksworld (2), Boxworld (5), Exploding Blocksworld (1), Fileworld (1), Tireworld (2), Towers of Hanoise (1), Zeno Travel (1).

• Goal-based. For this track, we ignored action costs and counted goal achievement as a unit reward (thus emphasizing approaches that maximized the probability of reaching a goal state). The domains and problems used were the same as in the Overall track: Blocksworld (7), Colored Blocksworld (2), Boxworld (5), Exploding Blocksworld (1), Fileworld (1), Tireworld (2), Towers of Hanoise (1), Zeno Travel (1).

• Overall, Non-Blocks/Box. Blocksworld and Boxworld dominated the full set and we wanted to see how subtler problems were handled. Domains: Exploding Blocksworld (1), Fileworld (1), Tireworld (2), Towers of Hanoise (1), Zeno Travel (1).

• Domain-specific. “Domain-specific” allowed human-tuned rules; “Domain-specific, No Tuning” did not (only automatically generated rules specific to the domain were allowed). They were evaluated using the generated domains: Blocksworld (8), Colored Blocksworld (6), Boxworld (5).

• Conformant. Planners in this category had to produce straight-line plans, “blind” to intermediate states encountered. We prepared “unobservable” versions of the domains to evaluate planners in this category. Domains: Blocksworld (7), Colored Blocksworld (2), Boxworld (5), Exploding Blocksworld (1), Fileworld (1), Tireworld (2), Towers of Hanoise (1), Zeno Travel (1).