Abstract
A recurring theme in mathematical software evaluation is the
generalization of rankings of algorithms on test problems to build
knowledge-based recommender systems for algorithm selection. A key
issue is to profile algorithms in terms of the qualitative
characteristics of benchmark problems. In this brief methodological
note, we adapt a novel all-pairs algorithm for the profiling task:
Given performance rankings for m algorithms on n problem instances
each described with p features, identify a (minimal) subset of p that
is useful for assessing the selective superiority of all combinations
of algorithms. We show how techniques presented in the mathematical
software literature are inadequate for such profiling purposes. In
conclusion, we also address various statistical issues underlying the
effective application of this technique.
full paper