AutoML Decathlon 2022

Competition Description

---

Outline

In AutoML Decathlon, you will develop your submissions - a fully automated ML pipeline that receives the task data and metadata as input, and outputs a well-trained model - on a diverse set of 10 development tasks. Submissions will be evaluated on a set of 10 different test tasks, which will only be released upon the competition's conclusion.

A single aggregate scoring protocol will be used to quantify and rank submissions' performances on the development set. The same protocol will later be used for the test set as well. Additionally, we will impose compute time and hardware constraints on your submissions during this scoring protocol. The compute time will vary by task depending on size and difficulty. You can develop your submissions using the smaller development tasks to minimize costs, and leverage the organizers’ compute resources (see Important Resources) for the larger ones. Details on the scoring protocol and constraints can be found on the CodaLab page.

The competition comprises two phases — a development phase and an evaluation phase.

1. Development Phase; Jul 13, 2022 - Nov 10, 2022
   You will develop your methods using the development tasks provided in the starter kit. Each task will have a set of labelled training data and a set of unlabelled test data. You can submit your code to the competition platform at a capped frequency. Additionally, public leaderboards will be available on CodaLab and updated regularly. There will be one for each development task, and another for aggregate performance determined by the aforementioned scoring protocol.
2. Evaluation Phase; Nov 10, 2022 - Dec 07, 2022
   At the end of the Development Phase, each team’s last code submission will be automatically promoted to the Evaluation Phase. We will first perform a round of inspection and penalize unqualified submissions under the guidelines specified in the Rules. After that, each submission's final score and rank will be produced using the same scoring protocol used in the Development Phase. Finally, the competition winners and overall results will be announced, along with an analysis of the competition.

Prizes

To be considered a winner in the competition, your method must beat the various provided baselines, which are described on CodaLab. Following the conclusion of the Evaluation Phase, $15,000 will be awarded to the winning team. The top performers will also be featured prominently in a paper summarizing the results and key takeaways of the competition that the organizers will write.

To avoid conflicts of interest, all competition organizers are ineligible for prizes.