AI Seminar sponsored by Apple

Ivan Stelmakh -- PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review

Abstract: In this talk, we will focus on automated assignment of papers to reviewers in conference peer review. In the first part of the talk we will show that assignment procedure currently employed by NeurIPS and ICML does not guarantee fairness. Optimizing the total quality of the assignment over all papers, this procedure may discriminate against some submissions. In contrast, we will present the assignment algorithm that maximizes the review quality of the most disadvantaged paper, thus ensuring fairness. In the second part of talk, we will discuss the parallel objective of accuracy and show that under standard statistical model, our algorithm leads to the minimax optimal accuracy of the final decisions. Finally, we will present a novel experiment that allows for an objective comparison of various assignment algorithms in terms of both fairness and accuracy, and overcomes the inherent difficulty posed by the absence of a ground truth in experiments on peer-review. The results of this experiment corroborate the theoretical guarantees of our algorithm.