JCDL 2021 Tutorial:

Systemic Challenges and Solutions on Bias and Unfairness in Peer Review

Date and time:
Eastern Time: September 27, 10am-1pm and 7pm-10pm
Central time: September 27, 9am-noon and 6pm-9pm
Central European Summer Time: September 27, 3pm-6pm and September 28, 12.01am-3am
Indian Standard Time: September 27, 6.30pm-9.30pm and September 28, 3.30am-6.30am

Presenter: Nihar B. Shah

Abstract: Peer review is the backbone of scientific research and determines the composition of scientific digital libraries. Any systemic issues in peer review – such as biases or fraud – can systematically corrupt the resulting scientific digital library as well as any analyses on that library. They also affect billions of dollars in research grants made via peer review as well as entire careers of researchers. The tutorial will discuss various systemic issues in peer review via insightful experiments, several computational solutions proposed to address these issues, and finally a number of important open problems.

Slides: link
Detailed writeup and references: link

Topics:

• Prologue (Sections 1 and 2 of writeup)
• Noise (Section 3 of writeup)
• Fraud
  • Lone wolf (Section 4.1 of writeup)
  • Coalitions (Section 4.2 of writeup)
• Miscalibration (Section 5 of writeup)
• Subjectivity
  • Commensuration bias (Section 6.1 of writeup)
  • Confirmation bias (Section 6.2 of writeup)
  • Dr. Fox effect (Section 6.3 of writeup)
  • Hindering novelty (Section 6.4 of writeup)
  • Interdisciplinary research (Section 6.5 of writeup)
• Bias pertaining to author identities (Section 7 of writeup)
• Norms and policies
  • Author incentives (Section 8.1 of writeup)
  • Review quality (Section 8.2 of writeup)
  • Author rebuttal (Section 8.3 of writeup)
  • Discussions and group dynamics (Section 8.4 of writeup)
• Epilogue (Section 9 of writeup)