There’s a reproducibility crisis brewing in almost every scientific field that has adopted machine learning. On July 28, we’re hosting an online workshop featuring a slate of expert speakers to help you diagnose and fix these problems in your own research: sites.google.com/princeton.edu/rep-workshop/

Here’s the context. Dozens of scientific fields have adopted the prediction paradigm, and that’s great. But ML performance evaluation is notoriously tricky. At least 20 reviews in 17 fields have found widespread errors, and the list is quickly growing. reproducible.cs.princeton.edu/

In most cases, when the errors are corrected, the scientific claims being made don’t hold up. We think this is a crisis which, left unchecked, will undermine the credibility of the predictive paradigm and the use of ML in the sciences.

The scope of the workshop is about applied ML research, where the goal is to use ML methods to study some scientific question, not ML methods research, for example the typical NeurIPS paper. (That community is also undergoing a reproducibility reckoning.)

Sadly, each scientific field is independently rediscovering these pitfalls, and there is no common vocabulary. We badly need an interdisciplinary exchange of ideas, broader awareness of these pitfalls, and space to brainstorm solutions and best practices.

That’s what our workshop is for. We welcome you to join. Our expert speakers come from many disciplines including sociology, economics, computational social science, and computer science. They’ve each studied ML reproducibility in their respective fields.

We're so grateful to our speakers @b_m_stewart, @Gillesvdwiele, @jakehofman, @JessicaHullman, @m_serra_garcia, @michael_lones, @MominMMalik, Michael Roberts, and Odd Erik Gundersen. The organizers are @sayashk, @priyakalot, Kenny Peng, Hien Pham, and me, along with @princetonsml.

All are welcome to attend. These issues cut across scientific disciplines, so we'd appreciate help in spreading the word in your community. RSVP here to receive the Zoom link: docs.google.com/forms/d/e/1FAIpQLSeSPz9eCOtdKZ1ArESIGDGzqHjWSgQygxTMp2Ec6bJDnLR1kw/viewform