AI Reproducibility Crisis

In 2020, a team of researchers in India found that they could distinguish between COVID-infected and non-infected individuals by having an AI analyze the patients’ chest x-rays.

Human doctors were not able to distinguish between infected and non-infected people using these scans, so this was seen as a pretty big deal and the paper in which they outlined their machine learning-based technique has since been cited nearly 950 times.

Unfortunately, a fresh look at this research the following year found that the original AI probably wasn’t differentiating between chest-scan data, but instead looking at the backgrounds of the images provided; it was able to distinguish between different sorts of patients with better-than-luck accuracy, in other words, because of a distinction in how different sorts of patients were x-rayed, not because it was seeing something humans couldn’t in the x-rays themselves (the team behind that original research has contested this followup research, but hasn’t provided new data that definitively backs their claims).