Analysis of 70,000 Studies Reveals Preprints Are More Reliable Than Perceived

For years, the scientific community has debated the reliability of preprints – research papers posted online before formal peer review. However, a recent comprehensive analysis of over 70,000 biomedical studies suggests that these preliminary publications are far more dependable than commonly perceived. The study, posted on the preprint server bioRxiv, reveals that the central conclusions of biomedical preprints rarely undergo significant alteration following peer review, and surprisingly, preprints are retracted at roughly half the rate of papers that skip this initial online posting. The research, led by neuroscientist Ruslan Rust from the University of Southern California, aimed to challenge the widespread skepticism among scientists regarding preprint reliability. Rust, who often observed that peer review seldom led to major changes in his own field, sought to quantify this across a broader spectrum of biomedical research. Utilizing a large language model (LLM), Rust and his colleague meticulously extracted the main scientific conclusions from the abstracts of 72,644 biomedical manuscripts initially uploaded to bioRxiv between 2018 and 2025. The LLM then compared these conclusions with their eventual peer-reviewed versions. The findings are compelling: nearly 40% (39.9%) of the main conclusions remained entirely unchanged between the preprint and its journal-published abstract, while an additional 50% underwent only minor revisions. This means a substantial 90% of preprints saw their core conclusions either preserved or only slightly tweaked. A smaller proportion, just over 10%, experienced major changes. Interestingly, when changes did occur, they predominantly leaned towards more cautious language, with 8.4% adopting a more reserved tone compared to 4.2% using more confident wording after peer review. The extent of these revisions also varied by discipline, with major changes affecting only 7.2% of bioinformatics papers versus 17.5% of microbiology studies. The study also observed a notable trend: the frequency of major revisions declined significantly over time, dropping from 17% for papers posted in 2019 to merely 5.7% in 2024. Julian Sienkiewicz, an expert in AI tools, suggests this decline might indicate an overburdened peer-review system. Conversely, Rust posits that it reflects an evolving use of preprints, particularly influenced by the rapid dissemination demands during the COVID-19 pandemic, where early versions might have required more substantial revisions. He also suggests that more recent preprints might already incorporate preliminary feedback, making them more refined from the outset. Perhaps the most striking finding concerns research integrity: papers initially appearing as preprints were retracted at a rate of 8.1 per 10,000, significantly lower than the 18.7 retractions per 10,000 for comparable papers that bypassed the preprint stage. While the authors caution that this is an observational comparison based on relatively few retractions and does not definitively prove that preprints reduce retraction likelihood, it strongly correlates with increased scrutiny. Some researchers, however, advise caution, noting that preprints might be subject to selection bias, depending on who posts them and which studies are chosen for early dissemination. Nevertheless, this study provides robust evidence challenging the long-held skepticism about preprint reliability, potentially reshaping how the scientific community views early-stage research dissemination.