Solutions in… Automated Quality Reporting

Reproducibility is at the heart of scientific research – or at least it should be. Not only does it prove the rigor of research, it lays the foundations for the transparency and credibility of projects conducted. But there is growing concern about the number of studies with irreproducible results. Digital Science portfolio company, Ripeta, has developed an AI tool that scans manuscripts for the proper reporting of scientific method components to detect and predict the reproducibility of scientific research. Here, we speak to Leslie McIntosh, CEO of Ripeta, to find out how such tools can help advance scientific research.

What issues are caused by a lack of reproducibility in scientific papers?

Non-reproducible research questions the science. It is actually intriguing on one hand, as there may be something in the scientific process that has not been identified, such as variables that haven’t been identified or taken into account, that positively influence the science. However, far too many variables exist that could influence research outcomes. Ultimately, having reproducible research builds (or weakens) trust in the scientific work. Effective reproducibility lies at the heart of the scientific method.

Is the scientific community doing enough to support the quality of its research and reporting?

The scientific community is an ecosystem made up of stakeholders from varying backgrounds. Researchers, funders, institutions, and publishers must take some responsibility in improving the reporting of research. However, no single stakeholder is fully responsible for making a change. Improvement in research quality will come when multiple actors in this network decide to make changes – including aligning incentives for researcher promotion with conducting and reporting better quality research. 

Lastly, good research should be a commitment towards transparency even when accessibility of all pieces is not possible. As mentioned in our “Making Science Better” report (1), if we start from a simple construct of what good research is, then it starts with a well defined study objective or hypothesis.

What’s next for this project?

There are a few exciting things in the works. One is to expand what we have automated to include other variables and to also provide contextual feedback to help authors make improvements with less work. For example, if they stated their software but don’t cite it, the tool can offer a suggested citation or point towards how to find the citation. In addition to analyzing one paper, we are currently summarizing the results across a grouping (for example, multiple articles from one journal, research topic, company) and creating reports. The aggregated data will eventually be displayed through a dashboard for easier viewing.