Sixth Report - Reproducibility and Research Integrity

Although qualitative evidence indicates a potentially substantial scale of research integrity issues in the UK, there is a lack of quantitative evidence, including on the relative significance of the different causes of problems. This can only hamper efforts to evaluate … Read more Although qualitative evidence indicates a potentially substantial scale of research integrity issues in the UK, there is a lack of quantitative evidence, including on the relative significance of the different causes of problems. This can only hamper efforts to evaluate damage being caused to the UK research sector in terms of culture, performance, reputation and economic value—now and in the future. This in turn prevents the design of proportionate and effective solutions to any problems. Show less

Though the specific problems faced differ, all research disciplines are affected by the systemic issues that limit reproducibility. The established view that issues are concentrated in the social and medical sciences is outdated and tackling reproducibility challenges should be a … Read more Though the specific problems faced differ, all research disciplines are affected by the systemic issues that limit reproducibility. The established view that issues are concentrated in the social and medical sciences is outdated and tackling reproducibility challenges should be a priority in every research field, regardless of discipline and methodology. Show less

Despite significant delays, we welcome the establishment of UK Committee on Research Integrity (UK CORI) as a potential answer to existing research integrity challenges. We ask UK CORI to commit to produce an annual state of the nation report on … Read more Despite significant delays, we welcome the establishment of UK Committee on Research Integrity (UK CORI) as a potential answer to existing research integrity challenges. We ask UK CORI to commit to produce an annual state of the nation report on research integrity, including reproducibility, issues. These reports should contain a breakdown of the scale of these issues across different disciplines. They should also contain action plans for addressing identified issues. UK CORI should also publish an action plan detailing how it will carry out each of the steps set out in its strategy, including timelines and what resources will be committed for the activities it plans to carry out. Show less

It is disappointing that UK CORI’s recently published strategy did not mention reproducibility, especially since our inquiry highlighted that this is a major research integrity issue. UK CORI should make sure reproducibility challenges are given due attention and not overlooked … Read more It is disappointing that UK CORI’s recently published strategy did not mention reproducibility, especially since our inquiry highlighted that this is a major research integrity issue. UK CORI should make sure reproducibility challenges are given due attention and not overlooked in deference to other pressing research integrity issues. UK CORI should develop a sub-committee to focus on reproducibility challenges in research. This sub-committee should establish the relative weight of reproducibility within research integrity concerns, and UK CORI should use this evidence to plan its prioritisation. Show less

The increased application of artificial intelligence (AI) in scientific research presents a challenge to traditional research methods. The UK Committee on Research Integrity (UK CORI) should specifically: investigate the impact of deploying AI— and other increasingly complex software—on reproducibility in … Read more The increased application of artificial intelligence (AI) in scientific research presents a challenge to traditional research methods. The UK Committee on Research Integrity (UK CORI) should specifically: investigate the impact of deploying AI— and other increasingly complex software—on reproducibility in research; consider whether specialised action is needed; and set out these findings in its annual reports. Show less

Whilst significant reproducibility challenges are faced in research, to refer to the sum of these issues as a “crisis” risks detracting from the many successes of the UK’s scientific research base. Nonetheless, there is need for action to address the … Read more Whilst significant reproducibility challenges are faced in research, to refer to the sum of these issues as a “crisis” risks detracting from the many successes of the UK’s scientific research base. Nonetheless, there is need for action to address the significant problems caused by the prevalence or reproducibility problems in the scientific community. (Paragraph 50) 50 Reproducibility and Research Integrity Solving reproducibility challenges which emerge as research is conducted Show less

The UK’s lacks an established infrastructure for responding to research misconduct cases. The UK Government should lead on a co-produced framework with the UK Reproducibility Network, UKRIO and UK CORI, which sets out the roles and expectations for key actors … Read more The UK’s lacks an established infrastructure for responding to research misconduct cases. The UK Government should lead on a co-produced framework with the UK Reproducibility Network, UKRIO and UK CORI, which sets out the roles and expectations for key actors when cases of misconduct are identified. Show less

The UK Government should assess the benefits that an additional body, set up to investigate malpractice, could bring to the UK’s research integrity governance architecture.

Most reproducibility issues are, in the main, not the result of deliberate bad practice. Many of the incentives faced by individuals conducting research act against reproducibility. Whilst individuals must take responsibility for conducting work which prioritises robust analysis and transparency, … Read more Most reproducibility issues are, in the main, not the result of deliberate bad practice. Many of the incentives faced by individuals conducting research act against reproducibility. Whilst individuals must take responsibility for conducting work which prioritises robust analysis and transparency, and for promoting the importance of reproducibility within their research field, we recognise that aspects of the academic process, such as time pressures and the publication model, act against the promotion of research integrity and reproducibility. Therefore, the research community, including research institutions and publishers, should work alongside individuals to create an environment where research integrity and reproducibility are championed. Show less

Research institutions should model a culture of reproducibility by managing inordinate pressures on academics and encouraging the prioritisation of reproducibility in research outputs. This extends to encouraging openness around mistakes and their correction. In collaboration with the Higher Education Sector, … Read more Research institutions should model a culture of reproducibility by managing inordinate pressures on academics and encouraging the prioritisation of reproducibility in research outputs. This extends to encouraging openness around mistakes and their correction. In collaboration with the Higher Education Sector, Universities UK should implement a coordinated policy on minimum protected research time for research staff. Show less

Statistical experts and software developers are insufficiently recognised and renumerated within the university research sector. Funders and universities should develop dedicated funding for the presence of statistical experts and software developers in research teams. In tandem, universities should work on … Read more Statistical experts and software developers are insufficiently recognised and renumerated within the university research sector. Funders and universities should develop dedicated funding for the presence of statistical experts and software developers in research teams. In tandem, universities should work on developing formalised, aspirational career paths for these professions. Show less

Research funders should implement stronger tests for the presence of adequate software and statistical skills within research teams at the outset of a funding application. Where these skills are perceived to be lacking, UKRI should consider the feasibility and cost … Read more Research funders should implement stronger tests for the presence of adequate software and statistical skills within research teams at the outset of a funding application. Where these skills are perceived to be lacking, UKRI should consider the feasibility and cost of offering a dedicated methodological support system to research teams. Show less

Currently there is insufficient attention placed on reproducibility and research integrity training for university students and research professionals. Greater emphasis should be placed on the importance of reproducibility and research integrity in education and training at undergraduate, postgraduate and early … Read more Currently there is insufficient attention placed on reproducibility and research integrity training for university students and research professionals. Greater emphasis should be placed on the importance of reproducibility and research integrity in education and training at undergraduate, postgraduate and early career researcher stages. Part of this training at the undergraduate and postgraduate levels should include the routine production of replications. Show less

Institutions should incorporate mandatory reproducibility training and professional development plans for researchers across the course of their career. (Paragraph 83) Reproducibility and Research Integrity 51

Short-term research grants place restrictive limitations on researchers, which can be to the detriment of research integrity and reproducibility. UKRI should consult with a representative sample of researchers to understand whether their grants allow them sufficient time and funding to … Read more Short-term research grants place restrictive limitations on researchers, which can be to the detriment of research integrity and reproducibility. UKRI should consult with a representative sample of researchers to understand whether their grants allow them sufficient time and funding to do the work needed for ensuring their research is reproducible. UKRI should also implement a trial funding programme with an emphasis on ‘slower’ science. Show less

Uncertainty in the academic job market, especially at earlier career stages, acts as a strong additional disincentive against the prioritisation of reproducibility by researchers. Research funders, including UKRI, should work to impose a three-year minimum contract for post-doctoral researchers in … Read more Uncertainty in the academic job market, especially at earlier career stages, acts as a strong additional disincentive against the prioritisation of reproducibility by researchers. Research funders, including UKRI, should work to impose a three-year minimum contract for post-doctoral researchers in universities. (Paragraph 89) Solving reproducibility challenges which emerge in the communication of research Show less

The trend towards blanket open access in the communication of scientific outputs is positive. UKRI and other research funders should continue to implement open access policies until this figure reaches 100%, by the end of 2025 at the latest.

Currently, research outputs are frequently published without an associated link through to their open-source data and code. This prevents other researchers assessing work for its reproducibility. In all bar the most exceptional ethical and legal situations, researchers should share their … Read more Currently, research outputs are frequently published without an associated link through to their open-source data and code. This prevents other researchers assessing work for its reproducibility. In all bar the most exceptional ethical and legal situations, researchers should share their research data and code alongside published outputs. Show less

Journals should collectively encourage researchers to employ the FAIR (Findability, Accessibility, Interoperability, and Reuse of digital assets) principles within their research and should mandate the deposition of research data in open- access repositories alongside the publication of research outputs. Read more Journals should collectively encourage researchers to employ the FAIR (Findability, Accessibility, Interoperability, and Reuse of digital assets) principles within their research and should mandate the deposition of research data in open- access repositories alongside the publication of research outputs. Show less

We welcome UKRI’s use of data management plans. A continued emphasis on their importance as a condition of research funding is necessary.

Providing adequate funding for replication studies is an important precondition for ensuring researchers have the resources necessary to conduct them. UKRI should learn from its Dutch equivalent, NWO, by developing a pilot programme to fund replication studies. Read more Providing adequate funding for replication studies is an important precondition for ensuring researchers have the resources necessary to conduct them. UKRI should learn from its Dutch equivalent, NWO, by developing a pilot programme to fund replication studies. Show less

Aside from some notable exceptions, publishing routes for negative and confirmatory findings are not pursued thoroughly enough by the scholarly publishing industry. Publishers should review their journal portfolios to ensure that there are sufficient options for the publication of negative … Read more Aside from some notable exceptions, publishing routes for negative and confirmatory findings are not pursued thoroughly enough by the scholarly publishing industry. Publishers should review their journal portfolios to ensure that there are sufficient options for the publication of negative and confirmatory science, in line with the proportion of submissions which demand such routes. Show less

Publishers have a vital role in the maintenance of the scholarly record. Publishers should support academics who report issues with published research in their journals and should commit to timely publication of research error corrections and retractions where necessary—in our … Read more Publishers have a vital role in the maintenance of the scholarly record. Publishers should support academics who report issues with published research in their journals and should commit to timely publication of research error corrections and retractions where necessary—in our view this process should not take longer than two months. Publishers should also commit to timely deployment of technology to support the quality of the published record. (Paragraph 123) 52 Reproducibility and Research Integrity Solving reproducibility challenges in the assessment of research Show less

Reward structures in academia disincentivise reproducibility by placing disproportionate value on secured funding and frequent publication in prestigious journals.

The Future Research Assessment Programme (FRAP) is consulting on reforms for the assessment of UK higher education research. It should review the Research Excellence Framework assessment criteria to assure that transparency is a prerequisite of top- scoring research. It should … Read more The Future Research Assessment Programme (FRAP) is consulting on reforms for the assessment of UK higher education research. It should review the Research Excellence Framework assessment criteria to assure that transparency is a prerequisite of top- scoring research. It should also consider the effects of removing ‘originality’ from the top score bracket. Show less

Researchers should be assessed on the broader contributions to their academic field, including time spent conducting voluntary peer review and promoting reproducibility and research integrity. Funders, led by UKRI, should move towards the exclusive use of the ‘resume for researchers’ … Read more Researchers should be assessed on the broader contributions to their academic field, including time spent conducting voluntary peer review and promoting reproducibility and research integrity. Funders, led by UKRI, should move towards the exclusive use of the ‘resume for researchers’ format in funding calls by 2025. Show less

Peer review should not be viewed as a binary measure of quality versus unreliability for published papers. There is a wide range of competency, depth, and rigour in the analyses carried out during peer review as time-poor academics often do … Read more Peer review should not be viewed as a binary measure of quality versus unreliability for published papers. There is a wide range of competency, depth, and rigour in the analyses carried out during peer review as time-poor academics often do not have time to conduct detailed scrutiny. Where possible, journals should seek to publish peer review comments to improve transparency in the research publication process and to deter paper mills. Show less

The ‘registered report partnership model’ offers a good opportunity for researchers to have their methodologies peer reviewed at an early stage in the research process, allowing researchers to incorporate feedback into their research plans. Within this model, funding and the … Read more The ‘registered report partnership model’ offers a good opportunity for researchers to have their methodologies peer reviewed at an early stage in the research process, allowing researchers to incorporate feedback into their research plans. Within this model, funding and the principles for a study are agreed concurrently, with a guarantee of publication regardless of experiment outcomes. Publishers and funders should work together to offer a ‘registered report partnership model.’ Not only will this benefit researchers who will receive feedback on their research plans and guaranteed publication, but greater transparency in research methodologies will be achieved, which should have a positive impact on research reproducibility. (Paragraph 143) Reproducibility and Research Integrity 53 Show less