Sixth Report - Reproducibility and Research Integrity
Select Committee
Science, Innovation and Technology Committee
HC 101
10 May 2023
Recommendations
21 results
3
Deferred
Para 43
Mandate UK CORI to publish annual research integrity reports and a detailed strategy action plan.
Recommendation
Despite significant delays, we welcome the establishment of UK Committee on Research Integrity (UK CORI) as a potential answer to existing research integrity challenges. We ask UK CORI to commit to produce an annual state of the nation report on …
Read more
Government Response Summary
The response does not cover those recommendations which are addressed to UKCORI, who will respond separately.
Department for Science, Innovation and Technology
View Details
4
Deferred
Para 44
Establish a dedicated UK CORI sub-committee to address research reproducibility challenges
Recommendation
It is disappointing that UK CORI’s recently published strategy did not mention reproducibility, especially since our inquiry highlighted that this is a major research integrity issue. UK CORI should make sure reproducibility challenges are given due attention and not overlooked …
Read more
Government Response Summary
The response does not cover those recommendations which are addressed to UKCORI, who will respond separately.
Department for Science, Innovation and Technology
View Details
5
Deferred
Para 45
Require UK CORI to investigate AI's impact on research reproducibility in annual reports
Recommendation
The increased application of artificial intelligence (AI) in scientific research presents a challenge to traditional research methods. The UK Committee on Research Integrity (UK CORI) should specifically: investigate the impact of deploying AI— and other increasingly complex software—on reproducibility in …
Read more
Government Response Summary
The response does not cover those recommendations which are addressed to UKCORI, who will respond separately.
Department for Science, Innovation and Technology
View Details
7
Rejected
Para 60
Develop a co-produced framework for responding to research misconduct cases effectively
Recommendation
The UK’s lacks an established infrastructure for responding to research misconduct cases. The UK Government should lead on a co-produced framework with the UK Reproducibility Network, UKRIO and UK CORI, which sets out the roles and expectations for key actors …
Read more
Government Response Summary
The government rejects the recommendation to lead on a co-produced framework for responding to research misconduct cases, asserting that government should not investigate misconduct and that primary responsibility lies with research organisations.
Department for Science, Innovation and Technology
View Details
8
Rejected
Para 61
Assess benefits of establishing an additional body to investigate research malpractice
Recommendation
The UK Government should assess the benefits that an additional body, set up to investigate malpractice, could bring to the UK’s research integrity governance architecture.
Government Response Summary
The government rejects the recommendation to assess creating a new body for investigating research malpractice, stating it is not the government's role and that primary responsibility lies with research organisations.
Department for Science, Innovation and Technology
View Details
9
Accepted
Para 68
Foster an environment where research integrity and reproducibility are championed across the community
Recommendation
Most reproducibility issues are, in the main, not the result of deliberate bad practice. Many of the incentives faced by individuals conducting research act against reproducibility. Whilst individuals must take responsibility for conducting work which prioritises robust analysis and transparency, …
Read more
Government Response Summary
The government partially accepts the recommendation, stating that UKRI is already supporting the research community in promoting integrity and reproducibility through various initiatives. These include shifting incentives via narrative CVs, changes to the REF process, hosting a research resource hub, and establishing a Good Practice Exchange.
Department for Science, Innovation and Technology
View Details
10
Accepted in Part
Para 72
Require universities to model reproducibility culture and implement protected research time policy
Recommendation
Research institutions should model a culture of reproducibility by managing inordinate pressures on academics and encouraging the prioritisation of reproducibility in research outputs. This extends to encouraging openness around mistakes and their correction. In collaboration with the Higher Education Sector, …
Read more
Government Response Summary
The government partially accepts the recommendation, stating it does not require a formal response from the government on some aspects. It highlights UKRI's extensive existing work to shift incentives in the research system, including changes to the REF process and a £4.5 million commitment to the UK Reproducibility Network (UKRN), to promote research integrity and reproducibility.
Department for Science, Innovation and Technology
View Details
11
Accepted in Part
Para 76
Develop dedicated funding and career paths for statistical experts and software developers
Recommendation
Statistical experts and software developers are insufficiently recognised and renumerated within the university research sector. Funders and universities should develop dedicated funding for the presence of statistical experts and software developers in research teams. In tandem, universities should work on …
Read more
Government Response Summary
The government partially accepts the recommendation, agreeing on the need for better recognition of statistical experts and software developers and citing existing UKRI initiatives. However, it states there is insufficient evidence that new dedicated funding streams would be more beneficial than integrating support into existing funds.
Department for Science, Innovation and Technology
View Details
13
Acknowledged
Para 82
Increase emphasis on reproducibility and integrity in university research education and training
Recommendation
Currently there is insufficient attention placed on reproducibility and research integrity training for university students and research professionals. Greater emphasis should be placed on the importance of reproducibility and research integrity in education and training at undergraduate, postgraduate and early …
Read more
Government Response Summary
The government accepts the recommendation, stating UKRI's Collective Talent Funding programme and existing Doctoral Training Centres already aim to address the need for high-quality training in research integrity and reproducibility. However, it does not commit to new specific actions regarding greater emphasis or routine production of replications in training.
Department for Science, Innovation and Technology
View Details
14
Accepted
Incorporate mandatory reproducibility training and professional development for researchers throughout their careers
Recommendation
Institutions should incorporate mandatory reproducibility training and professional development plans for researchers across the course of their career. (Paragraph 83) Reproducibility and Research Integrity 51
Government Response Summary
The government accepts the recommendation, stating UKRI's Collective Talent Funding programme and its Doctoral Training Centres and Partnerships already aim to provide high-quality, consistent development and training for researchers across different career stages, supporting skill development for doctoral students.
Department for Science, Innovation and Technology
View Details
15
Accepted in Part
Para 88
Implement a trial funding programme for 'slower' science and consult researchers on grant sufficiency
Recommendation
Short-term research grants place restrictive limitations on researchers, which can be to the detriment of research integrity and reproducibility. UKRI should consult with a representative sample of researchers to understand whether their grants allow them sufficient time and funding to …
Read more
Government Response Summary
The government partially accepts the recommendation, stating UKRI already funds a range of long-term projects and believes changing incentives is crucial beyond just grant length. While committed to continuing stakeholder work and experimenting with funding approaches, it does not explicitly commit to a new consultation with researchers or a dedicated trial funding programme for 'slower' science.
Department for Science, Innovation and Technology
View Details
17
Accepted in Part
Para 98
Implement 100% open access policies for scientific outputs by UKRI and funders by 2025
Recommendation
The trend towards blanket open access in the communication of scientific outputs is positive. UKRI and other research funders should continue to implement open access policies until this figure reaches 100%, by the end of 2025 at the latest.
Government Response Summary
The government partially accepts the recommendation, supporting the aspiration for 100% open access and detailing UKRI's existing policy and its expansion to monographs from January 2024. However, it notes complexities in implementation requiring action from various stakeholders, implying a lack of commitment to the specific 2025 deadline.
Department for Science, Innovation and Technology
View Details
18
Accepted in Part
Para 101
Mandate researchers to share open-source data and code alongside all published outputs
Recommendation
Currently, research outputs are frequently published without an associated link through to their open-source data and code. This prevents other researchers assessing work for its reproducibility. In all bar the most exceptional ethical and legal situations, researchers should share their …
Read more
Government Response Summary
The UKRI open access policy seeks to ensure that research articles, monographs, book chapters and edited collections that acknowledge its funding are made freely accessible. UKRI welcomes and supports the aspiration for 100% of UK research articles to be open access.
Department for Science, Innovation and Technology
View Details
19
Accepted in Part
Para 104
Mandate deposition of research data in open-access repositories and encourage FAIR principles
Recommendation
Journals should collectively encourage researchers to employ the FAIR (Findability, Accessibility, Interoperability, and Reuse of digital assets) principles within their research and should mandate the deposition of research data in open- access repositories alongside the publication of research outputs.
Read more
Government Response Summary
The UKRI open access policy seeks to ensure that research articles, monographs, book chapters and edited collections that acknowledge its funding are made freely accessible. UKRI welcomes and supports the aspiration for 100% of UK research articles to be open access.
Department for Science, Innovation and Technology
View Details
20
Acknowledged
Para 106
Continued emphasis on data management plans as a condition of research funding is necessary
Recommendation
We welcome UKRI’s use of data management plans. A continued emphasis on their importance as a condition of research funding is necessary.
Government Response Summary
The government accepts, affirming that data management plans remain an important requirement and UKRI will be considering how to further enhance their utility as part of policy and funding application requirement reviews.
Department for Science, Innovation and Technology
View Details
21
Accepted in Part
Para 115
Develop a pilot programme to fund replication studies, learning from NWO's model
Recommendation
Providing adequate funding for replication studies is an important precondition for ensuring researchers have the resources necessary to conduct them. UKRI should learn from its Dutch equivalent, NWO, by developing a pilot programme to fund replication studies.
Read more
Government Response Summary
The government partially accepts, stating UKRI will learn from NWO's pilot but does not commit to developing its own. They highlight existing funding mechanisms for replication studies and will consider increasing focus on metascience through a new caucus.
Department for Science, Innovation and Technology
View Details
22
Deferred
Para 116
Review journal portfolios to ensure sufficient publication options for negative and confirmatory science
Recommendation
Aside from some notable exceptions, publishing routes for negative and confirmatory findings are not pursued thoroughly enough by the scholarly publishing industry. Publishers should review their journal portfolios to ensure that there are sufficient options for the publication of negative …
Read more
Government Response Summary
The government partially accepts, welcoming the concept of registered reports for enhancing reproducibility but stating that further evaluation of their costs and benefits is needed. UKRI will continue to facilitate discussions and contribute to addressing challenges in this area.
Department for Science, Innovation and Technology
View Details
23
Accepted in Part
Require publishers to support academics reporting issues and ensure timely research corrections and retractions
Recommendation
Publishers have a vital role in the maintenance of the scholarly record. Publishers should support academics who report issues with published research in their journals and should commit to timely publication of research error corrections and retractions where necessary—in our …
Read more
Government Response Summary
The government welcomes the support from the Committee for increasing the use of registered reports to enhance reproducibility, however further evaluation of the costs and benefits of registered report partnership models is needed.
Department for Science, Innovation and Technology
View Details
25
Accepted in Part
Para 131
Review Research Excellence Framework criteria, ensuring transparency and considering 'originality' removal.
Recommendation
The Future Research Assessment Programme (FRAP) is consulting on reforms for the assessment of UK higher education research. It should review the Research Excellence Framework assessment criteria to assure that transparency is a prerequisite of top- scoring research. It should …
Read more
Government Response Summary
The government partially accepts, stating that decisions on the next Research Excellence Framework (REF) fall to funding bodies, but the new framework will include assessment of integrity, openness, transparency, and research reproducibility, with detailed criteria developed in 2024.
Department for Science, Innovation and Technology
View Details
26
Accepted in Part
Para 133
Assess researchers on broader academic contributions, mandating 'resume for researchers' format by 2025.
Recommendation
Researchers should be assessed on the broader contributions to their academic field, including time spent conducting voluntary peer review and promoting reproducibility and research integrity. Funders, led by UKRI, should move towards the exclusive use of the ‘resume for researchers’ …
Read more
Government Response Summary
The government partially accepts, declining to commit all funders to the exclusive use of the R4RI format by 2025. However, UKRI is committed to adopting R4RI across its own funding opportunities by the end of 2023 and is working with other funders to encourage broader adoption.
Department for Science, Innovation and Technology
View Details
27
Accepted in Part
Para 142
Publish peer review comments to improve transparency and deter paper mills.
Recommendation
Peer review should not be viewed as a binary measure of quality versus unreliability for published papers. There is a wide range of competency, depth, and rigour in the analyses carried out during peer review as time-poor academics often do …
Read more
Government Response Summary
The government partially accepts the recommendation by welcoming the support for increasing the use of registered reports to enhance reproducibility, but notes that further evaluation of the costs and benefits of registered report partnership models is needed.
Department for Science, Innovation and Technology
View Details
Conclusions (7) Observations and findings — click to expand
1
Conclusion
Accepted
Para 27
Although qualitative evidence indicates a potentially substantial scale of research integrity issues in the UK, there is a lack of quantitative evidence, including on the relative significance of the different causes of problems. This can only hamper efforts to evaluate damage being caused to the UK research sector in terms …
Government Response Summary
The government states that the newly established UKRI Committee on Research Integrity (UK CORI) has a role in improving the evidence base on research integrity, thereby addressing the identified lack of quantitative data.
2
Conclusion
Not Addressed
Para 32
Though the specific problems faced differ, all research disciplines are affected by the systemic issues that limit reproducibility. The established view that issues are concentrated in the social and medical sciences is outdated and tackling reproducibility challenges should be a priority in every research field, regardless of discipline and methodology.
Government Response Summary
The government's response highlights the variability in reproducibility requirements and practices across different research fields funded by UKRI, without directly addressing the committee's call for tackling reproducibility to be a priority in every field.
6
Conclusion
Acknowledged
Whilst significant reproducibility challenges are faced in research, to refer to the sum of these issues as a “crisis” risks detracting from the many successes of the UK’s scientific research base. Nonetheless, there is need for action to address the significant problems caused by the prevalence or reproducibility problems in …
Government Response Summary
The government acknowledges the need for action on reproducibility challenges, highlighting existing efforts like the R&D People and Culture Strategy and the Future Research Assessment Programme to strengthen research culture and integrity without committing to new specific actions.
12
Conclusion
Acknowledged
Para 77
Research funders should implement stronger tests for the presence of adequate software and statistical skills within research teams at the outset of a funding application. Where these skills are perceived to be lacking, UKRI should consider the feasibility and cost of offering a dedicated methodological support system to research teams.
Government Response Summary
The government accepts the recommendation but provides no specific details on how it will implement stronger tests for software/statistical skills or consider a dedicated methodological support system for research teams.
16
Conclusion
Rejected
Uncertainty in the academic job market, especially at earlier career stages, acts as a strong additional disincentive against the prioritisation of reproducibility by researchers. Research funders, including UKRI, should work to impose a three-year minimum contract for post-doctoral researchers in universities. (Paragraph 89) Solving reproducibility challenges which emerge in the …
Government Response Summary
The government explicitly does not accept the recommendation to impose a three-year minimum contract for post-doctoral researchers in universities.
24
Conclusion
Accepted
Para 127
Reward structures in academia disincentivise reproducibility by placing disproportionate value on secured funding and frequent publication in prestigious journals.
Government Response Summary
The government partially accepts, explaining that UKRI has an extensive portfolio of ongoing work to shift incentives in the research system, including narrative CVs, changes to the REF, funding for the UK Reproducibility Network, and supporting the UK Committee on Research Integrity.
28
Conclusion
Acknowledged
The ‘registered report partnership model’ offers a good opportunity for researchers to have their methodologies peer reviewed at an early stage in the research process, allowing researchers to incorporate feedback into their research plans. Within this model, funding and the principles for a study are agreed concurrently, with a guarantee …
Government Response Summary
The government partially accepts the recommendation to offer a 'registered report partnership model' but provides no specific details on what aspects are accepted or how this will be implemented.