Fourth Report - Use of evaluation and modelling in government
Select Committee
Public Accounts Committee
HC 254
27 May 2022
Government Response (AI assessment · 19 of 19 classified)
Accepted
12
Recommendations
5 results
3
Accepted
No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation.
Recommendation
No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation. Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer. Supporting the accounting officer, the departmental directors of analysis …
Read more
Government Response Summary
The Government Analysis Function is piloting an assessment framework to monitor departments' implementation of the Analysis Functional Standard, and HM Treasury will build on its current training offer to ensure staff can confidently assure the quality of outputs from other departments.
HM Treasury
View Details
4
Accepted
Good quality modelling and evaluation is hampered by challenges in sharing data and a lack...
Recommendation
Good quality modelling and evaluation is hampered by challenges in sharing data and a lack of common data standards. Government does not always see data as a priority and data quality is often inadequate. We have drawn attention to poor …
Read more
Government Response Summary
The Cabinet Office will address barriers to data sharing and use through the national data strategy, with complete delivery by April 2025 and an interim progress report by December 2023; the CDDO has published the Data Sharing Governance Framework and is establishing the Data Sharing Playbook.
HM Treasury
View Details
6
Accepted
Model producers and users do not adequately assess the range of plausible outcomes and are...
Recommendation
Model producers and users do not adequately assess the range of plausible outcomes and are overly reliant on central estimates that do not reflect inherent uncertainty. All modelled information is uncertain, and decision-makers need information on the possible range of …
Read more
Government Response Summary
The Finance Function will work with the Analysis Function to identify appropriate analysis and modelling considerations relating to uncertainty and incorporate these into the Finance Standard and self-assessment tool.
HM Treasury
View Details
15
Accepted
HMT told us it is only recently that it has started to ask departments to...
Recommendation
HMT told us it is only recently that it has started to ask departments to explain why they do not plan to publish some evaluations.48 The Evaluation Task Force raised the issue that there were evaluations that the centre of …
Read more
Government Response Summary
The Evaluation Task Force is developing a publicly accessible online registry of UK government evaluations, allowing departments to publish evaluation plans/protocols and reports, and indicate estimated publication dates. The Analysis Function is reviewing its Functional Standard to incorporate improved practices across departments by the end of the year.
HM Treasury
View Details
18
Accepted
Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to...
Recommendation
Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to users the uncertainty in their models. There is limited evidence of departments using uncertainty analysis or developing contingency plans, and departments generally present outputs as …
Read more
Government Response Summary
The Finance Function will work with the Analysis Function to identify uncertainty analysis considerations and incorporate these into the Finance Standard and self-assessment tool.
HM Treasury
View Details
Conclusions (7) Observations and findings — click to expand
5
Conclusion
Accepted
Departments are not meeting government requirements on publishing evaluation plans and findings, and on transparency of models and their outputs. Transparency is critical for the public to understand the evidence behind decisions and how money is being spent. It also allows departments to learn lessons from past projects and from …
Government Response Summary
The government will publish and maintain an evaluation registry and review procedures around the quality assurance of models and incorporate improved practices across departments.
8
Conclusion
Accepted
Outside of the spending reviews, HM Treasury has few levers and exercises little scrutiny to ensure that departments comply with evaluation requirements and expectations.20 6 C&AG’s Report, Evaluating government spending, para 3.33; C&AG’s Report, Financial modelling in government, para 13, Figure 4 7 Qq 1, 18–19 8 Q 25 9 …
Government Response Summary
The Evaluation Task Force is working with HM Treasury to ensure that departments respond to and deliver on their settlement conditions relating to evaluation, using a central tracking system to monitor progress.
11
Conclusion
Accepted
To address the lack of overall oversight for both evaluation and modelling standards, the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard.31 The Analysis Function told us there will be an expectation in the …
Government Response Summary
The Government Analysis Function is piloting an assessment framework to monitor departments' implementation of the Analysis Functional Standard, and HM Treasury will build on its current training offer to ensure staff can confidently assure the quality of outputs from other departments.
12
Conclusion
Accepted
Government does not always see data as a priority and data quality is often inadequate.36 The NAO reported in 2019 that the quality and sharing of data is a clear example of a neglected and poorly planned activity, and there is a culture of tolerating and working around poor-quality data.37 …
Government Response Summary
The Central Digital and Data Office (CDDO) is delivering Mission 3 of the National Data Strategy, outlining how all departments will work towards better use of data, with the strategy completely delivered by April 2025.
14
Conclusion
Accepted
Government’s view is that that transparency is critical for the public to understand the evidence behind decisions and how money is being spent, and that transparency should be the default position.44 The Analysis Function consider that transparency of evaluation findings allows departments to learn lessons from past projects and make …
Government Response Summary
The Evaluation Task Force is developing a publicly accessible online registry of UK government evaluations, allowing departments to publish evaluation plans/protocols and reports, and indicate estimated publication dates. The Analysis Function is reviewing its Functional Standard to incorporate improved practices across departments by the end of the year.
17
Conclusion
Accepted
Uncertainty is inherent in modelled information and should be considered as part of all analysis. HM Treasury believes any model that does not explicitly address uncertainty is deficient.59 Decision-makers need information on the range of outcomes that may occur and their relative likelihoods to develop contingency plans and to avoid …
Government Response Summary
The Finance Function will work with the Analysis Function to identify appropriate analysis and modelling considerations relating to uncertainty and incorporate these into the Finance Standard.
19
Conclusion
Accepted
Departments present outputs from their models to HM Treasury as part of the spending review and budget process. HM Treasury uses these outputs for forecasting, budget planning and to monitor emerging risks. Departments usually provide HM 55 Qq 52–53 56 Q 54 57 C&AG’s Report, Financial modelling in government, paras …
Government Response Summary
The Finance Function will work with the Analysis Function to identify analysis and modelling considerations relating to uncertainty and incorporate these into the Finance Standard by March 2023.