Use of evaluation and financial modelling in Government

Public Accounts Committee Closed Inquiry
Opened: 12 Jan 2022 Closed: 2 Sep 2022 Parliament page
Government relies on hundreds – possibly thousands – of models for a huge variety of critical activities, from forecasting spending plans to guiding operational decision making, and increasingly relies on these models as more data is available and there is an increased emphasis on evidence-driven decision-making. When it comes to … Read more
7 Recommendations
12 Conclusions
1 Report
1 Oral session
1 Event
Activity timeline 4 events
2 Sep
2022
Report published
27 May
2022
Report published
9 Mar
2022
Oral evidence
9 Mar
2022
Formal meeting (oral evidence session) · The Wilson Room, Portcullis House
Oral evidence sessions 1 session
9 Mar 2022
View on parliament.uk
Use of evaluation in Government/Financial Modelling
Catherine Hutchinson · HM Treasury and the Cabinet Office Cat Little · Cabinet Office Professor Sir Ian Diamond · UK Statistics Authority Sir Tom Scholar · HM Treasury
Title HC No. Published Items Response
Fourth Report - Use of evaluation and modelling in government HC 254 27 May 2022 19 Responded
Recommendations & Conclusions
19 results
2 Recommendation Not Addressed
Fourth Report - Use of evaluation …
HM Treasury is not making full use of the spending levers it has at its...
HM Treasury is not making full use of the spending levers it has at its disposal to deliver a step change in the use of evaluation across government. We welcome the creation of the Evaluation Task Force and the increased … Read more
Government Response
The response does not address the recommendation of setting up a formal process for tracking spending review settlement conditions relating to evaluation, and instead discusses internal performance measures and consultations with the sector.
HM Treasury
View details
3 Recommendation Accepted
Fourth Report - Use of evaluation …
No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation.
No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation. Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer. Supporting the accounting officer, the departmental directors of analysis … Read more
Government Response
The Government Analysis Function is piloting an assessment framework to monitor departments' implementation of the Analysis Functional Standard, and HM Treasury will build on its current training offer to ensure staff can confidently assure the quality of outputs from other departments.
HM Treasury
View details
4 Recommendation Accepted
Fourth Report - Use of evaluation …
Good quality modelling and evaluation is hampered by challenges in sharing data and a lack...
Good quality modelling and evaluation is hampered by challenges in sharing data and a lack of common data standards. Government does not always see data as a priority and data quality is often inadequate. We have drawn attention to poor … Read more
Government Response
The Cabinet Office will address barriers to data sharing and use through the national data strategy, with complete delivery by April 2025 and an interim progress report by December 2023; the CDDO has published the Data Sharing Governance Framework and is establishing the Data Sharing Playbook.
HM Treasury
View details
5 Conclusion Accepted
Fourth Report - Use of evaluation …
Departments are not meeting government requirements on publishing evaluation plans and findings, and on transparency...
Departments are not meeting government requirements on publishing evaluation plans and findings, and on transparency of models and their outputs. Transparency is critical for the public to understand the evidence behind decisions and how money is being spent. It also … Read more
Government Response
The government will publish and maintain an evaluation registry and review procedures around the quality assurance of models and incorporate improved practices across departments.
HM Treasury
View details
6 Recommendation Accepted
Fourth Report - Use of evaluation …
Model producers and users do not adequately assess the range of plausible outcomes and are...
Model producers and users do not adequately assess the range of plausible outcomes and are overly reliant on central estimates that do not reflect inherent uncertainty. All modelled information is uncertain, and decision-makers need information on the possible range of … Read more
Government Response
The Finance Function will work with the Analysis Function to identify appropriate analysis and modelling considerations relating to uncertainty and incorporate these into the Finance Standard and self-assessment tool.
HM Treasury
View details
1 Conclusion Acknowledged
Fourth Report - Use of evaluation …
On the basis of two reports by the Comptroller and Auditor General, we took evidence...
On the basis of two reports by the Comptroller and Auditor General, we took evidence from HM Treasury, the Finance Function, the Analysis Function and the Evaluation Task Force about the use of evaluation and modelling across government.1 Cabinet Office … Read more
Government Response
The Evaluation Task Force (ETF) was established in 2021 as a joint HM Treasury and Cabinet Office unit. The ETF will also publish its strategy which will explain how it is working to improve the evaluation system in greater detail, including indicators to measure progress.
HM Treasury
View details
7 Conclusion Acknowledged
Fourth Report - Use of evaluation …
HM Treasury has used the last two spending reviews to increase the emphasis on evaluation...
HM Treasury has used the last two spending reviews to increase the emphasis on evaluation in departments, but acknowledges it has a lot more to do to address the persistent problems which prevent the provision and use of evaluation.17 Working … Read more
Government Response
The Evaluation Task Force is working with HM Treasury to ensure that departments respond to and deliver on their settlement conditions relating to evaluation, captured in a central tracking system and monitored by senior officials; approval for future funding can be withheld if departments don't meet their conditions of settlement.
HM Treasury
View details
8 Conclusion Accepted
Fourth Report - Use of evaluation …
Outside of the spending reviews, HM Treasury has few levers and exercises little scrutiny to...
Outside of the spending reviews, HM Treasury has few levers and exercises little scrutiny to ensure that departments comply with evaluation requirements and expectations.20 6 C&AG’s Report, Evaluating government spending, para 3.33; C&AG’s Report, Financial modelling in government, para 13, … Read more
Government Response
The Evaluation Task Force is working with HM Treasury to ensure that departments respond to and deliver on their settlement conditions relating to evaluation, using a central tracking system to monitor progress.
HM Treasury
View details
9 Conclusion Not Addressed
Fourth Report - Use of evaluation …
Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting...
Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer.24 Managing Public Money states that financial modelling and evaluation are an important part of the accounting officer’s role, and that they should comply with … Read more
Government Response
The response restates the conclusion.
HM Treasury
View details
10 Conclusion Acknowledged
Fourth Report - Use of evaluation …
While HM Treasury is an important user of department’s models, it does not comprehensively scrutinise...
While HM Treasury is an important user of department’s models, it does not comprehensively scrutinise and challenge a department’s quality assurance arrangements for the models it uses. There is no requirement for the Aqua Book to be followed as part … Read more
Government Response
The Government Analysis Function is piloting an assessment framework to monitor departments' implementation of the Analysis Functional Standard; the ONS is considering the future scope of the Analysis Function; HM Treasury will ensure staff who use outputs from other departments are able to confidently assure quality.
HM Treasury
View details
11 Conclusion Accepted
Fourth Report - Use of evaluation …
To address the lack of overall oversight for both evaluation and modelling standards, the Analysis...
To address the lack of overall oversight for both evaluation and modelling standards, the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard.31 The Analysis … Read more
Government Response
The Government Analysis Function is piloting an assessment framework to monitor departments' implementation of the Analysis Functional Standard, and HM Treasury will build on its current training offer to ensure staff can confidently assure the quality of outputs from other departments.
HM Treasury
View details
12 Conclusion Accepted
Fourth Report - Use of evaluation …
Government does not always see data as a priority and data quality is often inadequate.36...
Government does not always see data as a priority and data quality is often inadequate.36 The NAO reported in 2019 that the quality and sharing of data is a clear example of a neglected and poorly planned activity, and there … Read more
Government Response
The Central Digital and Data Office (CDDO) is delivering Mission 3 of the National Data Strategy, outlining how all departments will work towards better use of data, with the strategy completely delivered by April 2025.
HM Treasury
View details
13 Conclusion Acknowledged
Fourth Report - Use of evaluation …
When producing and using models, poor data can lead to additional time and increased difficulty...
When producing and using models, poor data can lead to additional time and increased difficulty when quality assuring the model inputs. This difficulty is compounded by a lack of data standards across government which means there are inconsistent ways of … Read more
Government Response
The Cabinet Office will address barriers to data sharing and use through the national data strategy, with complete delivery by April 2025 and an interim progress report by December 2023; the CDDO has published the Data Sharing Governance Framework and is establishing the Data Sharing Playbook.
HM Treasury
View details
14 Conclusion Accepted
Fourth Report - Use of evaluation …
Government’s view is that that transparency is critical for the public to understand the evidence...
Government’s view is that that transparency is critical for the public to understand the evidence behind decisions and how money is being spent, and that transparency should be the default position.44 The Analysis Function consider that transparency of evaluation findings … Read more
Government Response
The Evaluation Task Force is developing a publicly accessible online registry of UK government evaluations, allowing departments to publish evaluation plans/protocols and reports, and indicate estimated publication dates. The Analysis Function is reviewing its Functional Standard to incorporate improved practices across departments by the end of the year.
HM Treasury
View details
15 Recommendation Accepted
Fourth Report - Use of evaluation …
HMT told us it is only recently that it has started to ask departments to...
HMT told us it is only recently that it has started to ask departments to explain why they do not plan to publish some evaluations.48 The Evaluation Task Force raised the issue that there were evaluations that the centre of … Read more
Government Response
The Evaluation Task Force is developing a publicly accessible online registry of UK government evaluations, allowing departments to publish evaluation plans/protocols and reports, and indicate estimated publication dates. The Analysis Function is reviewing its Functional Standard to incorporate improved practices across departments by the end of the year.
HM Treasury
View details
16 Recommendation Not Addressed
Fourth Report - Use of evaluation …
Transparency of the models that departments use is poor: only 9 out of 17 departments...
Transparency of the models that departments use is poor: only 9 out of 17 departments have published their list of business-critical models since 2013, and only four of those have updated this since 2017.54 HM Treasury has not published its … Read more
Government Response
The response discusses publishing an evaluation registry and reviewing quality assurance procedures, which doesn't directly address the need to publish lists of business-critical models or improve guidance on model transparency.
HM Treasury
View details
17 Conclusion Accepted
Fourth Report - Use of evaluation …
Uncertainty is inherent in modelled information and should be considered as part of all analysis.
Uncertainty is inherent in modelled information and should be considered as part of all analysis. HM Treasury believes any model that does not explicitly address uncertainty is deficient.59 Decision-makers need information on the range of outcomes that may occur and … Read more
Government Response
The Finance Function will work with the Analysis Function to identify appropriate analysis and modelling considerations relating to uncertainty and incorporate these into the Finance Standard.
HM Treasury
View details
18 Recommendation Accepted
Fourth Report - Use of evaluation …
Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to...
Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to users the uncertainty in their models. There is limited evidence of departments using uncertainty analysis or developing contingency plans, and departments generally present outputs as … Read more
Government Response
The Finance Function will work with the Analysis Function to identify uncertainty analysis considerations and incorporate these into the Finance Standard and self-assessment tool.
HM Treasury
View details
19 Conclusion Accepted
Fourth Report - Use of evaluation …
Departments present outputs from their models to HM Treasury as part of the spending review...
Departments present outputs from their models to HM Treasury as part of the spending review and budget process. HM Treasury uses these outputs for forecasting, budget planning and to monitor emerging risks. Departments usually provide HM 55 Qq 52–53 56 … Read more
Government Response
The Finance Function will work with the Analysis Function to identify analysis and modelling considerations relating to uncertainty and incorporate these into the Finance Standard by March 2023.
HM Treasury
View details
Government Response AI assessment · 19 of 7 classified
Accepted 12
Acknowledged 4
Not Addressed 3
Total 7 recs + 12 conclusions