Use of evaluation and financial modelling in Government

Public Accounts Committee Closed Inquiry
Opened: 12 Jan 2022 Closed: 2 Sep 2022 Parliament page
Government relies on hundreds – possibly thousands – of models for a huge variety of critical activities, from forecasting spending plans to guiding operational decision making, and increasingly relies on these models as more data is available and there is an increased emphasis on evidence-driven decision-making. When it comes to … Read more
7 Recommendations
12 Conclusions
1 Report
1 Oral session
1 Event
Activity timeline 3 events
2 Sep
2022
Report published
27 May
2022
Report published
9 Mar
2022
Oral evidence
Oral evidence sessions 1 session
9 Mar 2022
View on parliament.uk
Use of evaluation in Government/Financial Modelling
Catherine Hutchinson · HM Treasury and the Cabinet Office Cat Little · Cabinet Office Professor Sir Ian Diamond · UK Statistics Authority Sir Tom Scholar · HM Treasury
Recommendations & Conclusions
19 results
2 Recommendation
Fourth Report - Use of evaluation …
HM Treasury is not making full use of the spending levers it has at its...
HM Treasury is not making full use of the spending levers it has at its disposal to deliver a step change in the use of evaluation across government. We welcome the creation of the Evaluation Task Force and the increased … Read more
HM Treasury
View details
3 Recommendation
Fourth Report - Use of evaluation …
No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation.
No single body is responsible for upholding modelling and evaluation standards and monitoring their implementation. Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer. Supporting the accounting officer, the departmental directors of analysis … Read more
HM Treasury
View details
4 Recommendation
Fourth Report - Use of evaluation …
Good quality modelling and evaluation is hampered by challenges in sharing data and a lack...
Good quality modelling and evaluation is hampered by challenges in sharing data and a lack of common data standards. Government does not always see data as a priority and data quality is often inadequate. We have drawn attention to poor … Read more
HM Treasury
View details
5 Conclusion
Fourth Report - Use of evaluation …
Departments are not meeting government requirements on publishing evaluation plans and findings, and on transparency...
Departments are not meeting government requirements on publishing evaluation plans and findings, and on transparency of models and their outputs. Transparency is critical for the public to understand the evidence behind decisions and how money is being spent. It also … Read more
HM Treasury
View details
6 Recommendation
Fourth Report - Use of evaluation …
Model producers and users do not adequately assess the range of plausible outcomes and are...
Model producers and users do not adequately assess the range of plausible outcomes and are overly reliant on central estimates that do not reflect inherent uncertainty. All modelled information is uncertain, and decision-makers need information on the possible range of … Read more
HM Treasury
View details
1 Conclusion
Fourth Report - Use of evaluation …
On the basis of two reports by the Comptroller and Auditor General, we took evidence...
On the basis of two reports by the Comptroller and Auditor General, we took evidence from HM Treasury, the Finance Function, the Analysis Function and the Evaluation Task Force about the use of evaluation and modelling across government.1 Cabinet Office … Read more
HM Treasury
View details
7 Conclusion
Fourth Report - Use of evaluation …
HM Treasury has used the last two spending reviews to increase the emphasis on evaluation...
HM Treasury has used the last two spending reviews to increase the emphasis on evaluation in departments, but acknowledges it has a lot more to do to address the persistent problems which prevent the provision and use of evaluation.17 Working … Read more
HM Treasury
View details
8 Conclusion
Fourth Report - Use of evaluation …
Outside of the spending reviews, HM Treasury has few levers and exercises little scrutiny to...
Outside of the spending reviews, HM Treasury has few levers and exercises little scrutiny to ensure that departments comply with evaluation requirements and expectations.20 6 C&AG’s Report, Evaluating government spending, para 3.33; C&AG’s Report, Financial modelling in government, para 13, … Read more
HM Treasury
View details
9 Conclusion
Fourth Report - Use of evaluation …
Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting...
Within departments, ultimate responsibility for upholding standards and securing improvements lies with that department’s accounting officer.24 Managing Public Money states that financial modelling and evaluation are an important part of the accounting officer’s role, and that they should comply with … Read more
HM Treasury
View details
10 Conclusion
Fourth Report - Use of evaluation …
While HM Treasury is an important user of department’s models, it does not comprehensively scrutinise...
While HM Treasury is an important user of department’s models, it does not comprehensively scrutinise and challenge a department’s quality assurance arrangements for the models it uses. There is no requirement for the Aqua Book to be followed as part … Read more
HM Treasury
View details
11 Conclusion
Fourth Report - Use of evaluation …
To address the lack of overall oversight for both evaluation and modelling standards, the Analysis...
To address the lack of overall oversight for both evaluation and modelling standards, the Analysis Function intends to strengthen the governance framework for the Analysis Functional Standard, including introducing a self-assessment framework to assess performance against the standard.31 The Analysis … Read more
HM Treasury
View details
12 Conclusion
Fourth Report - Use of evaluation …
Government does not always see data as a priority and data quality is often inadequate.36...
Government does not always see data as a priority and data quality is often inadequate.36 The NAO reported in 2019 that the quality and sharing of data is a clear example of a neglected and poorly planned activity, and there … Read more
HM Treasury
View details
13 Conclusion
Fourth Report - Use of evaluation …
When producing and using models, poor data can lead to additional time and increased difficulty...
When producing and using models, poor data can lead to additional time and increased difficulty when quality assuring the model inputs. This difficulty is compounded by a lack of data standards across government which means there are inconsistent ways of … Read more
HM Treasury
View details
14 Conclusion
Fourth Report - Use of evaluation …
Government’s view is that that transparency is critical for the public to understand the evidence...
Government’s view is that that transparency is critical for the public to understand the evidence behind decisions and how money is being spent, and that transparency should be the default position.44 The Analysis Function consider that transparency of evaluation findings … Read more
HM Treasury
View details
15 Recommendation
Fourth Report - Use of evaluation …
HMT told us it is only recently that it has started to ask departments to...
HMT told us it is only recently that it has started to ask departments to explain why they do not plan to publish some evaluations.48 The Evaluation Task Force raised the issue that there were evaluations that the centre of … Read more
HM Treasury
View details
16 Recommendation
Fourth Report - Use of evaluation …
Transparency of the models that departments use is poor: only 9 out of 17 departments...
Transparency of the models that departments use is poor: only 9 out of 17 departments have published their list of business-critical models since 2013, and only four of those have updated this since 2017.54 HM Treasury has not published its … Read more
HM Treasury
View details
17 Conclusion
Fourth Report - Use of evaluation …
Uncertainty is inherent in modelled information and should be considered as part of all analysis.
Uncertainty is inherent in modelled information and should be considered as part of all analysis. HM Treasury believes any model that does not explicitly address uncertainty is deficient.59 Decision-makers need information on the range of outcomes that may occur and … Read more
HM Treasury
View details
18 Recommendation
Fourth Report - Use of evaluation …
Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to...
Despite the importance of uncertainty analysis, model producers do not routinely assess or communicate to users the uncertainty in their models. There is limited evidence of departments using uncertainty analysis or developing contingency plans, and departments generally present outputs as … Read more
HM Treasury
View details
19 Conclusion
Fourth Report - Use of evaluation …
Departments present outputs from their models to HM Treasury as part of the spending review...
Departments present outputs from their models to HM Treasury as part of the spending review and budget process. HM Treasury uses these outputs for forecasting, budget planning and to monitor emerging risks. Departments usually provide HM 55 Qq 52–53 56 … Read more
HM Treasury
View details