Monitoring and Evaluation
SRUC has published a Rapid Evidence Review of Monitoring and Evaluation Frameworks for Agricultural Support into support thinking of the Monitoring and Evaluation (M&E) framework for the Scottish Government's Agricultural Reform Programme (ARP) that is being developed to monitor and evaluate actions taken to meet its policy outcomes. This report provides evidence on M&E frameworks and methods being used in selected countries, including the EU.
Globally, and within Scottish Government policy domains, there is an increased focus on monitoring and evaluation (M&E) of public policy to demonstrate (a) value for money for the public purse and (b) effectiveness of policy interventions in progress towards policy goals. Indeed, the Organisation for Economic Cooperation and Development (OECD) now recommends that all members institutionalise public policy evaluation across all government departments.
Since the UK withdrew from the European Union (EU) in 2020, Scotland is no longer part of the Common Agricultural Policy (CAP) and is, therefore, no longer part of the 2023-27 iteration of the CAP and its new Performance, Monitoring and Evaluation Framework (PMEF). As such, the Scottish Government and other UK administrations are evolving their own agriculture and rural community policy designed to address Scottish needs. As part of that process, the Scottish Government is developing a monitoring and evaluation framework that can monitor progress towards policy outcomes.
Whilst the specific terminology often differs, the key methodological approaches taken by different administrations on monitoring and evaluation frameworks follow a similar hierarchical taxonomy of indicators for use in monitoring and evaluating policy interventions. Most evaluation frameworks now include the United Nation’s (UN) Sustainability Goals as key outcomes, with national, sectoral and regional indicators and outcomes monitoring progress on these higher international goals.
This report examines examples of monitoring and evaluation frameworks that have been, or are being, developed in other jurisdictions as agricultural policy evolves to be more outcome-focused. Developments in the EU, the Republic of Ireland, Northern Ireland, Wales and England have been reviewed alongside recent developments on monitoring and evaluation for the Scottish National Adaptation Plan 2024-2029 (SNAP3), the Scottish Environment Strategy, the Climate Change Plan, the Good Food Nation Plan and the Rural Delivery Plan.
The EU have evolved their M&E framework for the 2023-27 CAP, adopting 159 outputs, results, context and impact indicators (137 unique) that will be monitored as part of the new performance-based delivery model. Each Member State (MS) has had to prepare an Evaluation Plan for their CAP Strategic Plans (the latter of which sets out the policy ambitions, intervention logic, schemes, targets and budgetary allocations – similar to the evolving Scottish Rural Support Plan). Many of these indicators would be useful in a Scottish context, including as part of any monitoring for the Rural Delivery Plan, or National Island Plan.
Other UK administrations are still evolving their M&E frameworks for agricultural policy, and nothing is published on indicators and frameworks at the time of writing. Each administration appears to be taking different approaches that reflect their policy emphasis and stage of policy development. For example, Defra has its delivery plan, as well as their 25-year Environment Plan Framework Indicators and Evaluation Strategy, whilst in Northern Ireland, progress to agreeing 11 key headline indicators appears to have been made. This is an evolving topic, and it is likely mutually beneficial to consider the development of an inter-administration evaluation learning network for Government analysts across the UK (similar to the EU CAP Network).
Similarly to provisions in the Agriculture and Rural Communities (Scotland) Act 2024, the Agriculture (Wales) Act 2023 requires Welsh Ministers to develop a 5-year support plan that details how Welsh Ministers will use their powers (including details of schemes) to achieve Sustainable Land Management objectives. Further, the Welsh Government must also report to Senedd Cymru on (i) annual financial and non-financial performance, (ii) 5-yearly sustainable land management report detailing progress towards indicators and objectives; and (iii) an impact report every 5-years that includes an evaluation of policy impacts and effectiveness in reaching their Sustainable Land Management objectives. With these similarities, monitoring activity in Wales, and developing analytical and policy linkages may be meritorious.
Regardless of the extent of progress in the development of M&E frameworks there are important lessons to be learned from other administrations (including the EU Member States) approaches to monitoring, data acquisition and baselining – particularly in the biodiversity and soils domains.
As the Scottish Government has a commitment to maintain alignment with the EU where practicably possible there may be sound arguments to adopt the EU framework approach with their layers of context, impact, results and outcome indicators. To simplify the messaging on progress towards desired outcomes, there may also be sound justification for the Scottish Government narrowing the number of headline indicators used that can be informed by a series of secondary indicators (as being done in Northern Ireland approach).
In Scotland, it is important that any M&E framework for agricultural policy in Scotland utilises indicators that adequately reflect the full extent of Scottish Government’s policy ambitions for agricultural policy (rural economy, rural society, nature recovery, climate mitigation and adaptation, animal health and welfare, sustainable and regenerative practices). However, it is also important that such a framework can help inform other policy domains, for example demonstrating how policy interventions contribute to SNAP3 (or the Natural Environment Bill, Scottish Environment Strategy, the Climate Change Plan, the Good Food Nation Plan and the Rural Delivery Plan) outcomes and indicators, as well as the National Performance Framework.
HM Treasury’s Magenta Book provides useful information and guidance on design of policy evaluations. The SNAP3 provides a useful M&E framework that could form the building blocks (and approach) from which to develop a new M&E framework. In particular, the Monitoring Maps developed in SNAP3, alongside the use of Theory of Change models (SNAP3, Defra, EU) may provide a useful basis for a M&E framework for agricultural policy in Scotland from which appropriate indicators can be derived.
