Evaluation and the policy cycle

1 – Policy cycle: what it is, and what it is used for

The policy cycle is an approach used to plan and analyse the different phases of policy development (Giorgi 2017; HM Treasury 2011; Young and Quinn 2002). There are many ways to represent the policy cycle (see examples in Figure 5 and Figure 6 below). As highlighted by Young and Quinn (2002), “it is important to emphasise that policy processes are never as linear, or cyclical, as implied in the model. But, looking at the policy process in terms of these stages or functional elements can help us to understand how this process does (or should) work.”

Figure 5. The policy cycle (Source: Young and Quinn 2002).

Note: this policy cycle is named ROAMEF cycle (based on the initial of each step)
Figure 6. The ROAMEF policy cycle (Source: HM Treasury 2011).

As emphasised by Giorgi (2017), “the number and names of each phase can vary but the essence behind each step remains consistent”. Giorgi then summarizes the main general steps of a policy cycle as follows:

1. Agenda setting: “The general approach starts out with agenda setting which identifies the problem or issue that needs addressing. This first step often has specific phases of ‘defining the issue’ and ‘understanding the situation’.”
2. Considering and formulating policy options / alternatives: “This is then followed by steps which formulate and assess the different alternative courses of action and preparation for delivery.”
3. Choosing and specifying (designing) the preferred option: “In the following phase, Government decides on the course of action (which includes maintaining the status quo i.e. taking no action).”
4. Implementing and monitoring: “The decision made in the previous step will then be put into practice through implementation and monitoring”.
5. Evaluating and providing feedback for next period: “The final phase (which is the first step in the next cycle) is about assessing the effectiveness of the policy in terms of its intended objectives, outcomes and impacts. This ‘assessment of effectiveness’ is done through evaluation and adapting lessons learned into the future delivery of the policy.”

As reminded in the UK Magenta Book, “in practice this one-directional relationship rarely holds, the process is often iterative and there are significant interdependencies between the various elements” (HM Treasury 2011).

The qualitative feedback collected from policy stakeholders by Giorgi (2017) confirms that they usually know about policy cycle’s representations, but that they don’t necessarily use it. Mostly because it describes “how things should work in theory rather than how they actually worked in practice. Though the steps in the cycle were depicted as neat and compact, the real world was much messier and complex”. Still, most of the stakeholders interviewed in this study acknowledge that the policy cycle provides a basis to present, analyse and discuss a policy and particularly the process of its development.

2 – How evaluation fits into the policy cycle

(this section’s title is taken from the UK Magenta Book: see pp.14-15 of HM Treasury 2011)

In the above representations of the policy cycle, evaluation is shown as one step of the cycle, being the last one and closing the loop. This indeed corresponds to the usual definition of ex-post evaluations. However, evaluation practices and the policy cycle are much more interrelated in practice, as pointed in the UK Magenta Book:

“evaluations can, in fact, occur at practically any other time. And importantly, decisions affecting and relating to any evaluation will almost always be taken much earlier in the policy process” (HM Treasury 2011)

This point is essential. In the qualitative survey done by Giorgi (2017), “interviewees often stated that, though evaluation was embedded throughout the policy cycle, having it as the final step suggested it is something you only think about at the end.” Thinking about evaluation only at the end of the policy cycle is a major source of difficulties to conduct evaluations: not enough time available to get evaluation results for the decision making process, problems with data collection, difficulties to find or reconstitute the initial policy theory and objectives, etc.

At the opposite, Giorgi’s interviewees recommended a more integrated approach: “ ‘preparation for evaluation’ and ‘evidence gathering’ happened or should happen at each phase while the completion of a formal evaluation happened at the final step ‘evaluate & adapt’ ”.

Several stakeholders interviewed for EPATEE (see Bini et al. 2017) also pointed that integrating evaluation in the policy design was a good practice: results from previous ex-post evaluations and/or ex-ante evaluations of policies under consideration can inform the design process. Then thinking about evaluation from the start (i.e. when designing a policy) helps ensuring the feasibility of future ex-post evaluations, particularly by optimizing data collection.

“Through evaluation we can address several issues in the policy cycle, such as how a policy has been implemented, who, how and why has it been affected, if savings have been achieved and determine where it needs to be adapted, continued or ended”

“Evaluation should follow the whole policy cycle and be used in the planning as well as in the controlling (results) of the policy. Systems that incorporate this comprehensive approach seem to be more successful”

“During the design of a policy, an evaluation advisor should be present to ensure a good ex-post evaluation (e.g. if the data collection is not well design it is somewhat very difficult to evaluate the policy or at a large cost – which is somehow the reason for a lack of evaluation), the design should be ‘evaluation-friendly’ ”.

Quotes from the EPATEE interviews (Bini et al. 2017)

Beyond the usual good practice of planning evaluation early in the policy cycle, the integration of evaluation into the policy cycle should thus be seen in the two ways, as shown in Figure 7 below.

Figure 7. Two-way integration of evaluation into the policy cycle.

Using the descriptions of evaluation process (see Figure 3 and Figure 4 in part ) and policy cycle (see Figure 5 and in part Figure 6), Figure 8 represents them in a simplified way as joint processes with key interactions (red arrows in the figure), in order to illustrate more in details how they can be integrated.

Figure 9 and Figure 10 then provide a zoom about the stages of evaluation planning/preparation and conducting/using evaluation respectively.

As reminded above about the policy cycle, these processes are not necessarily linear. In particular, a good integration of evaluation into the policy cycle would mean multiple, and almost on-going, interactions between both.

Figure 8. Simplified joint representation of the policy cycle and evaluation process.

Figure 9. Zoom on the evaluation planning/preparation stage (linked to specifying/designing the policy).

Figure 10. Zoom on the conducting/using evaluation stage (linked with revising/adapting the policy).

Based on the interviews with policy stakeholders, Giorgi (2017) identified possible conflicts between the desired ideal of how evaluation should fit into the policy cycle and how things work in practice. She summarized this in the following key points on how does/should evaluation fit into the policy cycle:

• “Evidence gathering and preparation for evaluation is going on in different ways across all phases; this is/should be adapted and fed back to the relevant phase;
• Evaluation cannot delay policy development; there needs to be a timely input with quick feedback loops;
• Co-produced working between policymakers and analysts at each phase automatically embeds evaluation in each step, challenging the view that evaluation is done separately by the evaluation team and is, therefore, something that happens to policy rather than a way of working; and
• The policy cycle is/should be a ‘cycle’; the ‘final’ step in one wheel is the ‘first’ step in the next wheel ad infinitum.”

Based on this analysis and the review of the EPATEE case studies (see Broc et al. 2018), a set of key issues for integrating evaluation into the policy cycle could be identified.

Table 3. Key issues for integrating evaluation into the policy cycle.