Good practices for the integration of evaluation into the policy cycle

The overall approach of aiming at integrating evaluation into the policy cycle can be summarized as follows:

“Evaluation needs to be an integrative, continuous process not a one-off exercise at the end or a series of self-contained steps, it needs to become a way of working” (Giorgi 2017)

This part presents examples of good practices collected in the EPATEE case studies, the second EPATEE online survey (Bini et al. 2018) or in the survey done by Giorgi (2017). They are not meant to be exhaustive, but to reflect empirical findings from the analysis of current practices for samples of evaluations. The objective is to contribute to experience sharing and capacity building.

The good practices are presented as actions that can be done by persons or units in charge of evaluation within public bodies. But they can also be considered by evaluators, private stakeholders or more generally anyone interested in the development of evaluation practices.

These actions are structured according to the main issues identified in Table 3, and grouped in two categories, respectively short-term and medium-term actions.

The first category can be considered whenever launching a new evaluation (or better whenever launching a new or revising an existing policy). The second category includes suggestions to improve practices and facilitate the integration of evaluation into the policy cycle over time.

Some of the actions presented in the following sections can overlap as the different issues are often linked. The Annex provides two summary tables, respectively for short-term and medium-term actions.

The actions presented below should be seen as suggestions. They do not necessarily apply to all contexts and situations. Their relevance also depends on the “magnitude” or ambition of the evaluation activities, the political stakes, size of the policy evaluated, history of the policy, etc.

1 – Political will (top-management commitment)

1.1 – Clarify expectations, what evaluation can bring and how it can be used

Political will to do evaluations is often linked to policymakers’ interest in what they can get or learn from an evaluation. This can thus depend on their background and previous experience with policy making and evaluation. Ensuring that all parties involved in the evaluation are on the same line about what it can (or cannot) bring can thus be critical for the decision to launch an evaluation or the success of the evaluation.

Another key issue is policymakers’ willingness to accept risk and failure. Extreme cases being to reject doing evaluation either because one may be sure that the policy is successful (so no need to verify it), or because one may be afraid of the results that the evaluation could show. Experience sharing is then a way to overcome preconceived ideas. Particularly by showing that understanding reasons of failures, weaknesses or limitations is the best solution to learn how to design successful policies.

1.2 – Analyse the current policy framework and processes, how evaluation can fit in, and what stakeholders can ask for evaluations

There is an overall trend to institutionalize evaluation processes. This can take various forms, for example:

• legal provisions to ensure that evaluations will be done for certain types of policies, at certain stages of the policy cycle, etc.;
• creation of services or institutions dedicated to evaluation (e.g. evaluation unit or evaluation body);
• expanding the role or missions of existing institutions (e.g. Court of Auditors, Parliament).

More generally, an evaluation will occur in a given policy framework that is important to take into account to identify the stakeholders who can be involved in the evaluation process, what the timing of the evaluation should be to be in line with decision processes, etc. It is therefore useful to analyse the main policy frameworks, which is mostly related to medium-term actions.

1.3 – Mandatory provisions for evaluation

In the interviews and online surveys done for the EPATEE project, several stakeholders pointed that how many policies are evaluated (and to what extent) sometimes depends on the national laws and regulation. Systematic evaluation requirements were reported for some countries (e.g., Germany, Sweden, UK). Such requirements were sometimes mentioned to apply mostly for ex-ante evaluation or impact assessment (e.g., Austria), or to be focused on “large budget” policies (e.g., Italy). Some stakeholders also told that the lack of evaluation requirements and/or guidelines was a key reason for the lack of evaluation in their country.

This point goes beyond the previous one, in case a general lack of evaluation is observed. It should indeed be noted that obligations to evaluate can have counterproductive effects, for example: making that evaluation is perceived as a burden, leading to evaluation reports that are not read or used.

^* several stakeholders have highlighted that EU requirements are often considered as a “must” by national bodies (which is not always the case for national requirements)

2 – Resource allocation (time, people, budget)

2.1 – Discuss evaluation means when deciding the budget for the policy measure

As pointed by one interviewee during the interviews done for EPATEE, evaluation costs are usually very small compared to the cost of implementing the policies. Still, it was a recurring feedback that not enough budget was available for evaluation. This can be partly explained because evaluation is sometimes considered late in the policy cycle, at a time where most of the budget for the policy has already been used.

Beyond budget, this issue also encompasses other types of means that can be needed for the evaluation, for example the time needed from implementing staff to collect and report data.

One way to tackle this issue can thus be to ensure that the means needed for the evaluation are discussed when deciding about the overall means committed to the policy.

This issue is also related to evaluation planning (see 6.3.1) and organisation (see 6.5). Particularly, several stakeholders pointed that doing evaluation should as much as possible avoid to create new administrative processes and work including new information collection systems (see 0 about identifying synergies).

Likewise, actions to show the added value of evaluation (see 6.1.1) can help to get adequate evaluation means.

The discussion about evaluation means can therefore be an opportunity to discuss the cost-efficiency of evaluation efforts: what the evaluation can bring vs. what means it would require. Experience shows that stakeholders involved in policy implementation are more inclined to contribute to evaluation (e.g. for data collection), when they have a clear view about what their contribution (e.g. data) will be used for.

2.2 – Define criteria to assess the needs in evaluation means

The previous point shows that it can be useful to have criteria or guidelines at hands to have quick insights about the means that can be needed, depending on the type of evaluation objectives. Such tools can then help for evaluation means to be taken into account in budgetary discussions. It can also contribute to enable an early planning of evaluation (see 6.3.1).

Developing such criteria or guidelines is usually a medium-term action.

3 – Evaluation planning and preparation

3.1 – Make early planning of evaluation a common practice

Taking into account future evaluation needs when designing or revising a policy is a usual no-brainer of the recommendations about evaluation practices. However, feedback from stakeholders confirms that it is not yet a common practice.

This could be due to various reasons: priority given to action/implementation, different stages of the policy cycle being managed by different services or institutions, etc.

Based on the experience from past evaluations and the feedback from stakeholders, the two tables below include examples of respectively short-term and medium-term actions that can help to ensure that evaluation will be considered along the policy cycle, and not perceived or delivered as a one-off standalone activity or a collection of self-contained steps (risks identified by Giorgi 2017).

3.2 – Timing issue

Matching the timeframes of evaluation and decision (or consultation) processes is a recurrent challenge for evaluators, who often mention the lack of time as one of the key difficulties they face.

Giorgi (2017) analysed that one way to tackle this challenge is to work on a mutual understanding between policy officers and evaluators.

“With an open acknowledgement from both sides, this timing mismatch, according to interviewees, can be addressed. External evaluators, researchers and/or civil service analysts, on the one side, need to be confident in sharing emerging findings and insights live at that point in time through outlets such as learning sessions. The idea being that emerging findings are ‘good enough’ and tell the narrative of the ‘impact at this point in time’ to feed into policy development decisions. Policymakers, on the other side, need to feel comfortable with these caveats and with the risk that final results and end conclusions may differ when all the data has been collected and analysed.” (Giorgi, 2017)

3.3 – Preparation

For general guidance about how to prepare an evaluation, see the sources mentioned in part 3 |. We deal here with aspects of the preparation that can have an interaction with the policy cycle, and that were highlighted by stakeholders about their experience with evaluations. See also section 6.5 about the aspects related to the organisation of the evaluation.

4 – Legitimacy and credibility

As mentioned in section 6.1, a successful integration of evaluation into the policy cycle requires political will or stakeholders’ interest in evaluation. Another key factor is how evaluation is perceived by stakeholders, and particularly in terms of legitimacy and credibility. Both are needed for stakeholders to have confidence in the evaluation results (see part 4 of Broc et al. 2018).

The legitimacy of the evaluation and the credibility of its results are essential for evaluation findings to be discussed fairly by the different parties, and then to be used when considering changes in the policy or other decisions.

When stakeholders have confidence in the evaluation results, this also helps strengthening the legitimacy of the policy and related decisions. This can then translate into securing or increasing the funding for the scheme (see Irish case study) or the involvement of stakeholders such as participants to voluntary agreements (see Finnish case study).

The level of confidence may depend on how stakeholders perceive the credibility of the evaluators, the relevance and reliability of the evaluation methodology, if they were involved in the evaluation process, if the results are transparent (not only public, but also well documented).

The two tables below suggest actions to create conditions to ensure legitimacy and credibility.

^* About the independency issue, feedback from the interviews and online surveys done for the EPATEE project has shown different points of view and interpretation of what an independent evaluation would be, and how to ensure independency of the evaluators. To summarize, commissioning external evaluations is one, but not the only way. Alternatives can be that evaluations be done by a distinct service or institution than the one(s) in charge of deciding, funding and implementing. The possibility for an independent public body (e.g. Court of Auditors) to do evaluations on its own initiative also proved to provide good guarantees.

^** About the documentation issue, in most cases, evaluators will not publish all the details of their methodology (e.g. about modelling, sampling technics). Either due to intellectual property, commercial confidentiality, or because this would lead to too long and indigestible reports. However, a minimum documentation is always possible to provide a view on the main aspects of the methodology and data issues. This is essential to keep the memory of the evaluation, avoid misinterpretation of the results, and enable a review by external experts.

5 – Organisation

5.1 – Clarify the roles, responsibilities, contacts and possibilities to contribute

A clear organisation helps the evaluation process to be well identified by the different services, bodies, stakeholders involved in the policy cycle. This is thus a prerequisite to facilitate the integration of evaluation activities into the policy cycle. Aspects about planning are dealt with in section 3. This section is focused on the roles and responsibilities of the different parties who can be involved in the evaluation process.

Experience shows that clearly assigning the role to coordinate or supervise the evaluation to a central contact creates favourable conditions for a partnership between evaluators and practitioners (i.e. persons in charge of or involved in implementing the policy).

“Such a set-up ensured a clearly defined role and set of responsibilities providing a sense of ownership and a ‘stake in’ the outcomes to those running the evaluation. This mutually beneficial partnership approach seemed to work best with clear delegation, factoring in evaluation into delivery activities from the onset and, where relevant and appropriate, using specialist organisations.” (Giorgi, 2017)

Another key aspect of the organisation of the evaluation is the way the different parties (policymakers, policy officers, delivery practitioners, etc.) will be involved in the evaluation process. As reminded by Giorgi (2017), “the policy cycle is not the sole responsibility of the policymakers, it needs to be shared and owned across all relevant parties.” Which also applies for evaluation that is not the sole responsibility of the evaluators (or evaluation commissioners).

This implies to clarify the roles and responsibilities, but also to create favourable conditions for interactions between parties, opportunities for discussions, etc.

Such organisation leads to the evaluation not being perceived as a process separate from the policy cycle or developments, but as a part of it. Moreover, as highlighted by Giorgi (2017), “a wider sense of purpose and belonging in an evaluation can help nurture buy-in”.

5.2 – Identify possible synergies with existing processes

A frequent feedback from stakeholders about evaluation is that it should avoid to create additional administrative burden. And that it should as much as possible take advantage of existing tools, processes, etc.

Discussions with all parties (on short term) and a review of the different processes of the policy cycle (on medium term) can help identify opportunities of synergies. Synergies can be particularly helpful to optimize data collection. See also section 3 about planning.

6 – Communication and mutual understanding

An evaluation can involve people with different backgrounds (e.g. statistics, economics, engineering, sociology) and having different experience / connections with the policy evaluated. They can thus have different perceptions or views about what evaluation means, what it should be. Likewise about the objectives of the policy, its delivery scheme or its results. Ensuring a good communication can therefore be challenging, while it is essential for information sharing and to get a complete picture of the policy. Interactions / exchanges between the different parties, and particularly between policy officers (and policymakers whenever possible) and evaluators or policy analysts, are ways to favour the integration of evaluation into the policy cycle, and to avoid evaluation to be perceived as a fully separate process.

^* the “right” contact is not only about finding the contact for the communication link. It is also about finding the contact that is open to the evaluation process, as explained by one of the respondents to the second EPATEE online survey:

“Finding the right people to work together as a team. One person can easily dominate discussions; you need people who are willing to learn, rather than to say that they already know the answers.” (see Bini et al. 2018)

7 – Communication about the evaluation and its results

The review of the EPATEE case studies highlighted that the communication about the evaluation and its results can be as important as doing the evaluation (see part 12 of Broc et al. 2018).

Making it clear from the start of the evaluation, what the audience will be, to whom the evaluation report will be communicated, to whom the main findings will be communicated (and how), and most importantly what the evaluation findings will be used for, all this has proven to be effective ways to motivate the different parties (including the evaluators) to take actively part in the evaluation process.

Anticipating debates or misinterpretations of the results is also important to avoid misuse of evaluation findings. At the opposite, adapting the communication format to the targeted audience can increase the attention given to the evaluation findings, thereby increasing their use.