Interact Logo Sharing Expertise

Practical Handbook for Ongoing Evaluation

2. What is ongoing evaluation and why is it needed?

"Many good programmes do not always add up to good policy, good programme documents do not necessarily translate into good projects and good projects do not always ensure the success of a programme" (Evalsed)

2.1. Evaluation in general: why do we need evaluation?

2.1.1. Evaluation is part of the programme life cycle

Evaluation provides reflection on the performance of the programme and enables the programmes to receive independent feedback on the relevance, effectiveness, efficiency and/or consistency of the programme. Evaluation enables us A) to learn internally: involved stakeholders are able to learn and improve their programme, and B) to present externally: the results of the programme can be presented to a broader public by an independent evaluator.

Evaluation is an important element in the European programme cycle (see diagram below). Each programme follows the cycle of programme development, identification of projects, appraisal of projects, financing of the projects, implementation of the programme through the projects and evaluation of the programme.

Diagram showing programme life cycle for European programmes

Source: European Commission

Evaluation provides input into the programme cycle at several stages: when the programme is being developed, an ex ante evaluation checks that the programme complies with the European requirements and is relevant, effective and efficient in its set-up. Without a report from the ex ante evaluator a programme will not be approved by the European Commission. Furthermore, evaluation is useful to reflect on a programme's progress towards achieving its objectives and to enable strategic decisions to be made during the programme (ongoing evaluation). Ongoing evaluation can influence both the scope of the programme and the selection of projects, as well as programme implementation arrangements. Finally, evaluation can draw lessons from implementation and assess the results and impact of a programme (ex post evaluation).

Experience of Bernard Schausberger, Cross-Border programme Austria-Slovakia:
Consider the evaluation as a process with opportunities to stop, reflect, etc.

Experience of Maria Eriksson, Cross-Border programmes Sweden:
Ongoing evaluation should be used as a tool for continuous learning and exchange of experience between projects, actors and programmes.

Tip from Evalsed:

Evaluations must be fully integrated into programme planning and management. Programme managers need to think of evaluation as a resource: a source of feedback, a tool for improving performance, an early warning of problems (and solutions) and a way of systematising knowledge. Evaluation is not simply an external imposition.

The European Commission uses three different types of evaluation (see Annex for the text from the Regulations):

  • Ex ante evaluation. Ex ante evaluation is compulsory for each Territorial Cooperation programme. An Operational Programme has to include a summary of the findings of the ex ante evaluators; otherwise the European Commission will not approve the programme. Ex ante evaluation verifies the relevance of the programme and provides insight into its foreseen effectiveness, efficiency and durability. It also ensures that the programme is in line with the European requirements. For the programmes of the Structural Funds programming period 2007-2013 the ex ante evaluations were mostly completed during 2006-2007.
  • Evaluation during the programme period (hereafter ongoing evaluation). Each Territorial Cooperation programme has to carry out ongoing evaluation to assess the programme, either from a strategic or an operational perspective or both. This can be either a 'traditional' mid-term evaluation half way through the programme or an evaluation at another juncture with a specific or overall content. This new evaluation culture means that for the first time each programme is now flexible in deciding on the design, timing and content of the ongoing evaluation.
  • Ex post evaluation. Ex post evaluation is not compulsory for Territorial Cooperation programmes. It is only compulsory for the European Commission to perform an ex post evaluation of all programmes under the Territorial Cooperation objective, which has to be completed by December 2015.

Experience of Ieva Kalnina, INTERACT, and Marie-Jose Zondag, evaluation expert:
How to compensate for the 'missing link' – the non-availability of ex post evaluation for the Territorial Cooperation programmes? Suggestions: 1) Reserve budget in the Evaluation Plan for ex post evaluation; 2) Focus on the results of the projects. Combine an ex post evaluation with a publication on the results of the operations, the lessons learnt and suggestions for improvement. North West Europe has done this for the 2000-2006 results, for example.

2.1.2. Distinguishing between monitoring and evaluation Monitoring is the ongoing process of collecting and using standardised information to assess progress towards objectives, resource usage and achievement of results and impacts. Regular monitoring should provide detailed operational information (mainly on outputs and results achieved by the projects and the programme, financial absorption and the quality of implementation mechanisms) and allow for detection of warning signs. It usually involves assessment against agreed indicators and targets. In conjunction with evaluation information, effective monitoring and reporting should provide decision-makers and stakeholders with the knowledge they need to identify whether the implementation and outcomes of a programme are unfolding as expected and to manage the initiative on an ongoing basis Monitoring forms the basis for evaluation and should provide valuable data for discussing the need for, timing and focus of evaluation at programme level.

Not all information can be provided by or extracted from the monitoring system, which is why follow-up in the form of evaluation is required. Monitoring data rather serves as a source of initial/additional information to be further processed and used for analysis and reporting on strategic aspects dealt with by evaluation. When a programme has well-functioning monitoring systems and relevant data collected, an evaluation will require less budget resources.

It is therefore important to think in advance what information has to be collected by the programme, particularly from the projects, and establish which data will serve what purpose. With regard to using the monitoring data in evaluation processes, a decision has to be taken on what data from the monitoring system will be made available to the evaluators (all, part, none).

Experience of Anke Mollers, Transnational programme North West Europe:
During monitoring we very often notice that the projects interpret and quantify the indicators differently. A common understanding is necessary in order to produce comparable and useful results. This is also relevant for the evaluation, as the latter very often builds on monitoring.

Experience of Marie-Jose Zondag, evaluation expert:
Identify in good time which data are missing in the monitoring system and whether an evaluation could be needed/useful to identify this information.

2.2. European requirements: ongoing evaluation is required but flexible

The term 'ongoing evaluation' is new in 2007-2013 and stands for a new evaluation culture. In the 2000-2006 period all programmes had to carry out a mid-term evaluation, but this is no longer necessary. The whole approach to evaluation has become more strategic and flexible in the 2007-2013 period. The European Council Regulation 1083/2006 requires ongoing evaluation to be carried out, but its scope, time frame and extent are to be decided by the programmes themselves.

The only cases specified in the Regulation 1083/2006 (Article 48.3):

  • A programme shall carry out an evaluation when the monitoring of Operational Programmes reveals a significant departure from the goals initially set;
  • A programme shall carry out an evaluation when Operational Programme revisions are proposed.

Where proposals are made for the revision of operational programmes, as referred to in Article 33, analyses shall be provided on the reasons for the revision, including any implementation difficulties, and the expected impact of the revision, including that on the strategy of the operational programme.

The European Commission sees ongoing evaluation as a process taking the form of a series of evaluation exercises. Its main purpose is to follow the implementation and delivery of an Operational Programme and changes in its external environment on a continuous basis, in order to better understand and analyse outputs and results achieved and progress towards longer-term impacts, as well as to recommend, if necessary, remedial actions.

This flexibility in terms of scope, time frame and content enables the programmes to develop an evaluation according to their own wishes and needs. With this approach the European Commission is trying to ensure that evaluation becomes an integral part of the policy and programme implementation process, thus promoting greater effectiveness and efficiency of European interventions. In this way programmes can organise evaluation so that it answers their needs. The learning aspect of evaluation is therefore stressed by the European Commission, to fine-tune the strategic focus towards the future and/or to improve programme management and implementation.

2.3. Suggested approach

2.3.1. Flexible ongoing evaluation requires a pro-active approach by programmes

One of the key words in the concept of ongoing evaluation is 'flexibility'. Programmes no longer need to be evaluated at a fixed point in the programme life-cycle, and evaluation is obligatory only when changes in the programme are foreseen or where the programme is departing significantly from the objectives set. This enables Territorial Cooperation programmes to decide whether to pursue one or a series of evaluations and what kind of evaluations to carry out, e.g. thematic, cross-programme, strategic, operational, or partial evaluations. Therefore a strategic approach, planning and a pro-active approach on the part of the programme becomes very important: When to perform evaluation? What kind of evaluation to perform? How best to fit evaluation into the programme time frame? How to link evaluation with monitoring at programme level? Evaluation should be undertaken and designed in accordance with the internal needs of the programme. Actual or potential difficulties (or any other information) revealed by monitoring could give rise to an evaluation. Evaluation could also be undertaken to review strategic and operational aspects which cannot be analysed on the basis of monitoring data alone and/or which require a neutral (outsider's) view of what's going on.

Documents and activities are evaluated in order to improve their quality, efficiency and the coherence of their intervention. Ongoing evaluation is the responsibility of the Territorial Cooperation programme. It is based on the principle of proportionality (see below), carried out by independent evaluators (either internal or external experts), and the results are made (partially or completely) public.

In this respect, programmes (Joint Monitoring Committee, Managing Authority and Joint Technical Secretariat) need to be pro-active, both in ensuring strong links between monitoring and evaluation and in defining the periodicity/regularity of these exercises in order to guarantee continuous delivery of information and analysis to be used for management purposes. Establishment and further development of Evaluation Plans is strongly recommended by the European Commission. As the Working Document from the European Commission indicates the proposed flexible approach emphasises the need for stronger links between monitoring and evaluation on the one hand, and between these two often interlinked exercises and decision-making on the other. The section 'The difference between monitoring and evaluation' goes into more detail on the complementarity between the two.

2.3.2. Key evaluation issues

When an evaluation is being performed, the core issues to evaluate are the relevance, effectiveness, efficiency and consistency of the programme. The European Commission's Working Document No. 5 also mentions these aspects, which are explained below with more specific reference to Territorial Cooperation programmes.

  • Relevance: The relevance aspect checks if the programme is still relevant in the policy context, if it is dealing with the issues that are actually needed, as defined in the SWOT Is this programme useful and helping those for whom it is intended? Have there been any new developments or changes since the SWOT was conducted? What social, economic and environmental changes have taken place in the programme area (natural disasters, economic crisis, domestic policy changes, etc.)? How do these changes affect the programme's objectives? The expected relevance is checked in the ex ante evaluation. The ongoing evaluation assesses whether the relevance is still as expected, or if there are changes in the programme or its context which influence its relevance. Additional questions to check the relevance of Territorial Cooperation programmes and their activities are, for example:
    • The added value of cooperation: To what extent was cooperation necessary to achieve the effects? What has been the intensity of cooperation? What elements has cooperation added to the programme's activities and achievements? How relevant was the cooperation? Was the intervention additional to what would otherwise have taken place?
    • Innovativeness: How innovative are the programme and its operations? What new approaches have been deployed or results produced? Were the operations actually innovative and relevant?
    • Synergy: Has the programme in any way complemented and enhanced the effect of other related domestic policies or macro-regional strategies?
  • Effectiveness: The effectiveness aspect checks whether the programme is actually having an effect, if the foreseen objectives are being achieved, and if the actions taken are appropriate in order to obtain an effect. To check the effectiveness an analysis is made of output and results and an assessment of their compliance with the expected objectives in order to understand why there are or may be varying degrees of success. What has been the actual effect of the programme? What is the programme's progress towards achievement of the objectives? Are the actions taken appropriate to produce the desired and foreseen effect? Could more or better effects be obtained by using different instruments or actions? Are the outputs, results and impacts well defined to achieve the objectives of the programme? Have there been (un)successful actions with a significant effect? Again, an initial check will have been done in the ex ante evaluation. The ongoing evaluation will look at the actual progress and delivery. Effectiveness of Territorial Cooperation programmes is to be evaluated through e.g.:
    • The effect on territorial cohesion: In what way has the programme contributed to the territorial cohesion of the programme's territory and its policy?
    • Durability of cooperation: Are the results and impacts lasting? How durable are they over time? Will the impacts continue if there is no more public funding? Will the cooperation continue, and in what form?
    • Utility: Does the programme have an impact on the target groups or populations in relation to their needs?
  • Efficiency: The efficiency aspect checks if the programme is well managed, if it delivers value for money and if the time is used efficiently. Processes and the means and resources mobilised are compared. How well are the programme's resources managed? Could better effects be obtained for the same cost? Is the programme well managed and delivered in an efficient and legitimate way? This is a crucial but often very sensitive issue requiring constant verification through both monitoring and evaluation processes. Efficiency of Territorial Cooperation programmes can be evaluated by looking at e.g.:
    • Efficiency of cooperation: Has the programme been efficient in using for example auditors, translators, meetings, travel costs, transactions, reporting? Have cultural obstacles been dealt with efficiently? What procedures could be simplified to avoid cultural obstacles or inefficient cooperation?
  • Consistency: The consistency aspect checks if the following verifications are performed: 1) that the priority axes are complementary, not overlapping and contributing to the objectives of the programme and 2) that the programme does not overlap with other policy initiatives or programmes. Initial verifications should be performed in the ex ante exercise and further verifications should take place during the project selection and decision-making processes, but there may also be a need to analyse the programme framework at certain stages during its implementation – particularly if major changes have taken place in the programme territory or general environment.

Overview of key evaluation issues and specific territorial issues, brief explanation and stages of evaluation

Source: INTERACT, on basis of European Commission documents

2.3.3. Main principles of evaluation

The evaluation issues represent the main issues to be evaluated. In order to ensure that an evaluation is performed under good conditions, the Commission recommends following four main principles that govern any evaluation process:

  • Proportionality (Article 13, 1083/2006): Proportionality means that activities should be in proportion to the scale and resources of a programme. Proportionality should be reflected, for example, in the Evaluation Plan by the number and scope of evaluations proposed during programme implementation.
  • Independence (Article 47.3, 1083/2006): To ensure credible results, evaluations should be carried out by experts or bodies (internal or external) that are functionally independent of the certifying and audit authorities and preferably also of the programme's operational management (Joint Technical Secretariat/Managing Authority) and decision-making bodies (Monitoring Committee). The independence of evaluation can also be enhanced by including various stakeholders in the Evaluation Steering Groups.
  • Partnership (Article 11, 1083/2006): Consultation and participation of stakeholders in the evaluation process (planning, evaluation, reports and follow-up) is important for a good evaluation as it ensures learning, openness and transparency. The involvement of the stakeholders can take place at different levels:
  • As a minimum the evaluators should make sure that stakeholders provide evidence (data, information, judgements, etc.) during the evaluation process.
  • Alternatively, the stakeholders can be involved in steering the whole evaluation exercise. This is best achieved through their involvement in the Evaluation Steering Group.
  • Transparency: Publishing evaluation reports is recommended by the European Commission in order to stimulate public debate on evaluation findings. The easiest way is to publish the entire evaluation report on the programme's website.
  • Usefulness: Evaluation is a process set up by the programmes to obtain useful information. From the planning phase onwards, the programme authorities should think about the intended use of the evaluation outcomes and follow-up action. Thus there are three elements to usefulness:
  • The evaluation activities themselves should be useful. Will the right information be evaluated, the right questions posed? Is it useful to have a face-to-face interview or will phone interviews or a questionnaire be sufficient? What information is it actually useful to obtain from projects?
  • How to make the evaluation useful afterwards? How can the evaluation be used for publicity? What publicity actions will be useful for presenting the results of the programme?
  • How to ensure that the recommendations will actually be dealt with? What kind of follow-up action plan will be used to ensure that the results of the evaluation are actually used?

The following chart provides an overview of the key evaluation issues, related questions and their use in different stages of evaluation.

Overview of evaluation principles, brief explanation and stage of evaluation

Source: INTERACT, on basis of European Commission documents

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.