"Evaluation design must provide clear and specific objectives and appropriate methods and means for managing the evaluation process and its results." (European Commission, SEC(2007)213)
Ensuring a good and useful evaluation is very much linked to good Terms of Reference (ToR), planning and a clear idea from the Monitoring Committee of what they want to do with the evaluation. This part therefore provides some practical guidance on how to develop good Terms of Reference. It focuses mainly on Terms of Reference for external experts. For internal evaluation the Terms of Reference will be more compact.
There are no requirements regarding the Terms of Reference in the European Regulations, except for the stipulation that "the Member States shall provide the resources necessary for carrying out evaluations, organise the production and gathering of the necessary data and use the various types of information provided by the monitoring system" (Art 48.1 Council Regulation No 1083/2006).
Evalsed and the European Commission's Working Document No. 5 provide clear guidance on the content and process of an evaluation. Information from these documents is included in this text. Furthermore, national and European procurement rules have to be followed.
The Terms of Reference (ToR) are the basis of a contractual relationship between the programme and the evaluation experts responsible for carrying out the task. The Terms of Reference are essential, irrespective of whether evaluation is carried out by external or internal evaluators. The Terms of Reference normally specify the scope of an evaluation and state the main motives and the questions to be answered. They summarise the available information and outline a possible approach or minimum requirements, leaving scope for suggestions from the evaluators. The Terms of Reference describe the distribution of tasks and responsibilities among the people participating in the evaluation process (Steering Group, evaluators, beneficiaries). They fix the time frame and the budget. They specify the qualifications required of the evaluation team as well as the criteria to be used to select an evaluation team. The Terms of Reference should be brief (typically 5-10 pages), supplemented if necessary by administrative annexes. An example is included in the tools section.
The steps involved in developing Terms of Reference and the contracting process
Step 1. Decision to evaluate
The decision to start an evaluation will be taken by the Monitoring Committee of the Territorial Cooperation programme. INTERACT recommends ensuring the commitment of the Monitoring Committee and thus the involved countries/regions by asking them at least the following four questions:
INTERACT suggests establishing an Evaluation Steering Group to guide the evaluation process, including the writing of the Terms of Reference, the procurement procedure, the selection process and the actual evaluation. This Evaluation Steering Group could consist of people from the programme (Joint Technical Secretariat and possibly Managing Authority).
Step 3. Writing of Terms of Reference
INTERACT recommends drawing up a timetable for the writing of the Terms of Reference, their publication, and the start and completion dates of the evaluation. Furthermore, a person or team should be selected by the Joint Technical Secretariat/Managing Authority/Monitoring Committee to write the Terms of Reference. This could be people from the Programme or external experts. INTERACT recommends that the Terms of Reference are developed in cooperation with the stakeholders to ensure a useful evaluation. Cooperation programmes must include persons from all participating Member States, e.g. Monitoring Committee members, but also technical persons responsible for the collection of data, monitoring systems, etc.
For further guidance on how to write the Terms of Reference, see the Tools section 'Example structure and content of Terms of Reference'.
Experience of Marie-Jose Zondag, evaluation expert:
Experience of Agnieszka Gintowt-Dziewaltowska, Polish Ministry of Regional Development and Marie-Jose Zondag, evaluation expert:
How to increase interest on the part of potential evaluators and minimise the risk that no proposals will be submitted in response to our call for tenders? Suggestions: 1) When the decision to launch the particular evaluation is taken, it is a good idea to post an announcement about the forthcoming evaluation on the website, giving some information about the subject and scope of the planned research. Evaluation companies will then have more time to think about their proposals, methodology, time schedule; 2) Ensure that the budget is sufficiently appealing for experts to respond; 3) Ensure that experts have sufficient time to develop an adequate proposal.
Experience of Marie-Jose Zondag, evaluation expert:
National procurement rules have to be followed, and if the service contract is worth more than €°137°000 European procurement rules have to be followed. However, the rules differ from country to country. There are four possible procedures :
A. Open procedure: any interested economic operator may submit a tender .
B. Restricted procedure: any economic operator may request to participate and only candidates invited to do so may submit a tender .
C. Negotiated procedure: the contracting authority consults the economic operators of its choice and negotiates the terms of the contract with them. In some MS negotiated procedures are possible and sometimes even more appropriate for the selection of the evaluator, as the content and quality of the evaluation very much hinges on the personal quality/qualifications of the evaluator.
D. Framework contract.
Experience of Bernard Schausberger, Cross-Border programme Austria-Slovakia:
The Evaluation Steering Group guides the selection of the proposals. Some tips from INTERACT to ensure that the right evaluators are selected:
The template below is based on the structure used in Evalsed, the European Commission's Working Document No. 5 and the experience of the evaluation expert. As there are no minimum requirements for the Terms of Reference the template is an example that can be adapted to the programmes' needs.
Section 1. Objective and scope of the evaluation
The Terms of Reference start by explaining:
For an evaluation to be useful, it should be formulated as precisely as possible. Often contractors, not wanting to influence the evaluation team too much, are reluctant to express in advance the changes they think should be made or their doubts concerning the effectiveness of a particular action. It is better to avoid having to ask the evaluator for extra work which would raise the costs of the evaluation. Experience shows that it is better to be as precise as possible.
Section 2. Main users and stakeholders of the evaluation
Who are the individuals, groups or organisations that have an interest in the intervention to be evaluated or who might be interested in the process or in the results of the evaluation itself?
Section 3. Evaluation questions
Check if the formulated questions can actually be answered in the evaluation. 1) Is the data required to answer the question available? 2) Are there already any results that can be evaluated? In 2000-2006, for example, many mid-term evaluations could hardly evaluate the effectiveness of the operations as most of them had only just started. 3) Is the question clear? If the question can be interpreted in different ways the programme might get a proposal that does not match its expectations. 4) Is it necessary to have an evaluation to answer this question, or could the monitoring system or a discussion be sufficient? This check may lead to a decision not to undertake the evaluation, to postpone it, or to revise the questions.
What types of evaluation questions are possible?
Evaluation questions can pertain to different levels. They can be:
Experience of Kai Böhme, evaluation expert:
The Terms of Reference should contain a review of the current status of information on the programme and its effects. This will include which documents are available, what analysis has already taken place and some brief extracts when needed to prepare a good proposal. INTERACT suggests indicating who is in charge of providing information from, for example, the monitoring system. Furthermore, it is important to mention if certain information/data are not available and will have to be collected by the evaluator to ensure a realistic calculation of the offer.
Section 5. Main methods or techniques to be used
If the Evaluation Steering Group has clear ideas on the methods or techniques to be used, these can be included in the Terms of Reference. It is a good idea to indicate whether the programme would prefer the evaluation team to use this approach or if they are open to other approaches.
In general the approach will be as follows: 1) identify what needs to be done, 2) data collection, 3) analysis of data, 4) provide judgement.
The choice is generally made to maintain sufficient flexibility to allow those responding to the Terms of Reference to differentiate in terms of the relevance and clarity of their methodological proposals. This is especially important in the selection phase because assessing the methodological qualities of the proposals is a crucial step in selecting the right evaluator. When possible from an administrative point of view, the best way is to determine a budget (see below) and describe only the main outlines of the method in the Terms of Reference; then select the team that proposes the most promising method. Those selecting the team will then need to be capable of judging the methodological quality of a tender.
Evalsed provides an overview of possible methods and techniques of evaluation. Here are some suggestions:
Section 6. Time schedule & reporting
INTERACT recommends indicating the time period of the evaluation and important dates – including procurement deadlines - in the Terms of Reference. If an evaluation is linked to certain events or decision-making moments, it is a good idea to mention this and to take some extra time to ensure that the evaluation will be ready in time.
If the meetings of the Evaluation Steering Group have to take place on certain dates and/or in specific locations, this should also be indicated in the Terms of Reference.
INTERACT recommends indicating in the Terms of Reference what type of report(s) the programme expects. What should be included in the draft final report? Will there be several interim or specific reports? What will the evaluation report be used for? Who are the target group? Should the evaluation contain a summary for a broader audience? In what way does the programme want to make use of the evaluation and what type of report(s) is therefore requested? How will this report link to the annual report? Would the programme like to have recommendations in the field of 1) finance, 2) content and/or 3) implementation?
Section 7. Indicative budget
It is good practice to suggest an indicative budget and then leave those competing for an evaluation by open tender to suggest what they would be able to provide for the budget available. This allows value-for-money assessments to be made. It also provides the contractor of the evaluation with greater control over expenditure. Some Member States require the maximum budget to be published.
Section 8. Required qualifications of the team
It is a good idea to reflect in advance what criteria are important for the programme. Here are some possible choices which could help to develop a list of required qualifications:
Be realistic in the demands. The higher the qualifications the programme asks for, the more limited the number of potential proposals will be. How important are the qualifications of the evaluators? In any case the quality of the proposal should be more important than experience as such.
Experience of Marie-Jose Zondag, evaluation expert:
The Terms of Reference could specify the deadline, the modes of transmission (post, fax, e-mail), the language in which the tender must be drawn up, how long the offer should remain valid, etc. It should also indicate the criteria according to which the proposals will be judged. Especially for larger evaluations it can be useful to indicate the required structure of the proposal. The Terms of Reference should state, for example, the relative importance in percentage points that will be given to:
It is recommended to make a clear list of assessment criteria for the proposals and to include these (in a summarised form) in the Terms of Reference. Many proposals use the assessment model shown below. In some cases:
Some suggestions regarding the assessment team have been provided in previous sections.
Assessment grid for '… [title of evaluation]' (… [reference number])
Name (or number) of proposal/evaluator: …
Score on content: