Interact Logo Sharing Expertise

Practical Handbook for Ongoing Evaluation

6. How to develop Terms of Reverence for evaluation?

"Evaluation design must provide clear and specific objectives and appropriate methods and means for managing the evaluation process and its results." (European Commission, SEC(2007)213)

Ensuring a good and useful evaluation is very much linked to good Terms of Reference (ToR), planning and a clear idea from the Monitoring Committee of what they want to do with the evaluation. This part therefore provides some practical guidance on how to develop good Terms of Reference. It focuses mainly on Terms of Reference for external experts. For internal evaluation the Terms of Reference will be more compact.

6.1 European requirements & European guidance

There are no requirements regarding the Terms of Reference in the European Regulations, except for the stipulation that "the Member States shall provide the resources necessary for carrying out evaluations, organise the production and gathering of the necessary data and use the various types of information provided by the monitoring system" (Art 48.1 Council Regulation No 1083/2006).

Evalsed and the European Commission's Working Document No. 5 provide clear guidance on the content and process of an evaluation. Information from these documents is included in this text. Furthermore, national and European procurement rules have to be followed.

6.2 Possible approach

The Terms of Reference (ToR) are the basis of a contractual relationship between the programme and the evaluation experts responsible for carrying out the task. The Terms of Reference are essential, irrespective of whether evaluation is carried out by external or internal evaluators. The Terms of Reference normally specify the scope of an evaluation and state the main motives and the questions to be answered. They summarise the available information and outline a possible approach or minimum requirements, leaving scope for suggestions from the evaluators. The Terms of Reference describe the distribution of tasks and responsibilities among the people participating in the evaluation process (Steering Group, evaluators, beneficiaries). They fix the time frame and the budget. They specify the qualifications required of the evaluation team as well as the criteria to be used to select an evaluation team. The Terms of Reference should be brief (typically 5-10 pages), supplemented if necessary by administrative annexes. An example is included in the tools section.

The steps involved in developing Terms of Reference and the contracting process

Source: INTERACT

INTERACT suggests taking the following steps (see the overview above):

Step 1. Decision to evaluate

The decision to start an evaluation will be taken by the Monitoring Committee of the Territorial Cooperation programme. INTERACT recommends ensuring the commitment of the Monitoring Committee and thus the involved countries/regions by asking them at least the following four questions:

  • Reasons for the evaluation? It is recommended to have a clear link to e.g. a particular problem, a strategic objective, policy issues, the future of the programme, etc.
  • Who is in charge of the overall exercise? Who should be involved in the Evaluation Steering Group, who will write the Terms of Reference, who will liaise between the administration and evaluation teams? They should be senior enough to have direct access to the policy-makers in order to share with them the knowledge that the study will produce.
  • What budget will be made available for the evaluation?
  • Who should perform the work? Internal or external evaluators (see below)?
  • How will the results be used?
Tip from Evalsed:Consideration should be given at an early stage to how evaluation findings will be put to use. Evaluations can also be helpfully combined with other evidence to inform debates. (Evalsed).


Step 2. Organisation of the evaluation process

INTERACT suggests establishing an Evaluation Steering Group to guide the evaluation process, including the writing of the Terms of Reference, the procurement procedure, the selection process and the actual evaluation. This Evaluation Steering Group could consist of people from the programme (Joint Technical Secretariat and possibly Managing Authority).

Step 3. Writing of Terms of Reference

INTERACT recommends drawing up a timetable for the writing of the Terms of Reference, their publication, and the start and completion dates of the evaluation. Furthermore, a person or team should be selected by the Joint Technical Secretariat/Managing Authority/Monitoring Committee to write the Terms of Reference. This could be people from the Programme or external experts. INTERACT recommends that the Terms of Reference are developed in cooperation with the stakeholders to ensure a useful evaluation. Cooperation programmes must include persons from all participating Member States, e.g. Monitoring Committee members, but also technical persons responsible for the collection of data, monitoring systems, etc.
If the programme knows exactly what is needed, INTERACT advises to indicate this as precisely as possible in the Terms of Reference (what the evaluators have to do and what NOT to do). This makes it less likely that you will receive proposals for activities other than those the programme is aiming for.

For further guidance on how to write the Terms of Reference, see the Tools section 'Example structure and content of Terms of Reference'.


Experience of Marie-Jose Zondag, evaluation expert:
What to do when the programme does not really know what to evaluate and how to perform an evaluation? Suggestions: 1) Invite a few evaluators to brainstorm on ideas; 2) Hire an external expert to write the Terms of Reference in close cooperation with the programme.

Experience of Agnieszka Gintowt-Dziewaltowska, Polish Ministry of Regional Development and Marie-Jose Zondag, evaluation expert:

How to increase interest on the part of potential evaluators and minimise the risk that no proposals will be submitted in response to our call for tenders? Suggestions: 1) When the decision to launch the particular evaluation is taken, it is a good idea to post an announcement about the forthcoming evaluation on the website, giving some information about the subject and scope of the planned research. Evaluation companies will then have more time to think about their proposals, methodology, time schedule; 2) Ensure that the budget is sufficiently appealing for experts to respond; 3) Ensure that experts have sufficient time to develop an adequate proposal.

Experience of Marie-Jose Zondag, evaluation expert:
Link the design of the evaluation directly to the decision(s) to be taken for the further implementation of the programme.



Step 4. Publication of Terms of Reference

National procurement rules have to be followed, and if the service contract is worth more than €°137°000 European procurement rules have to be followed. However, the rules differ from country to country. There are four possible procedures :

A. Open procedure: any interested economic operator may submit a tender .

B. Restricted procedure: any economic operator may request to participate and only candidates invited to do so may submit a tender .

C. Negotiated procedure: the contracting authority consults the economic operators of its choice and negotiates the terms of the contract with them. In some MS negotiated procedures are possible and sometimes even more appropriate for the selection of the evaluator, as the content and quality of the evaluation very much hinges on the personal quality/qualifications of the evaluator.

D. Framework contract.


Experience of Bernard Schausberger, Cross-Border programme Austria-Slovakia:
It should be stressed that the tendering procedures at least have to comply with the national procurement rules of the seat of the Joint Technical Secretariat/Managing Authority. If the in-house experience on procurement law is not 100%, a procurement expert should be consulted.


Experience of Bernard Schausberger, Cross-Border programme Austria-Slovakia:
Good Terms of Reference have to be based on very concrete perspectives regarding the scope of the reports, the number of interviews to be performed, the number, format and location of meetings: the Evaluation Steering Group should devote sufficient time to developing the work steps and outputs of the evaluation; if all these things are still very vague a framework contract might be the better option.



Step 5. Selection of winning proposal

The Evaluation Steering Group guides the selection of the proposals. Some tips from INTERACT to ensure that the right evaluators are selected:

  • Include clear, transparent criteria and procedures in the Terms of Reference.
  • Ask around about good evaluators, check who has done an evaluation before and what the experience was with this evaluator.
  • Agree in advance who will select the proposal. It could be the Evaluation Steering Group but they could ask others to assess the proposals. This should include the people in charge of the evaluation, representatives of the (potential) users of the evaluation and sometimes an independent expert. Ensure that people with experience are included (especially for the quality assessment).
  • Agree in advance which procedure will be used. Will each person make their own judgement, after which the scores will be combined and the average taken as the result? Or will there be a discussion to decide jointly on the score? What if there are conflicting assessments?
  • Agree in advance which criteria will be used for judging the quality and experience of the team and the price. The qualifications and previous experience of the team are always important, and especially so if the methods proposed are experimental or do not completely fulfil the specifications set forth in the Terms of Reference. Also, knowledge of the local/regional cultures and languages may be an additional criterion. An example Assessment Form is included in the subsection 'tools'. Assessing the proposed price for the services is an important aspect of the selection process, but should not be overestimated. As a rule of thumb for any evaluations that are not entirely routine, the financial criterion should not exceed 20-25% of the overall assessment.

6.3 Tool: Example format for structure and content of Terms of Reference

The template below is based on the structure used in Evalsed, the European Commission's Working Document No. 5 and the experience of the evaluation expert. As there are no minimum requirements for the Terms of Reference the template is an example that can be adapted to the programmes' needs.

Template for structure and content of Terms of Reference

Source: INTERACT

Section 1. Objective and scope of the evaluation

The Terms of Reference start by explaining:

  • Which programme and programme period is the subject of the evaluation?
  • What is the purpose/objective of the evaluation?
  • What is the scope of the evaluation in terms of content, geographical area and intended use?
  • The following questions & remarks might help to define the objective & scope:
  • Why would the programme like to start an evaluation? When would an evaluation be useful for the programme, the policy-makers and the stakeholders?
  • Are there specific aspects of the programme that need attention or that the programme has doubts about/problems with? Are the objectives of the programme being achieved? A certain topic, priority, geographical area, time period, projects, link to other programmes?
  • Is the evaluation supposed to focus on its relevance, effectiveness, efficiency or only one of these? Should it focus on an operational aspect (how to improve the programme now, modify implementation procedures, reallocate funds) or a strategic one (what needs to be dealt with in future, what change of policy direction is needed)?
  • What will be done with the conclusions? Will they be used? By whom? For what purpose (deciding, debating, informing)? When?
  • Is it politically appropriate to perform such an evaluation at this particular time or in this particular context? Is there a conflict situation that could compromise the success of the exercise? Has a recent study already answered most of the questions? Are evaluation priorities sufficiently stable?

For an evaluation to be useful, it should be formulated as precisely as possible. Often contractors, not wanting to influence the evaluation team too much, are reluctant to express in advance the changes they think should be made or their doubts concerning the effectiveness of a particular action. It is better to avoid having to ask the evaluator for extra work which would raise the costs of the evaluation. Experience shows that it is better to be as precise as possible.

Section 2. Main users and stakeholders of the evaluation

Who are the individuals, groups or organisations that have an interest in the intervention to be evaluated or who might be interested in the process or in the results of the evaluation itself?
How will the evaluation be monitored and guided? Is there an Evaluation Steering Group, who are the members of this Group, what will be their role and how often will they meet the evaluators?

Section 3. Evaluation questions

  • The importance of evaluation questions in an evaluation design cannot be overstated. They not only provide guidance for the work of the evaluator but also to the programme on the issues they really want to deal with. The temptation otherwise is to gather large quantities of data and produce sometimes technically sophisticated indicators which make little contribution to practice or policy.
  • Formulate questions that the stakeholders will find useful, for which there is a real need for answers. If a question is only of interest in terms of new knowledge, without any immediate input into decision-making in the programme or into the public debate, it is more a matter of scientific research and should not be included in an evaluation.
  • When the evaluation is more horizontal (overall programme), less questions can be formulated than when it examines a particular problem/part of the programme.
  • Questions should be directly linked to the main objective of the evaluation.

Check if the formulated questions can actually be answered in the evaluation. 1) Is the data required to answer the question available? 2) Are there already any results that can be evaluated? In 2000-2006, for example, many mid-term evaluations could hardly evaluate the effectiveness of the operations as most of them had only just started. 3) Is the question clear? If the question can be interpreted in different ways the programme might get a proposal that does not match its expectations. 4) Is it necessary to have an evaluation to answer this question, or could the monitoring system or a discussion be sufficient? This check may lead to a decision not to undertake the evaluation, to postpone it, or to revise the questions.

What types of evaluation questions are possible?

Evaluation questions can pertain to different levels. They can be:

  • Descriptive questions intended to observe, describe and measure changes (what happened?)
  • Causal questions which strive to understand and assess relations of cause and effect (how and to what extent is that which occurred attributable to the programme?)
  • Normative questions which apply evaluation criteria (are the results and impacts satisfactory in relation to targets, goals, etc.?)
  • Predictive questions which attempt to anticipate what will happen as a result of planned interventions (will the measures to counter unemployment in this territory create negative effects for the environment or existing employers?)
  • Critical questions which are intended to support change, often from a value-committed stance (how can equal opportunity policies be better accepted by small and medium-sized enterprises? or what are effective strategies for reducing social exclusion?)

Experience of Kai Böhme, evaluation expert:
The topics should be focussed on the areas where there is the highest need for learning and honest reflection at programme level. As far as possible the evaluations should not only focus on the number of projects in a specific field but also on their characteristics and (expected) achievements.



Section 4. Available information

The Terms of Reference should contain a review of the current status of information on the programme and its effects. This will include which documents are available, what analysis has already taken place and some brief extracts when needed to prepare a good proposal. INTERACT suggests indicating who is in charge of providing information from, for example, the monitoring system. Furthermore, it is important to mention if certain information/data are not available and will have to be collected by the evaluator to ensure a realistic calculation of the offer.

Section 5. Main methods or techniques to be used

If the Evaluation Steering Group has clear ideas on the methods or techniques to be used, these can be included in the Terms of Reference. It is a good idea to indicate whether the programme would prefer the evaluation team to use this approach or if they are open to other approaches.

In general the approach will be as follows: 1) identify what needs to be done, 2) data collection, 3) analysis of data, 4) provide judgement.

The choice is generally made to maintain sufficient flexibility to allow those responding to the Terms of Reference to differentiate in terms of the relevance and clarity of their methodological proposals. This is especially important in the selection phase because assessing the methodological qualities of the proposals is a crucial step in selecting the right evaluator. When possible from an administrative point of view, the best way is to determine a budget (see below) and describe only the main outlines of the method in the Terms of Reference; then select the team that proposes the most promising method. Those selecting the team will then need to be capable of judging the methodological quality of a tender.

Evalsed provides an overview of possible methods and techniques of evaluation. Here are some suggestions:

  • Planning and structuring of evaluation: concept/issue mapping, stakeholder consultation, evaluability assessment, logic models, formative/developmental evaluation;
  • Obtaining data: social survey, beneficiary survey, interviews, priority evaluation, focus groups, case studies, local evaluation, participatory approaches & methods, use of secondary source and administrative data, observational techniques;
  • Analysing information: input/output analysis, econometric models, regression analysis, experimental approaches, Delphi survey, SWOT;
  • Tools to inform evaluative judgements: cost-benefit analysis, benchmarking, cost effectiveness analysis, economic impact assessment, gender impact assessment, environmental impact assessment, strategic environmental assessment, multi-criteria analysis, expert panels.


Section 6. Time schedule & reporting

6A. Time schedule

Example timetable

Activity Date M1 M2 M3 M4 M5 M6 M7 M8 M9
Public procurement started*                    
Deadline for proposals                    
Inform evaluators of result*                    
Start evaluation with kick-off meeting                    
Info gathering and analysis                    
Meetings between programme and evaluator(s)                    
Draft final report                    
Feedback & meeting with Evaluation Steering Group                    
Final report                    
Presentation and discussion of final report                    
                     

M= month
Source: INTERACT

INTERACT recommends indicating the time period of the evaluation and important dates – including procurement deadlines - in the Terms of Reference. If an evaluation is linked to certain events or decision-making moments, it is a good idea to mention this and to take some extra time to ensure that the evaluation will be ready in time.

If the meetings of the Evaluation Steering Group have to take place on certain dates and/or in specific locations, this should also be indicated in the Terms of Reference.
6B. Reporting

INTERACT recommends indicating in the Terms of Reference what type of report(s) the programme expects. What should be included in the draft final report? Will there be several interim or specific reports? What will the evaluation report be used for? Who are the target group? Should the evaluation contain a summary for a broader audience? In what way does the programme want to make use of the evaluation and what type of report(s) is therefore requested? How will this report link to the annual report? Would the programme like to have recommendations in the field of 1) finance, 2) content and/or 3) implementation?

Section 7. Indicative budget

It is good practice to suggest an indicative budget and then leave those competing for an evaluation by open tender to suggest what they would be able to provide for the budget available. This allows value-for-money assessments to be made. It also provides the contractor of the evaluation with greater control over expenditure. Some Member States require the maximum budget to be published.

Section 8. Required qualifications of the team

It is a good idea to reflect in advance what criteria are important for the programme. Here are some possible choices which could help to develop a list of required qualifications:

  • Experience with evaluation or rather with the programme or rather with a specific topic?
  • Experience with the different cultures or merely an understanding of Structural Funds?
  • Academic research or practical solutions?
  • Experience with certain methods?
  • Experience with certain countries/regions? Knowledge of certain organisations?
  • Knowledge of certain languages?
  • Qualifications of the company or qualifications of the individual experts?
  • Experience with European studies or local studies?

Be realistic in the demands. The higher the qualifications the programme asks for, the more limited the number of potential proposals will be. How important are the qualifications of the evaluators? In any case the quality of the proposal should be more important than experience as such.


Experience of Marie-Jose Zondag, evaluation expert:
How to deal with a potential conflict of interest for the evaluator? Suggestion: 1) Ask for a declaration from the evaluator that he/she will not deal with the parts that could involve a conflict of interest for him/her.


Experience of Marie-Jose Zondag, evaluation expert:
What to do if the evaluator does not have a lot of experience or is not very familiar with Territorial Cooperation programmes and Structural Funds? Suggestions: 1) Include a comparison with other programmes; 2) Include a more experienced evaluator to act as advisor; 3) Closely monitor the evaluator.



Section 9. Submission rules and assessment criteria

The Terms of Reference could specify the deadline, the modes of transmission (post, fax, e-mail), the language in which the tender must be drawn up, how long the offer should remain valid, etc. It should also indicate the criteria according to which the proposals will be judged. Especially for larger evaluations it can be useful to indicate the required structure of the proposal. The Terms of Reference should state, for example, the relative importance in percentage points that will be given to:

  • The quality of the methodological approach;
  • The qualifications and previous experience of the team;
  • The price.

6.4 Tool: Example format for assessment of proposals

It is recommended to make a clear list of assessment criteria for the proposals and to include these (in a summarised form) in the Terms of Reference. Many proposals use the assessment model shown below. In some cases:

  • The content and budget are assessed separately and are therefore sent in separate envelopes;
  • The assessment is done anonymously: the names of the evaluators will be made invisible in the proposal.

Some suggestions regarding the assessment team have been provided in previous sections.

Example format

Assessment grid for '… [title of evaluation]' (… [reference number])

Name (or number) of proposal/evaluator: …

Topic Maximum points Initial assessment Revised assessment
I Organisation and methodology E.g. 60 [Total score for I] [Total score for II]
Rationale (understanding of and reflection on the Terms of Reference, risks & assumptions) e.g. 20 [score] [score]
Strategy (approach, activities, timetable, milestones, logical framework) e.g. 40 [score] [score]
II Evaluation team e.g. 40 [Total score for II] [Total score for II]
Experience (description of company / consortium, division of tasks). E.g.
(a) experience with INTERREG & ERDF evaluations, (b) experience with the programme area, (c) experience with Territorial Cooperation.
e.g. 20 [score] [score]
Experts (CVs, division of tasks) e.g. 20 [score] [score]
III Qualitative assessment      
Strengths   [text] [text]
Weaknesses   [text] [text]
Total 100 [score] [score]

Source: INTERACT


Score on content:
Score on budget:
Total score:
Assessment performed by: …
Date: …
Signature: …


Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.