Why do it

“Have we made a positive difference?” is the question we ask ourselves at the end of a training as we pass out the evaluation forms. Aren’t we ever hopeful that the training experience has “stuck” and left some lasting impression with participants? Impact is an area that funders will generally require us to address as our programmes conclude.

The systematic collection of information so that we understand how we carried out our activities, what learning took place alongside these, and how well we have moved ourselves and others towards our goals is what evaluation is all about. It is about learning from what we do. That learning means having documentation and reflecting on what was successful and why, and what tasks might be replicated in the future.

Project evaluation may feel like a burden, as if it is something that we do primarily for the benefit of ourselves. However, evaluation is a feature of our commitment to an internal process of continuous learning. So as we discuss evaluations in our organisations, it might be more helpful to think about what we might learn through our evaluations and not just how we will try to show impact.

When we carry out trainings, we are already doing some kind of non-formal assessment by paying attention to things like if participants are engaged, or if they are able to carry out activities that we ask them to. Ultimately, however, we want to know about what learners are thinking, how they are interpreting their experiences and, ultimately, what difference their engagement in our projects may make at a later point when they are no longer in our sights.

On occasion I have the good fortune to run into a learner I worked with years before. Sometimes they mention how valuable the experience was or how much they have been able to apply it. Of course, this is gratifying and would be captured in the “results” basket of an evaluation if I had been able to administer a follow-up survey to the training. This is one of the reasons to do long-term evaluation studies: to see “what sticks”.

But evaluations will not necessarily confirm that we have reached our results. This is particularly the case when we are working in highly complex environments, where the medium- and long-term results of our training efforts will be influenced by a range of environmental conditions and actors.

Here are some of the things that I have learned from short-term studies carried out on my programmes. (I won’t say which ones!)

• Participants liked the activities in the training but they didn’t have the opportunity to apply these in practice.
• Even though concepts may have been presented in the training and included in materials, they were not really clear, especially their application to “real life”.
• Sometimes materials are not written in a way that is friendly or appealing, especially for youth.
• Learners simply don’t have the time to complete a multi-week course.

These examples illustrate the ways in which evaluations can help us in understanding the processes and results of our programming. It’s about getting a continuous “reality check” on our efforts and improving as we plan ahead.

 How to do it

There are many guides and toolkits available to practitioners to consult in designing educational evaluations, some of which are listed at the end of this article. The resources I’ve included here have been developed for the field of EDC, HRE and leadership development. If these don’t meet your needs, you might want to look for other evaluation resources or supports, for example, around youth leadership or social networking.

Most evaluation studies involve mixed methods, that is, a combination of qualitative data (for example, through interviews and observations) as well as quantitative data (from surveys and regular project data). Other techniques include observations and content analysis of key documents. The methods we decide to use flow from what we really need to document. This brings us back to what we want to learn.

We can use backward mapping when we develop our evaluation designs, beginning with the stated goals and objectives of our projects. In our evaluation, we will want to document if we have been able to reach our anticipated goals. This is a “results evaluation” or “outcomes evaluation”.

We may also want to pay attention to whether or not our project processes transpired the way we expected in terms of reaching our goals. Each project has some version of an underlying “logic chain” regarding the relationship between our project processes (our “interventions” such as trainings) and the results we are looking for. We probably also have some ideas about the change processes we are fostering, including those related to learning. Were our assumptions about these processes correct? This is known as a “process evaluation”.

As part of project management we’ll also be documenting our activities. Common activity outputs for EDC/HRE training programmes are the number of trainings, participants (disaggregated by background), publications, partnerships, etc.

After we have identified what we would like to document, we then need to figure out how best to capture this information. This is called data collection methodology, and it brings us back to data collection methods.

Allow me to share an example based on the online monitoring and evaluation course I teach for human rights education. If I were to develop a monitoring and evaluation plan for this online course, I would first want to monitor all of the “inputs” from the HREA side. These would include the syllabi, the course website, the readings, weekly instructor messages and feedback. Measuring whether or not these were given to learning is very straightforward. If we wanted to assess the quality of these inputs, we might ask for learner feedback. We might also devise other means of assessing quality, for example, how current the readings are.

An evaluation of the course could have two general dimensions. One might be process oriented. Did learners engage with the course? Did they participate in the weekly online discussions and submit written work? We can document this quite readily through the work done on the course website.

The other dimension has to do with learner results. In the short-term, HREA would like the learners to acquire new knowledge such as key evaluation terms and skills related to developing a logical framework. These results can be documented through pre- and post-survey evaluations of learners. We might also assess before-after logical frameworks. In the short- and medium-term we might look for course alumni to apply their learnings. We could explore this through surveys as well as interviews, and try to understand not only the “good examples” but instances where alumni did not apply skills they learned.

A final point I’d like to make in this “how to do it” section has to do with our ways of working with people. In the field of EDC/HRE, evaluation processes are now being held to the light in the same way that project development is. Are our evaluation processes participatory and transparent? Do beneficiaries have the opportunity to weigh in on evaluation design and potentially help with data collection and analysis (not be involved only as a data source)? These ethical considerations have to do with an increasing recognition among evaluators to be not only culturally sensitive but to genuinely collaborate with participants in evaluation processes. This is in the true spirit of EDC/HRE!