Evaluation

EvaluationEvaluation isn’t something to consider only at the end of your project. It's common practice to use monitoring and evaluation methodologies to track progress as you go, take in feedback, and make necessary adjustments. 

With this in mind there are both ‘progress evaluation’ and ‘final evaluation’ approaches to consider. The Impact Pathways model focuses on both, in the same way we emphasise equally the importance of process and product in creating impact – what you come out with at the end (product), and how you did it (process). Using both progress and final evaluations are ways of understanding the impact of both approaches.

The amount of evaluation you can do will depend on your resources. A full evaluation can be huge, and some large non-profits or universities can spend years evaluating an initiative. Unfortunately funders often don't provide a lot of resources for evaluation, but if you have the chance we recommend putting five percent of your budget into this work -  it will help you know what worked and didn't work so you can do a better job next time. The documentation can also be a powerful tool in seeking support from funders, partners and communities.

Involving your team, partners, stakeholders and the community is important in evaluation – you’re looking for their thoughts more than your own. Their experience will reveal the insights and impact your Video for Change initiative has made.

A final evaluation allows you to go over your steps and reflect on successes and failures, key learnings and experiences. You can also collect statistics, surveys and more, and use these to assess how you did. The final evaluation doesn’t mean all activities end after this. Often relationships built throughout an initiative lead to the continuation of working together, through new projects or other forms of collaboration. An evaluation process can set the stage for the next collaboration.

A progress evaluation can happen at any point of your project where you need to check in. These can be as short as 15 minutes in the midst of a stage, or could be a milestone evaluation once you’ve completed a stage – e.g. at the end of your filming. All data and information gets rolled into your final evaluation so it's very important to collect as you go – trying to find everything at the end will be difficult and will take longer than if you do it as you go. The more information you have, the more you'll be able to get a picture of what has occurred.

This is a worksheet for doing feedback loops, or doing short evaluations as you progress with the initiative.

 

Exploring Methods – Qualitative and Quantitative

There’s been much debate in the documentary and impact field regarding the right balance of quantitative versus qualitative measurement.

Quantitative methods include things such as conducting surveys, and gathering statistics and data. It's about collecting and analysing numbers to present an idea about what’s happening. Qualitative methods include things such as interviews, observations, focus groups as well as using media to record stories.

With more and more people consuming and sharing video online, gathering data about your film has become increasingly easy. Tracking likes, hits, clicks and more can be provided by social media platforms or systems such as Google Analytics or the open source alternative Piwik. Be careful in being seduced by data – as Economist Ronald Coase said, "If you torture the data long enough, it will confess” – meaning data can be manipulated to say whatever you want it to say. 

Both qualitative and quantitative methods are useful depending on what you need to know, though in Video for Change contexts qualitative tends to be preferred. Below we have provided some examples of each. We suggest you read up and then mix and match as needed to create 'impact stories’.

We surveyed Video for Change practitioners to find what their mix and match looks like. The diagram below shows the most frequent methods.

Evaluation Survey

There are quite a few different methods used and most practitioners use at least two or three. You can read the analysis of the survey results here.

 

Impact Stories

An impact story is a way of combining the result of your evaluation methods together to create a compelling narrative about the impact of your initiative. Your impact story will draw on the documentation you’ve collected during each stage of your initiative, as well as at the end. This narrative aims to communicate a deep understanding about what has happened and why.

An impact story can make use of a wide variety of media – infographics, survey results, photos, writing, audio and of course video, you’re a film-maker after all! Your narrative should include elements of transformation: moments where real change took place, or individuals whose behaviour or mindsets have changed. It should also take into consideration your original objectives – did you meet them? What fell short? What were the unexpected outcomes?

Constructive has a helpful set of tips for how non-profits can create compelling impact stories.

InsightShare also has some great tips about how you can be creative in measuring impact, while also showing that a lot of the impact only shows after time. In social change ‘instant’ success is rare.

Below a good example of an impact story by WITNESS.

 

WITNESS Impact Story

Cambodia: Supporting Grassroots People’s Movements through Video

WITNESS first encountered our Cambodian partners at LICADHO over a decade ago while working together on a campaign against forced evictions. LICADHO is a grassroots organisation working to expose human rights abuses and support grassroots people’s movements throughout Cambodia. Since then, thanks to multiple WITNESS trainings and collaborations, the organisation has gone on to train many more human rights defenders across Cambodia and share our resources on how to use video for justice and accountability.

LICADHO has translated several of WITNESS’ training videos into Khmer and begun to use them in their own trainings. Inspired by WITNESS – and equipped with high-level production and editing skills learned from WITNESS – a trainee, Chaly Johnson, has started to produce his own video series on filming with mobile phones. This is the ultimate testament to the strength of our model, proving not only that our individual resources are effective, but also that we can empower activists to carry this work forward independently. LICADHO now regularly develops its own materials, some adapted from WITNESS and others not, with far less support.

Thanks to the credibility the organisation has built with its local partners, LICADHO is able to organically grow its own capabilities and share the WITNESS methodology broadly within Cambodia. The Venerable Loun Sovath (otherwise known as the ‘multimedia monk’), the leader of LICADHO and recipient of the Martin Ennals Award for Human Rights Defenders, states: “LICADHO is so grateful for all the support and guidance we have received from WITNESS, [which] has played a crucial role forming LICADHO into video activists.”

 

English