At the end of 2008 I finished an evaluation of organisational learning at SNV – The Netherlands Development Organisation. I worked with another consultant, Sara Methven, and a small team of SNV head office staff. The evaluation involved field visits to SNV’s work programmes in Vietnam, Lao PDR, Kenya and Ethiopia and extensive interviews with SNV staff in The Hague and a number of other countries where SNV works.
During the evaluation we were keen to involve as many people as possible so we gave the work an identity: ‘Improving Through Learning’; launched it through the SNV in-house newsletter and tried to create a blog to keep any interested SNV staff members informed of progress and actively involved in the evaluation process. I say “tried to” because this proved to be more difficult than we anticipated.
Nevertheless, as a result of our limited experience I am convinced of the potential of blogging to increase levels of awareness, participation and ownership of evaluations of large-scale and geographically dispersed organisations.
Since SNV did not at the time have an intranet capable of supporting blogging, my first thought was to set up a WordPress blog. I spent time writing posts chronicling our thinking about the evaluation in order to to keep potential subscribers informed of progress. However, making the blog available to staff only (not publicly available) when the number of staff was potentially in the hundreds was beyond my capabilities so the posts were inaccessible to others and hence unread. Time was passing fast so we looked for an alternative tool to share our thinking. We decided to use D-Groups as this was a technology that all SNV staff are familiar with. D-groups are targeted at people involved in international development and SNV was an early adopter of the technology so we started to set up an ‘Improving Through Learning’ D-group for the evaluation. The protocols involved in this were straightforward but time-consuming so by the time we got to the point where staff were signing up to read the posts (transferred from the earlier WordPress blog) Sara and I were reaching the end of the evaluation. Despite the time constraints, we received useful feedback on a couple of sections of the draft report that we posted on the D-Group so we got a tantalising feel for the potential of working in this way.
The final draft of the evaluation report was completed early in January of this year. I was disappointed that I had been unable to make better use of the blogging idea due to my own limitations in understanding the technical side but I have learned some important lessons in the process.
The first lesson is to get agreement from the client right from the outset about setting up a blog. You also need to agree what it will be used for – informing people, engaging them in the process, gathering data, getting feedback on initial analysis and sections of the report? It could be any or all of these. We raised the idea in our proposal to SNV. They liked the idea but looking back I didn’t move fast enough to get things up and running.
Second lesson – it is important to start early with setting up the blog. That means finding out very early in the evaluation preparations what technology exists in the organisation and what people are used to using. We should have thought about a D-Group at the outset as it was a technology familiar to SNV staff. OK, it wasn’t a blog exactly but in the absence of in-house blogging facilities it could deliver what we needed. SNV is in the process of introducing an intranet system that supports blogging so it should be much easier in the future to achieve what we tried – and failed – to do.
Third lesson – by setting up a way of keeping people informed that they themselves sign up to you can create a community that can contribute to and influence the evaluation process. We posted not just ‘news’ of how the evaluation was progressing but also reflections on what we were learning and draft sections of report-writing. For example, we created a timeline of important events related to the evolution of organisational learning at SNV. This was very useful for us as evaluators and proved to be interesting for SNV staff – many of whom had never considered the ‘big picture’ of organisational learning in this way before. We posted our timeline for comment and got some very helpful feedback. I’m convinced that this could be very valuable for any evaluation but particularly in large organisations where it is unlikely that the evaluators will have the chance to gather the views of everyone. Of course, evaluators need to be aware that those signing up are a self-selected group who, by definition, have an interest in the subject of the evaluation so their views may not be representative of the staff group as a whole. Nevertheless, blog postings and comments are likely to provide a rich source of ideas and comments that may otherwise be denied to evaluators through lack of time or access.
One further benefit of a blog in evaluation is the creation of a community that can be asked to contribute ideas for following-up the evaluation findings and recommendations. The ideas that come from a group of interested, self-selected individuals may well be different from those that come from people who have more formal responsibilities for follow-up. Stimulating the expression of ideas in this way can only enrich the discussion about utilisation of evaluation.
An unintended benefit is the ‘legacy effect’ of bringing together people within the organisation who share a common interest and who may choose to keep in touch.
Blogging in evaluation is not ‘the answer’ to participatory evaluation – it can only potentially involve those that have access to the technology and an awareness of how to use it. Nevertheless, I am convinced that using blogging and on-line groups in evaluations and other consultancy work is worth further exploration as a way of helping to strengthen participation in evaluations and improve the utilisation of evaluation results.