On Friday, November 27th 2015, an expert group of panelists convened in Kathmandu, Nepal at the 2015 Evaluation Conclave to share their expertise on “evaluating programmes in the water, sanitation and hygiene sector”. The following article features comments from Dr Robert Chambers, a world renowned expert on evaluation and community led total sanitation, on how to evaluate the impact of Wash programming.

Q: Can a community ever be truly open defecation free? Is it actually a meaningful indicator?

When Kamal Kar first developed community led total sanitation (CLTS), the focus was on the whole community working together to become completely open defecation free (ODF). It was new to talk about open defecation, but the incentive was to become ODF, so that people are no longer eating one another’s shit. Now, there are at least 20 countries where there is a national policy not to have a hardware subsidy and to go for complete ODF status. The achievements have been remarkable.

A community becoming ODF is a good indicator. It gives a clear target for total collective behaviour change. At the same time It can be difficult for a community to be absolutely and totally ODF. What matters is a social commitment to becoming totally ODF, because then people will apply pressure on any who want to continue open defecation to stop, enhancing sustainability. The challenge is to optimise strictness in verification.

A danger with counting ODF communities is generating misleading data. Whenever there are awards for ODF, there are dangers of distortion. In India, there was extreme exaggeration in the Nirmal Gram Puraskar. Many communities were declared ODF and it wasn’t true. A study showed that hardly any of the communities were actually ODF

There were rewards, huge competition, and lax verification. The numbers generated were nonsense. This led Indian authorities to crack down. One of the problems is the inflation of figures that later have to be scaled down, which can be embarrassing.
Advertisement

You need to become cautious about the declarations that are made. The solution is to optimise – being not too strict and not too lax in verification and making it a learning process for communities which are not successful first time.

Q: How can programmes be better designed to ensure accurate ODF verification?

A good way is for communities to be encouraged to achieve ODF status to their own satisfaction and then request verification. This can take many different forms. Sometimes it can be done locally by mixed teams. Sometimes by teams from neighbouring communities. Sometimes by a third party like an independent NGO or trained consultants.

There is no one-size-fits-all. In Kenya and Zambia, verification was contracted to a third party but this proved slow and expensive and other more decentralised methods had to be adopted. The conventional wisdom is that it is good to have verification teams that know local conditions.

Many countries now have their own protocols for verification and certification, India has two phases for becoming ODF, which is very sensible. Everywhere there are issues about how strict to be or not to be. India is in a difficult position. In line with the CLTS approach, India is going for ODF communities.

They also want decent verification. But this may mean that progress, though good, will appear slow, especially when the chances of having ODF communities are limited by the inherent difficulties of the subsidy system. This makes it very tough for those running the programme. Still, there are promising signs in some States.

Q: What are the factors that lead to the communities starting to openly defecate again?

These are many and can be foreseen. A Plan study of sustainability in four countries in Africa found hardly any improvements in toilets in two years or more between the ODF declaration and the time the study took place. People had constructed simple pit latrines, and these often began to deteriorate. Some toilets were getting dirty, and were then disused because they were dirty. Others collapsed.A big issue in India is that people don’t want to face having to empty a full pit. So they want large septic tanks which will take a long time to fill and they often use it sparingly and selectively. So there is partial usage with some household members continuing to defecate in the open some or all of the time, reserving the toilet for women or visitors, or old or sick people, or bad weather or the middle of the night.

At other times, people will think they are being socially responsibly by going outside and not filling the pit. Also, when there is pressure and a queue, for instance when children need to go before school, the men in the family may feel it is considerate and responsible to go outside. There is also widespread belief that it is healthier to go in the open air.

This whole area is really difficult and contested. It is easy to pass judgement. I don’t want to pass judgment on anybody. In India, it is a very difficult situation.

Q: What cautionary note would you like to sound for those that are working on ODF verification campaigns?

Campaigns and the drive towards achieving ODF communities are vital. Nothing should undermine these. But chasing targets and giving rewards for their achievement are full of dangers of false claims.

We know that in a successful campaign, there can and should be pressure to get on with it and become ODF, and there can be competition between Districts. The competition should be for honesty and learning. It is better by far to say that there are delays than to make misleading claims. Optimising accuracy and learning how to do better need to be recognised and rewarded.

This blog first appeared in the WSSCC Partner Zone in The Guardian on 20th January 2016.