Bangladesh

Avoiding soundbites on project success or failure…

By Dr Nick Nisbett, IDS research fellow and theme 3 research leader for Transform Nutrition and Ferdous Jahan, Professor of Public Administration, Dhaka University

Last week in Dhaka we were hosted generously by BRAC and ICDDR,B at a conference on the boosters and constraints to delivery of large health and nutrition programmes, where we presented the results of the project Impact Evaluation of DFID Programme to Accelerate Improved Nutrition for the Extreme Poor in Bangladesh. This was a three year, multi-methods mammoth evaluation of the nutritional impact of two of DFID’s Bangladesh programmes focusing on the extreme poor and another benefitting the urban poor. These were Economic Empowerment of the Poorest (EEP) (AKA Shiree), the Chars Livelihoods Programme (CLP) and Urban Partnerships for Poverty Reduction (UPPR).  We got a lively reaction from the audience and some tips on how we might better present our results in future, where all too often failure to find the expected, ambitious impact (in this case on child stunting) is translated into the soundbite that programme x failed, or approach y is not worth repeating.

Ferdous and Nick presenting the evaluation in Dhaka, February 2017

This would be a sad outcome of the evaluation – ignoring the valuable lessons of these programmes and DFID’s innovative and much needed approach to try to link up the livelihoods needs of the extreme poor more holistically with specific actions designed to improve child nutrition.There is a lot riding on such models working in future, particularly in Bangladesh, where as one of us argued at the conference, most nutritional gains have not come from such holistic nutrition programming but wider improvements in aspects of people’s lives which may now start to plateau.

Supporting child outcomes in poor families more holistically…

The evaluation began in 2013 when DFID and partners recognised that the livelihoods elements of the programmes alone were not leading to nutritional improvements in the children of beneficiary households.  The food security and productive asset transfer / income creation/support elements of these programmes were likely to benefit broader household food security. But ensuring household food security and boosting income is a necessary, but not sufficient, measure to ensure improvements in the nutritional status of children.  Among other things (including sanitary, healthy environments) mothers and carers need the right support for breastfeeding and feeding in the early years so that children receive the immunity, nutrient boosting and pathogen-free benefits of breastfeeding, and are then fed sufficient, diverse diets alongside breastmilk in the critical developmental years until two.

Three to four years into the operation of these programmes, therefore, DFID added an additional component of nutritional support for mothers, children and adolescent girls.  This was to be delivered by women recruited within the community (community nutrition volunteers – or CNVs), trained to deliver a package of iron and folic acid tablets to benefit mothers and adolescent girls along with collective (e.g. courtyard meetings) and individual level (home visits) awareness raising activities. Children below two were to benefit directly from a combination of breastfeeding/feeding support to their mothers, with provision of deworming tablets and five nutrient micronutrient powders to be sprinkled on their food.

Getting ahead of our ambitions…

Ultimately this was a big ask of the three programmes, already tasked with running complex livelihoods delivery.  Ideally it would have been designed in at the beginning, rather than as a separate component, but given that this combination of support (livelihoods + nutrition) was untested, it seemed like a reasonable plan. Perhaps what was difficult though in this case was the usual donor pressures to show results within the space of two years – a theoretically possible but practically ambitious target of reducing child stunting.

We are all guilty of pushing ambitious targets on stunting, because it is a marker of chronic undernutrition – a risk factor in 45% of all child deaths and a sign of other profound deficiencies which can lead to poorer, less healthy lives. However, those who have been in the evaluation and nutrition game for some time now, including Purnima Menon, who was at the conference, have been arguing that this doesn’t necessarily mean we should be applying ambitious stunting targets to all nutrition programmes – because so many things need to happen in concert to produce both well nurtured but also all round healthy children. As Purnima’s presentation showed, even the best run programmes which focus only on child feeding practices  will not show a result on stunting and nor should they be expected to, particularly within a short evaluation time span. Asking mothers to diversify children’s diets also requires time and money to do so. Perhaps the programmes we evaluated, because of their wider and more holistic approach to livelihoods, had this opportunity, but it was untested water in this combination and at this scale.

Learning what worked…

The evaluation (randomised between ‘livelihoods only’ and ‘livelihoods plus nutrition’ recipients) was therefore set up to look at impacts earlier on in the ‘impact pathway’ – e.g. in changes in the knowledge and behaviours around particularly optimal child feeding practices (e.g breastfeeding within three hours of birth, which hugely boosts immunity, avoiding feeding water before the child is six months, having a sufficient, diverse diet whilst still including breastmilk thereafter). And we wanted to explain why and how all this happened or didn’t, so a process evaluation element monitored how the programmes were implemented, whilst a qualitative community based element considered barriers, constraints and wider contexts within the communities themselves.

To cut to the chase, what we learnt at the conference was to be even more careful not to say, in future, that there were no impacts on child nutrition resulting from this type of model (to be fair that’s not what we said or exactly how the audience responded – we exaggerate!).  And ultimately, we saw that the community nutrition volunteer based model did not fail.  The programmes delivered some of the key outputs in terms of getting nutrition workers in place, and they visited most families, delivering the IFA tablets, mircronutrients and deworming tablets and some counselling.  Mothers generally had positive things to say about the workers – they trusted them and when we tested the workers’ knowledge at endline they scored reasonably highly, so they were well trained. Mothers’ knowledge about the benefit of iron in diets increased; children consumed the supplements or other iron-rich food as a result and we also saw improvements in some cases in terms of non-introduction of water or other substances before six months.

…and what didn’t…

But where things fell down were not on general programme delivery, but on the long neglected specifics of implementation– not on whether the nutrition volunteers went to households, but how often they went there, how long they spent there, how many topics they discussed, whether they were age appropriate for the situation of the mother and child (for example, too much time was spent on explaining the benefits of exclusive breastfeeding when mothers already had children too old for this).  The worker’s caseloads were too high, they were travelling too far in difficult to reach areas, spending less time with mothers than ideal and were not receiving the type of supportive supervision and monitoring which would have detected and improved these critical aspects of delivery earlier. So the types of indicators most likely to be of long term benefit to immunity or child development (ie consuming diverse and sufficient food high in micronutrients) and/or likely to actually lead to improved growth (e.g. diets including animal source-food) did not shift.  No surprise, then, that neither, ultimately, did stunting.

In the qualitative study communities mothers reported the difficulties of following prescribed practices when they were time poor, or couldn’t afford diverse diets, or time to cook separate meals for fussy children, or didn’t have the freedom to decide on child feeding practices themselves, or were equally likely to value opposing advice from e.g. mothers-in-law.

…..Nonetheless….

Despite the enormous pressure DFID are under to produce results, they have not shied from commissioning these types of evaluations which get to the heart of implementation complexity, particularly for these new types of much needed models. We’re slowly getting the evidence through that such mixed models (e.g. in a much smaller trial in Bangladesh, combining cash transfers and child feeding support) can have significant results (yes, even on stunting).  We’re still not convinced that similarly ambitious goals for stunting should be set for two year programmes experimenting at scale, even while it is still instructive to try to measure any change.  But what we have learnt is immensely valuable for future programming (and again, to their credit, DFID have taken on board all the recommendations so far). Ultimately, therefore, such knowledge leaves us better prepared for “delivering success at scale”.

The evaluation programme was led by IDS in partnership with IFPRI, BRAC Institute of Governance and Development (BIGD) and Prof. Ferdous Jahan of Dhaka University; ITAD, CNRS and was part of the MQSUN programme led by PATH. See here for the final report and here for a shorter report brief. The Delivering for Success Conference was hosted by BRAC and ICDDR,B in collaboration with the Transform Nutrition and Future Health Systems Consortia.