Evidencing Impact Across a Diverse Portfolio of Research

This section sets out the approach undertaken by the Impact Initiative to capture and communicate the impact of research projects as part of its knowledge-brokering role for the Economic and Social Research Council-Foreign, Commonwealth & Development Office (ESRC-FCDO) Strategic Partnership. It describes how diverse dimensions of impact were recognised and articulated and the value of micro-impacts. In contrast to other approaches, including the UK’s Research Excellence Framework and its Impact Case Studies, the Impact Initiative worked collaboratively with researchers at all stages of their projects’ life-cycles, identifying and evidencing changes that research processes have contributed to. The benefit of a narrative-based approach that structured impact stories around a challenge or problem, a research process, and subsequent changes was further enhanced by a very concise, accessible format. For knowledge brokers and donors the key lesson is around the value of a collective, programme-level approach to developing impact stories. The Impact Initiative’s collection of these is impressively broad and combines rigour with accessibility. As a body of work it provides a unique insight into how impact is understood and achieved in development studies, and the process of building this collection has created a rich learning environment.

Introduction

This section documents the Impact Initiative’s approach to identifying stories of impact across a broad portfolio of development research and packaging them in accessible ways. This is important learning at a time when UK Overseas Development Aid is coming under increasing scrutiny and globally the expectations of research donors and the wider public around the measurable, visible impact of evidence continues to increase (Parsons et al. 2020). In the absence of large programme evaluation budgets, it is becoming increasingly common for research organisations and donors to produce narratives that seek to identify, with some supporting evidence, the difference they are making. As an often-contested concept, evidencing research impact can be a challenge. For the ESRC-FCDO Strategic Partnership funded portfolio of research projects the publication of a series of impact stories has demonstrated that impact happens in many ways and sometimes in small steps. The 24 stories span education, conflict, health, urbanisation, poverty and inequality, and many other topics across South Asia, Africa, and Latin America. They illustrate that, despite the complexity of evidence impact processes, programme-level knowledge brokers are in a strong position to work collaboratively with researchers to showcase rich bodies of impact. We also explore how a process of identifying impact stories that is not constrained by project life-cycles can build researchers’ capacity and confidence to engage with the impact agenda and provide valuable learning for those designing research projects and seeking to maximise their impact.

Why telling stories of impact is important

The language and art of impact is highly complex and shaped by widely differing understandings of how change happens. Subsequently, capturing and communicating impact can be elusive and often hard to visualise or articulate. The Impact Initiative communications team were tasked with developing practical tools and processes to tell the stories of research impact in clear and compelling ways, without diminishing the content.

‘One analogy might be to think of ‘impact’ similar to a pot of water boiling over:  most of what happens is gently heating, bubbling and gaining energy over time - we can think of these as incremental outcomes. We see the impact when the water finally boils over’

Nasreen Jessani, Senior Researcher (Centre for Evidence Based Health Care, Stellenbosch University) Faculty (Johns Hopkins University Bloomberg School of Public Health) (pers. comm. 2020)¹

This presented challenges, as the concept of impact is often contested and debated – so how could it be possible to document demonstrable impact when there is debate on what impact looks like? In addition, even if we were able to agree on definitions of impact, many researchers are hesitant to claim that their project has achieved it. Of particular concern is the issue of attribution. In a complex world in which there is unlikely to be a simple linear relationship between a project and a societal change can anything be claimed? These challenges forced us to focus our attention away from big outcomes to processes and changes that suggested there were compelling narratives to be told. The process of the storytelling revolved around discussions with the research teams and identifying those small impacts that contribute to a bigger picture. This might be a discernible shift in the perspectives of a particular stakeholder group over time, a change in critical relationships, or an increase in the capacity of key groups to engage with research. The stories that emerged were tangible and honest and added up to an exciting description of how development research makes a difference in the world.

We started by taking inspiration from the work of others (Tilley, Ball and Cassidy 2018; Shaxson 2018). We looked at the process and formats that the Research Excellence Framework (REF) Impact Case Studies (ICS) take. This is the system that assesses the quality of research across all UK universities. Here the ICS narrative describes how research, conducted during a specific timeframe at a named institution, resulted in a change and had an effect on or benefited culture, the economy, the environment, health, public policy, quality of life, or society, using qualitative and quantitative evidence.

While these direct approaches have their place, and efforts to simplify impacts can make it easier to capture and tell the story, they can miss the point. The focus on a specific timeframe and the attempt to demonstrate how one project impacted ‘culture’ or ‘economics’ highlight the tension between attribution and contribution. We realised that our stories could not, and should not, attempt to claim one project as the sole reason for social change. Instead, our stories would demonstrate the different dimensions of impact that responded to the definitions set out by the ESRC impact toolkit (ESRC 2020a; ESRC 2020b). We were also inspired by the simple techniques and clear impact success stories presented by organisations such as UK Collaborative on Development Research (UKCDR 2020) and FCDO’s (formerly Department for International Development’s) Stories of Change (DFID 2014). We recognised that in the context of social science, and its ability to influence meaningful social change, it is essential to embrace the diverse ‘ingredients’ that contribute to shaping the bigger picture.

To identify those moments, we knew that it would take time to build trust and develop relationships with research teams so that we could have honest discussions. We started from a belief in the need to work collaboratively with the researchers. Our approach to co-creating stories of impact was created partly to strengthen skills and networks. We did not focus solely on projects that had recently closed or reached some critical point of uptake (however one defines this). Instead, we engaged with anyone who had an interesting story to tell and encouraged researchers to share incremental change and learning. Both live and long-closed projects had important learning to share around responding to emerging policy agendas, influencing practice, and engaging marginalised communities.

The result has been that we have produced a series of impact products – a collection of stories, chapters, and films.² Behind the production of these are strong relationships that have been built between researchers, with a view to deepening our collective understanding and appreciation for research uptake and impact. It is our hope that these connections and ways of working will continue to feed into their research long after the Impact Initiative programme has finished. It has not been a linear path, there have been challenges and diversions along the way, but it has been one of shared learning and demonstrates an important approach to supporting and promoting the impact across such a diverse research portfolio.

How to tell stories of impact

The importance of compelling narratives that make the impact of research less abstract and more concrete has been an important feature of impact assessment in academia (Davidson 2017). Discourse analysis of the highest scoring impact case studies from the UK’s REF showed that case studies that clearly articulated benefits to specific groups and provided evidence of this scored much better (Reichard et al. 2020). The impact stories that we have produced are clear and succinct and draw on supporting evidence. To make them accessible, we intentionally kept them short (two pages) and we stuck to a simple structure, with terms of reference developed and shared with each of the authors, who were always members of the research teams. The stories focus on the issue that the research planned to address, the action taken, including how they overcame any challenges, and the impact that the research had. Importantly, they all include further reading to evidence the impact in detail. We made it clear that this narrative approach was not a rigorous evaluative exercise. However, despite their brevity, the impact stories are told in a way that ensures that the research activity and subsequent impact is neither simplified nor exaggerated. They all provide links to further studies and background reading to add to their depth.

When it came to writing the impact stories, we encouraged researchers to demonstrate their impact by identifying at least two modes of impact as set out within the ESRC-FCDO Guiding Principles on uptake, impact, and communication of research framework of impact (Figure 5.1). The Impact Initiative had modified this to include a fourth mode of impact around strengthening networks and relationships, which reflects our understanding of impact as a process rather than simply as a set of outcomes

1. Instrumental: impacts on policy and practice – a change in direction attributable to research.
2. Conceptual: changing ways of thinking, raising awareness and contributions to knowledge.
3. Capacity building: building capacity of researchers and intermediaries to strengthen research uptake approaches.
4. Networks and connectivity: building and strengthening networks, connecting the supply of evidence with the demand for it.

Figure 5.1: The Impact Initiative’s wheel of impact

Source: Georgalakis and Rose (2019: 2).

We encouraged the researchers to think about their intended impacts and the pathways they were pursuing to achieve them. In having these conversations we honed in on the ‘micro-impacts’ (Clark and Goodier 2019) – that conversation with a politician, that email from an adviser at an international agency, a line in a policy document, a new committee with diverse representation in a village. Our view was that, across the research portfolio and in the broader policy agendas, these micro-impacts add up to more than the sum of their parts. The list was surprisingly broad, and surprisingly tangible. We then focused on how this impact could be presented in a narrative structure and evidenced with images, quotes, and further information.

Specific types of evidence included:

• Reference to the research in policy documents
• Reference to research by policymakers, e.g. letters from ministers or officials supporting the story being told
• Reference to research in practitioner guides
• National statistics that give indicative evidence to support the story
• Survey evidence from individuals who have benefited.

The process of writing these stories very much depended on the capacity of the researcher to articulate their story. In some cases, researchers took the story and ran with it with little more than copy-editing and support to layout the template; at other times it was led by a communications and impact professional and involved a lot of back and forth between the communications professional and the researcher to really get to the heart of the story. Frequently, projects had only anecdotal evidence of impact, and considerable time was spent by the Impact Initiative in following up with stakeholders to obtain quotes and other validations of claimed changes in capacity, understanding, and evidence use.

Working across so many stories led to us producing our Impact story recipe book (Impact Initiative 2018)³ as a way of demonstrating a variety of ingredients for impact. At workshops we also encouraged researchers to use our handy blank template at the back of the book so that they could practise telling their own stories of impact.

I presented the impact story⁴ in my own words and style in dialogues with government leaders and practitioners in Sierra Leone, Kenya, Tanzania, Uganda and India on a continual basis over the past three years. It helped people appreciate the difference between top down and bottom-up approaches and to see the value added of the latter.

(Michael Wessells, Professor of Clinical Population and Family Health, Columbia Mailman School of Public Health and Poverty Alleviation Research Fund Grantholder, pers.comm. 2019).⁵

Collectively the stories demonstrate the multiple pathways and approaches that can be taken to achieve impact. Notably, despite the simple structure, no two stories are the same. They also present the richness and diversity of the projects funded by the ESRC-FCDO Strategic Partnership. This accumulation of stories highlighted context and key moments of change and understanding but were also accessible in tone and language. As Mandu Reid, Leader of the UK Women’s Equality Party, said when reflecting on the portfolio of research on gender, ‘to meaningfully tackle persistent poverty – particularly how it manifests for women – [research] must encompass a much broader range of considerations. The selection of ESRC-DFID-funded research presents pertinent examples of how this applies in practice’ (Impact Initiative 2019c). This point is crucial, as we know that stories tap into the way that people make decisions. Simple storytelling mechanisms of plot, central characters, and context help transcend complex information for time-poor people (SPARKOL 2018). They also help bring the complex research to life, making it more relatable and inspiring change. In humanising the stories, we are able contribute to a richer understanding of complex development issues. The accumulation of smaller impacts within a body of work using this narrative style provides a powerful record of how investments in social science contribute to development and learning.

Moments of impact and change

We fully acknowledge that evidence comes in many forms, and impact can appear in so many ways. Pamela Mason, Strategic Lead for International Development, ESRC, explained in an interview⁶ with the Impact Initiative that the potential impact of research goes way beyond policy and different types of impact:

We talk about policy change as kind of this gold standard impact… It has the most far-reaching consequences, but actually much more incremental impact are equally valuable. Capacity building, training people to conduct research but also in terms of practice in their own work and their own skill-set, changing the discourse around the subject and increasing the body of knowledge and the understanding that we have about a policy related subject, but not necessarily going so far as changing that policy.

(Mason 2018)

We worked closely with the researchers to identify those moments of impact and change, while also acknowledging that impact within a small project is more often about contributing to change as opposed to ‘owning’ it. At a macro level this led us to look at the different research methods and activities that projects did to bring about change at various points (i.e. not just thinking about impact in terms of influencing policy alone, but the journey of impact overall). This informed our work to identify a set of six practices (Clark and Goodier 2019) that could lead to research impact. At the micro level, we felt it was really important to document the different levels of impact – from local to global, conceptual to instrumental; the collection of impact stories presents a picture of a programme’s activities and contributions to impact over time.

While written stories have a long shelf life, the spoken word often has the added advantage of giving a hint at the people behind the story. At the Power of Partnership: Research to Alleviate Poverty event in 2018,⁷ we approached storytelling from another angle. We invited participants to share their stories in the impact storytelling booth. The ‘booth’ was in fact a cosy side room that provided a space to break away and think about the basics – we asked individuals to record their name and present a brief overview of their project and why it mattered, being clear about the challenge the research addressed, and what impact or intended impact happened as a result.

The storytellers only had three minutes to tell the story before the app on the iPad stopped recording (Sahle 2018). What could possibly go wrong? It turns out not much actually – once people were in the room they were happy to tell their stories and sometimes they even recorded one story and then came back to tell it again from another angle or, occasionally, another language. These are not polished videos, but they allowed researchers to tell their stories out loud and in just three minutes. This encouraged the researchers to think about those moments of impact and the changes they had made in their research that were really worth sharing. After the videos were complete, we checked to see if new stories had been told and in a few cases we went on to work with the researchers to capture the impact in written accounts. It was also a moment in time to hear these stories and see if there were shared experiences, overlaps, and impacts that pointed to the bigger picture of impact.

Dissemination and engagement

Every output produced by the Impact Initiative has been accompanied by an engagement plan. This makes it clear who needs to hear the story and what influence you hope the research findings can have. This distinction between dissemination and engagement is important. In moving away from the linear ‘pump it and share’ model of dissemination, we entered a dialogue with researchers to share and build better understanding of the issues. Often this helped to determine follow-on formats such as blogs or videos – most of the time it provided new networks and rich discussions.

It is also important to note that, while the Impact Initiative is active on social media and has connections with different networks to disseminate and media relationships, often the most effective way to engage is for the researchers to share their messages and stories themselves. This is echoed in the feedback from grantholders – when they have their impact story or Research for Policy and Practice paper in hand, they are equipped with a concise and powerful message to give to different stakeholders.

Victoria Brown, education, monitoring and evaluation and development Consultant and Raising Local Outcomes (RLO) grantholder, explained that her impact story⁸ was:

A great way to succinctly share our results and learnings and engage policymakers, practitioners, and other researchers in dialogue about why mother tongue education matters – and why it should be further funded and researched. We have used these findings to help influence the next phase of early grade reading programming in Uganda, and to inform bilateral programming from USAID and DFID, for example, as the next phase of reading programming is designed for Uganda. We have also shared this with the IGC and Uganda’s National Planning Authority during the ongoing design of Uganda’s new National Development Plan, which has a large focus on education.

(Brown pers.comm. 2019).⁹

Ricardo Safra de Campos, Lecturer in Human Geography at University of Exeter and RLO grantholder, met and subsequently collaborated with other researchers funded by the Joint Fund at the Power of Partnership event in 2018. They went on (supported by the Initiative) to host a policy event on the cross-cutting issues they work on. He describes not only the impact of the published outputs but also the fact that he was well positioned with these outputs to get them into the hands of the people that they wanted to influence.

The Impact Story¹⁰ along with the Research for Policy and Practice paper¹¹ have enabled dialogue with other researchers working on the field of migration, displacement and urbanisation. Those outputs have also facilitated engagement with urban planners interested in using participatory methods for urban governance. The Impact Story has been useful in following on funded work which is engaging with urban planners in Chattogram, Bangladesh. It was a useful instrument to show evidence of the impact of our research among policy and planning circles. The document was shared with both Minister of Planning and Disaster Management of Bangladesh as well as other national and international stakeholders.

(Safra de Campos pers.comm. 2019).¹²

A recurring challenge raised by researchers around the impact agenda is that the ‘big moment’ to share their research might come before they are ready, or long after they have shared their findings. It is here that we see the strongest justification for programme-level knowledge-brokering functions supporting researchers to be better positioned and prepared to share their learning irrespective of project life-cycles. If the moment came before the research was complete we needed to work with researchers to share what they already knew, being sure not to overclaim. If the moment came long after the research had finished, it was our job to go back to the researchers and provide opportunities to revisit, collaborate, and contribute their insights to a live debate.

This was true for the impact story Keeping African Girls in School with Better Sanitary Care (Dolan and Tofaris 2018).¹³ We drew on this story and worked with the lead researcher at several points – to feed into engagement at the 63rd UN Commission on the Status of Women in 2019 and media engagement as the topic arose in conversation internationally and in the UK. It also presented the opportunity to link the researcher with other researchers and policy actors working on connected issues. This story was also included in the Rethinking Impact: Applying the Gender Lens collection (Impact Initiative 2019a).¹⁴ This pack collected many of the impact stories, policy papers, media pieces, and other resources that in some way addressed gender equality. By including the story in this new compilation, we were able to give it an additional push to new audiences. For example, this pack was shared at events coordinated by the UK Department for Education, Period Poverty Taskforce, demonstrating the importance of mainstreaming gender throughout research and policymaking.

Conclusion

Stories are powerful, and told in the right way they can spark ideas and emotions and bring about change. While there are many ways to document research impact, often they can be perfunctory and very narrow in their focus. In focusing on the end impact goal, impact stories can miss the crucial moments of incremental and procedural change that occur over time. By creating clear, simple, but effective storytelling processes we have enabled researchers to tell those stories. Yet beyond the storytelling itself, the Impact Initiative has helped to build lasting relationships while deepening understanding of outcomes and the multiple processes employed to support wider learning.^15,16 It is also our belief that this approach has encouraged and enthused many in the academic community to see how they can tell their stories of impact and that often those stories will have value and use well beyond the lifetime of the original research project.

Working across diverse portfolios of research, knowledge-brokerage programmes such as the Impact Initiative are in a prime position to help researchers step back, spot connections, and produce compelling impact narratives. Not only do these stories highlight the achievements of individual projects, they collectively generate deep learning on the impact agenda itself and provide a compelling justification for investing in research to improve the lives of people around the world.