As a funder of research, assessing the quality of that research is of course on the top of our minds. We have well-established criteria and matrices: not that they provide infallible truths, but peer reviews if managed well give us very reliable assessments. Nuanced understanding of what makes high quality research and how we assess this in the research for development community is essential to promote research that stands to make a significant impact.
At the International Development Research Centre (IDRC), part of Canada’s international development programme, we aim to support the capacity of researchers as well as strong evidence. This is to ensure that research is grounded in specific contexts, and that local researchers are in the driving seat of building strong evidence and informing national development policy.
North or South Evidence?
And so we were extremely happy when an external, in-depth evaluation found that in the IDRC portfolio Southern-led projects fared as well as or better than Northern-led ones. Southern-based research and partnerships – which we at IDRC value in themselves – also deliver quality research, and this can be measured. This was summarised in an article by Jean Lebel and Robert McLean in Nature, ‘A better of Research from the South’.
What I conclude is not that one is better than the other. There is a large variation in quality in both, and differing elements contribute to quality. But what’s important is that there are no necessary trade-offs between good evidence and capacity building. This has important implications, also, for supporting research partnerships.
What does research quality look like?
To uncover these findings, the evaluation used the Research Quality Plus (RQ+) approach. This offers a first step toward a holistic understanding of the value of research for development. Based on long-standing work on outcome mapping, RQ+ provides a way to assess the value of research, which goes beyond project outputs, and includes potential impact in terms of the projects utility and capacity building.
Of course, these dimensions are very hard to assess, particularly as impact often occurs well after the end of a project – but RQ+ now offers us a way to bring these critical components of good research into the picture. For example, it accepts the growing understanding that research influence and impact begin before and through the research process, and so, RQ+ asks evaluators to appraise the extent to which researchers have positioned their work for uptake and long-term influence.
The framework can be summarised in a spider diagram as below, with quality scores assigned through peer review, in five dimensions (the scores are indicated by way of illustration only). Our experience shows that this can be applied in a dynamic ways, assessing the positioning for project success including in terms of uptake and capacity building.
Source: The above diagram draws on Figure 2.2 'Example of a Diamond Map Assessment of a Research Project' in GrOW Formative Evaluation for Mid-Term Review: Evaluation Report - February 2017 (Final Report) (page 12). Original © Universalia 2017, all rights reserved. With kind permission: Nanci Lee at Universalia.
We put this into practice in a review of projects under a joint research programme by the IDRC, the UK’s DFID and the William and Flora Hewlett Foundation, GrOW. Using the RQ+ approach for the midterm review, a lesson that emerged was how important partnerships are for the success and impact of research projects. The success of a project led by Carleton University on the role of small-scale mining for the livelihoods of marginalised women in central Africa built on a range of well-established partnerships: a multi-disciplinary research team at Carleton working closely with researchers in the DRC, Rwanda and Uganda, and in close alliance with the international NGO IMPACT (formerly PAC). The project engaged with country-based partners that actively incorporated the research findings, in both international advocacy and facilitating the voice of women miners.
Similarly, a GrOW-supported project on the impact of child care provision for women’s economic outcomes entailed close international academic collaboration (McGill University in Canada and APHRC in Kenya). The project in urban slums in Kenya showed that providing child care for poor women has the potential to benefit women, their children and the economy. Benefiting from the international partnerships, the results were presented at international fora, as well as to Nairobi city officials and the Kenya Private Sector Alliance (KEPSA), contributing to continued advocacy.
Finally, the partnerships also facilitated capacity building, in mutually advantageous ways. The projects provided opportunities for young researchers to grow their leadership role, in both research and policy engagement, with the support of senior researchers and the IDRC team implementing the programme. The team working on the mining project, for example, concluded: ‘Improved skills in conducting field research on gender dynamics was an outcome for all team members and researchers. We have all become better gender researchers as a result.’