Topic
In many important respects, all development is local. That is, the processes of poverty reduction, or livelihoods creation, that matter most take place at the micro level of individuals and their households. However, for too many impact assessment processes in the impact investing industry, the analysis stops at the investee enterprise—and the household remains invisible and under-analyzed, if it is analyzed at all. Alternatively, ultimate impacts in a theory of change are framed at the broadest, societal level, with the household again left out of the frame.
Yet longstanding research on poverty from other fields—agricultural economics, development economics and gender studies, to name three—underscores the fundamental importance of the household as a key unit of analysis for evaluation and monitoring of any development intervention, including impact investments. Indeed, when new revenue is generated from a job with an investee company, or from surplus accruing to the owner of an investee firm, that revenue almost always, in whole or in part, flows into the beneficiary’s household.
How that household converts, or fails to convert, this incremental revenue into, for example, increased access to education programs or health services for its children, more protein-intensive foods for family meals, hard-asset improvements to the family home (e.g., a new roof or extension), or investment in the inventory of a new family business (e.g., vegetable trading, small-motor repair) determines to a great degree the ability of that household to cope with and reduce its poverty conditions and create better livelihood pathways for its members.
Systematic evaluations of impact investments therefore must be based on theories of change that incorporate the household and employ data collection and analysis tools that are sensitive to changes within that unit of analysis and are cost-effective. One example is the Progress Out of Poverty Index, a field-based observational tool for which the Grameen Bank has developed a mobile application. A second tool also originally tested in Asia is the Citizen Report Card, which enables individuals to assess the effectiveness and efficiency of the services and products they use. A third example is the Qualitative Impact Assessment Protocol, or QUIP, which examines self-reported attribution through streamlined semi-structured household interviews and focus groups. Other poverty research points to specific, granular and well-tested household-level metrics that can be used.
Emerging thinking on the broadest range of impact evaluation strategies, such as that by Rogers, Rogers and Peersman, and Stern et al, can be tapped in designing household-level assessments. Such impact-evaluation strategies may include, but are not at all limited to, randomized controlled trials or other counterfactual designs. In fact, impact-evaluation designs which take into account contextual factors, and which are capable of assessing positive and negative, direct and indirect, and intended or unintended results of an impact investment, are especially appropriate at the household level.
Tool
Impact Investment Results Chain
Source: E.T. Jackson and Associates Ltd. 2016
Exercise
An impact investment fund in East Africa makes equity investments in and provides loans to small and medium-sized businesses in poor regions. It has been making these investments for five years, and has been an active and public advocate for SMEs as vehicles for poverty reduction. Now some of the fund’s own investors—a local foundation and a foreign DFI—want to better understand the link between SMEs and households in the poverty reduction process and have commissioned an impact evaluation. You and your colleagues are asked to serve as consultants to the team designing the evaluation. Form small groups, choose a chair and rapporteur, and take 30 minutes to answer the following questions: What are the top three to five issues you recommend be examined in the evaluation? What obstacles do you anticipate the evaluators will encounter when they collect data in the field? How can they overcome or minimize these obstacles? Record your responses on flip charts or slides. Your rapporteur will have five minutes to present your group’s advice on these questions, in a facilitated plenary session.
Readings
Alkire, S., C. Jindra, G. Robles and A. Vaz. Multidimensional Poverty Index – Summer 2016: Brief Methodological Note and Results, University of Oxford, 2016. https://www.researchgate.net/publication/304077670_Multidimensional_Poverty_Index_-_Summer_2016_Brief_Methodological_Note_and_Results
Copestake, J. The Qualitative Impact Assessment Protocol, Centre for Development Studies, University of Bath, 2016. https://www.gov.uk/dfid-research-outputs/the-qualitative-impact-assessment-protocol-quip#contents
Grameen Foundation. Progress Out of Poverty Index, Website: http://www.progressoutofpoverty.org/
Rogers, P. Overview: Strategies for Causal Attribution, Methodological Briefs: Impact Evaluation 6, Office of Research, United Nations Children’s Fund (UNICEF), Florence, 2014. https://www.betterevaluation.org/tools-resources/overview-strategies-for-causal-attribution
Rogers, P. and G. Peersman. Impact evaluation: challenges to address, Better Evaluation, January 23, 2015. http://betterevaluation.org/blog/impact_evaluation_1
Stern, E. et al. Broadening the range of designs and methods for impact evaluation, Better Evaluation, 2012. http://betterevaluation.org/resources/overview/impact_evaluation_methods
World Bank. Citizen Report Card and Community Score Card, Washington, DC., No Date. https://www.betterevaluation.org/sites/default/files/community_scorecard_process_worldbank03.pdf