In May 2021, one of Start Network’s local members, from the Democratic Republic of the Congo (DRC) responded to the Nyiragongo volcanic eruption which saw 450,000 people displaced and evacuated. Two months later, an independent monitoring review took place to reflect and learn about the effectiveness and impact of the response. However, as Start Network is proactively working to decolonise evidence and challenge western perceptions of what makes a ‘successful project’ it took this opportunity to also ask the affected community “how should humanitarian interventions be evaluated?” Questions were asked to explore local perceptions of programme success and find different ways to measure these, in order to ensuring Start Network and other aid agencies are held accountable by people that have been affected by crises. We engaged with a local consultant to interview a small sample of 10 members of the affected community (6M,4F) living in the disaster camps and surrounding areas to hear their ideas on this topic. As well as answering questions they also made some suggestions of important questions that we need to be asking during our evaluations.
The feedback we received highlighted three core areas which we should be considering in future evaluations, which are summarised below:
Meeting the real needs
Unsurprisingly, finding ways to evaluate whether the interventions met the true needs of the affected community came out as important across all of those interviewed. Taking time to explore the relevance and quality of the help was emphasised throughout.
“The population has multiple needs so I would say the NGO must do everything to identify the most important need and meet it.”
– Male community member, living in Munigi Village
“[You need to evaluate]…whether the aid corresponds to the demand of the victims, and whether the aid has been distributed equitably”
- Female community member, staying in the temporary disaster camp
Staff behaviour
The behaviour of, and interaction with agency staff was the area which participants spoke about the most. However, it is worth noting that this may be a reflection of which topic they felt most comfortable and confident to discuss, rather than which area they felt was the most important indication of a project’s ‘success’. Within this, three key areas to investigate and evaluate which emerged were:
1) How the crisis-affected population felt about their interactions with staff and whether they felt listened to was highlighted as important by two of the participants.
“NGOs must have staff able to communicate with everyone, not members of staff with a superiority complex who think that the disaster victims do not have his level.” - Male community member living in Munigi Village
2) The level of respect and dignity with which they are treated, whilst monitoring any discriminatory behaviour. Some of the interviewees mentioned having seen or being aware of hitting, jostling, fighting and favouritism happening at distribution sites and being treated in unkind and unacceptable ways. Ensuring dignity without discrimination was perceived as paramount to defining a project’s ‘success’.
“Staff behaviour is also important because even if we are disaster victims, we deserve respect. But some NGOs that distribute aid in Goma or in other sites push people around, and especially in the case where the military is providing security - sometimes they hit you” – Male community member, staying in the disaster camp
3) The collaboration and involvement of the affected community in both the design and implementation of the response. Many participants felt that they could add significant value to the identification of needs and expressed a desire to be part of the aid distribution with the aim of reducing community conflict, which they said was often the case when aid was distributed through military personnel. As part of the independent monitoring, participants were asked how they thought they should be involved in projects. Respondents felt that NGOs and communities needed to work together to agree on responses and find solutions together. One respondent said that they should be involved in the whole process however respondents expressed that this isn’t currently happening as NGOs do not consult leaders prior to interventions. This could mean NGO’s may not fully understand the needs of communities.
Beyond the immediate
Considering the impact of the humanitarian intervention and whether it continued to serve the population once the intervention has ended was emphasised. Participants said that the evaluation should focus on whether the help had changed the affected persons situation positively after the intervention, and to consider how (if at all) the aid was enabling them to live independently in the long run. They felt it was important to consider the affected communities current needs to inform an evaluation. Meeting real needs, staff behaviour, and beyond the immediate impact of an intervention were the three core themes which were identified as important indicators of project ‘success’ by crisis-affected community members interviewed. As well as these areas, respondents suggested some questions which we should be asking during future Start Network evaluations.
Closing thoughts…
It’s encouraging to see these questions are not wildly different to some of the areas independent evaluations or monitoring reports explore currently. However, ensuring we capture how affected populations are treated, listened to, and worked with during a crisis is of significant importance in considering whether an intervention is considered ‘successful’ through the eyes of the communities affected by disasters. This is an on-going research theme for Start Network and we will be holding a session on decolonising evidence as part of the 2021 General Assembly where panel members will be leading an in-depth discussion on this. Further progress has also been made on how evaluations should take place in recent research on Pakistan’s heatwave response. The blog discusses the intervention itself and also how we could have measured the success of the activity differently. There is also community led innovation work being developed in the Start Network Hubs where communities self-identified needs are responded to through innovation methodologies which are then monitored through tools which have been developed in collaboration with the community. Alongside continued research, we will be reflecting on and changing the tools and approaches used to evaluate interventions as it is only at this point, where research meets practice, that we can truly work towards shifting accountability closer to those most harshly affected.