Skip to contents

Vulnerability Score Calibration

The usage of Vulnerability Scores is among the key tools to implement when specific humanitarian Interventions, such as Cash Based Assistance, requires to be targeted and prioritized for certain beneficiaries.

One challenge is to configure scoring formulas that allows to reflects compounded vulnerabilities, i.e. when different criteria interact with each others. It is therefore important to prevent “compensability” (meaning one criteria can be compensated by another one). To do that, geometric aggregations of indicators should be preferred over arithmetic aggregations (aka multiplying rather than summing up indicators). This implies to properly weight each of the criteria used to build the scoring formula.

In an ideal world of statistical inclusion, a representative dataset of the population should be accessible and such weighting would be performed using a variety of statistical models such as Proxy Mean Testing or Item Response Theory to name a few… In the reality of most humanitarian contexts, using expert opinions remains the default option. In such environment, experts can mean both humanitarian workers, authority representatives and refugees and other displaced communities leaders. Providing an opportunity for such local community leader to have their word to say is indeed a key element of accountability to affected population. One challenge in such context is therefore to generate a consensus on the selection and relative importance of assistance targeting criteria.

“If we have data, let’s look at the data. If all we have are opinions, let’s go with mine.” Jim Barksdale

The VulnerabilityScoreCalibration package provides easy access to advanced methods for the calibration of vulnerability scorecard through expert opinions. It aims at enabling a process to quickly build buy-in and consensus among consulted experts while ensuring full transparency. It combines two approaches based on specific type of surveys that can be used to consults the experts:

  • Quadratic Voting: A decision-making and consensus-building method where voting power increases quadratically, allowing individuals to express stronger preferences. It is used to assess the relevance of indicators (selection) -See an example of quadratic voting form.
  • Conjoint Analysis: A research technique that assesses preferences by presenting individuals with various criteria in combination to determine their relative importance. It is used to measure the importance of the criteria levels (weighting) - See an example of conjoint review form.

Typically, this would be used in the context of a dedicated short workshop where experts will be given to the opportunity to reflect on what vulnerability means within their specific context and understand the requirement on criteria formulation to build effective scoring formula. Then the two consultations are organised (basically a survey to fill within kobo) and expert can quickly review the results to ensure consensus building within a short time frame.

Please consult the vignette for a more thorough presentation

Process

The objective of the process is to create a vulnerability screening questionnaire in kobotoolbox. The questionnaire will include calculated indicators that will summarize the qualitative response through a quantitative index. For the questionnaire to be short, user-friendly and effective, it needs to include only the minimum set of questions and response that are identified by experts as the most relevant to the specific operational context.

To implement the approach, beside the workshop facilitation, key technical tasks needs to be performed. They mostly consist in the customization of standard questionnaire and the generation of the results in a format that can be easily consumed by experts within the workshop. For effective facilitation, those tasks should be performed as quickly as possible during the time frame of a minimum one-day presential workshop.

This approach has been initially designed and now extensively tested by UNHCR within the Americas Region. The resulting tools have been packaged within 4 functions reflecting the different technical tasks in the process:

  1. quadratic_prepare - based on an initially filtered list of indicators (max 25 indicators in 5 groups of max 5 indicators), Data expert build the xlsform file used in kobotoolbox to consult the experts about the relevance of each indicators.

  2. quadratic_review - based on the form and data collected from the experts, generate a slide deck used to review the results from the consultation. This slide deck should be used through a joint data interpretation session to further narrow-down the initial list of max 25 indicators. Field Experts are being introduced to the process before filling the kobo form -Eventually this stage is repeated if there is a lot of disagreement among experts. The output is the selection max 12 indicators for the next stage

  3. conjoint_prepare - based on a restricted list of criteria and response options within each criteria, build the xlsform file used in kobotoolbox to consult the experts about the importance of each criteria. This involves the mapping of responses options for the questions linked to the indicator to max 3 to 4 vulnerability criteria levels.

  4. conjoint_review - based on the form and data collected from the experts, generate a slide deck used to review the results from the consultation together with a xlsform file with the weights for the score. -Eventually this stage is repeated if there is a lot of disagreement among experts. Then based on the mapping of criteria to response options , the screening questionnaire with calculated scores as xlsform is generated and Data expert can revise the final xlsform for vulnerability scoring implementing the weights defined through the previous stage

In addition, the package includes a default library of indicators/criteria/questions/responses that can be used to quick-start the design. Please do not hesitate to contribute to the improvement of the library by sending a ticket

Usage / Installation

The package functions are directly accessible through a dedicated shiny app @ http://rstudio.unhcr.org/Vulnerability_Score_Calibration

To run the function locally from within Rstudio Desktop, once installed the right version of Rtools, you can install the development version of {VulnerabilityScoreCalibration} from GitHub with:

install.packages("pak")
pak::pkg_install("unhcr-americas/VulnerabilityScoreCalibration")

Feel free to raise issues and ask questions through a ticket