Vulnerability Score Calibration Process
Source:vignettes/VulnerabilityScoreCalibration.Rmd
VulnerabilityScoreCalibration.Rmd
#library(VulnerabilityScoreCalibration)What are vulnerability scores?
Similarly to credit score in the private sector, vulnerability score offers a way to combine in a single number (i.e. a score for instance between 1 & 100) a composite measurement of vulnerability status in order to assess if a person can qualify for assistance
When do you need to use vulnerability scores?
Necessary once the population size exceed a certain scale – case by case management is not do-able or comes with high risk of exclusion
Minimize subjective component in assistance allocation
Measurement allow to prioritize assistance (Cash, but also potentially training, livelihood, micro-credit, etc.) in function of available budget
When not all eligible people can be prioritised….
In protracted situations, not all eligible persons can be prioritised…
When do you need to use expert opinions?
When we can not use simpler targeting approach - that does not require to distinguish eligibility from prioritization
When we have experts….
When there’s no representative dataset available
When we have dataset that represents statistically well the whole population, it is possible to use statistical model, either:
proxy means testing: aka one outcome indicator can be used as single proxy of the vulnerability situation of the case (poverty, deprivation, etc.. )-> supervised classification
cluster population in consistent profiles that comes out from the data -> unsupervised classification - Item response Theory
But in most humanitarian situation, achieving good statistical representativenesss is challenging (volatile environment, hidden population to sample..)
What are the challenges with expert opinions?
agree on the relevant eligibility criteria
agree on the relative importance of each criteria
agree on what other are agreeing…
Without a specific facilitation approach, this can end into lengthy discussions…
An organised workshop facilitation
6 simple steps:
Step 1- Expert training
Step 2- Select the relevant eligibility criteria: quadratic voting
Step 3- Adjust the weight of criteria: conjoint analysis
Step 4- Review results and potentially iterate
Step 5- Implement the formula in the vulnerability scoring form
Step 6 - Apply Eligibility and prioritization threshold
Step 1- Expert training
“It is not enough to do your best; you must know what to do, and then do your best.”
W. Edwards Deming
- Understand the difference between output variable and eligibility criteria
Step 2- Select the relevant eligibility criteria: quadratic voting
Technical step by step tutorial: set up a quadratic vote on Kobotoolbox
Step 3- Adjust the weight of criteria: conjoint analysis
Conjoint analysis is a type of consultation designed to measure the average opinion from a group of experts through specific pool where experts should compare different stereotypical profiles one by one and assess their respective vulnerability level.
Quotes:
“If we have data, let’s look at data. If all we have are opinions, let’s go with mine!”
Jim Barksdale
Technical step by step tutorial: set up a conjoint analysis on Kobotoolbox
Step 5- Implement the formula in the vulnerability scoring form
Kobotoolbox form are often the default tool: The average weight obtained through conjoint analysis are implemented through a calculated filed using pow function
Do not display the score during the screening - only provide information on eligibility
Conclusion
When there’s no representative data, expert opinion is the default options
Creating buy-in on how to calibrate the vulnerability scoring formula is a key to the social acceptability
Combining organised consultation (quadratic voting and conjoint analysis) allows to leverage collective intelligence
As soon as you use a scoring system, the management of fluctuating prioritization threshold (i.e budget…) implies an efficient case management system: - Assistance cohort management - Continuous Vulnerability scoring - Appeal system