In 2014, roving health workers swept through neighbourhoods in Lahore, Pakistan on a drive to achieve complete immunization in one of polio’s last remaining bastions. Meanwhile, their managers struggled to track their progress using antiquated methods.
In this context, a team of researchers and policymakers conducted a pilot project, testing if a simple mobile phone app and online dashboard could open up a new flow of data between civil servants at different levels. This gave managers an improved means to monitor the tempo and geographical coverage of vaccinations and use that information to motivate higher performance among health workers.
This case study focuses on how quality information, strategically employed, can swiftly improve governance. It illustrates how a relatively simple tool – which leverages insights from behavioural economics and management science, and is adaptable far beyond the realm of public health – can overcome information bottlenecks between government actors, resulting in better service delivery for the poor.
Pay scales in government jobs tend to be relatively flat, and incentive programmes modest.
The rationale is sound: policies that grant managers too much discretion in doling out sweet positions and big bonuses give them opportunities to indulge in corruption and cronyism. And so, broadly speaking, public-sector employees receive uniform incremental pay raises year by year while their private-sector counterparts take part in dynamic incentive programmes and compete for performance-based raises.
In a textbook example of an unintended consequence, this widespread public-sector practice designed to solve problems in one area causes problems in another: public servants often have less incentive to excel than private-sector employees. Too often, the end product is deplorable public service delivery, particularly in developing-country contexts with weak monitoring and accountability systems. A large-scale World Bank study published in 2006 used unannounced visits to measure public-sector employee attendance in six developing countries on three continents, and found that on average 19 percent of teachers and 35 percent of health workers were absent.
Over the past two decades, governments and researchers have explored a wide range of policy tools to incentivize better performance among public servants in ways that cannot be gamed. These have included innovative monitoring techniques and methods to screen candidates with the right type of intrinsic motivations for public service.
Incentive programmes, such as pay for performance (P4P), have shown promise too, particularly in public health. These programmes have resulted in better preventive care for toddlers in Rwanda and reduction of anaemia in schoolchildren in China. In realms outside health care, P4P programmes have resulted in higher revenue collection among tax inspectors in Pakistan and higher classroom test-score averages among teachers in India.
Maternal and child health
Anaemia in school children
Test scores among children
Sample size: 116 health facilities in 19 districts
Date: 2006, published 2011
Results: Treatment group facilities showed a 23% increase in the number of deliveries conducted by professionals in a clinical setting. Preventive care visits increased by 56% for children 23 months or younger, and by 132% for children aged 24–59 months.
Caveat: No change in the probability women received any prenatal care or had more than 4 prenatal care visits.
Sample size: 3,553 students aged 9–11 in 72 primary schools in Northwest China
Date: 2009–11, published 2012
Results: School principals received bonuses for every student who changed from anaemic to non-anaemic. The treatment reduced anaemia compared with the pure control group by about 23%.
Caveat: Incentivized school principals were more likely to use subsidies for iron-focused supplements, whereas non-incentivized school principals used subsidies for supplements that could affect both iron and overall calorie intake.
Sample size: of 3,100 villages receiving community-based block grants, a subset received incentives based on performance on health, nutrition and education indicators
Date: 2007–09, published 2014
Results: 8 targeted health indicators were about 0.04 standard deviations higher in treatment areas compared to control. These effects were about twice as large in areas with low initial levels of performance. The main health reduction was a 15% decline in malnutrition rates.
Caveat: Malnutrition effects became more muted over time, and the relative gain of incentivized to non-incentivized areas declined over time as non- incentivized areas improved.
Sample size: all 482 property tax units in Punjab province
Date: 2011–13, working paper 2014
Results: Treated areas had a 46% higher growth rate in tax revenue, and a subgroup that rewarded on revenue saw a 62% higher growth rate. The revenue gains substantially exceeded the costs of the incentives.
Caveat: Bribe rates increased in incentive areas, possibly to compensate incentivized tax inspectors for foregone incentive payments.
Sample size: 500 schools in 5 districts in Andhra Pradesh
Date: 2004–2007, published 2011
Results: After 2 years, students in incentivized schools had test scores 0.27 standard deviations higher in math and 0.17 in language. Effects were even stronger after 5 years: 0.54 SDs higher in math and 0.35 in language. Students also did better in non-incentivized subjects.
Caveat: No evidence of any adverse consequences of the programme.
The challenge is that P4P depends on quality information flowing between government actors: for the system to work, accurate data on worker performance must reach managers. Researchers often supply and maintain the necessary data infrastructure in the context of impact evaluations, but the risk is that once the programme is scaled up and the researcher moves on, the system breaks down or falls prey to tampering.
A further challenge is that the system must be relatively simple to implement or it will defeat its own purpose. Every minute a nurse spends filling out paperwork to get a bonus is a minute not spent treating patients.
These challenges multiply when the health worker is not stationed in a clinic, but roving neighbourhoods on a vaccination drive. How do you create a swift, portable, easy-to-use system that provides managers with the necessary data to motivate health worker performance without giving them opportunities to cheat?
There’s an app for that.
In 2014 a pilot project in Pakistan, part of the Building Capacity to Use Research Evidence (BCURE) programme funded by UK Aid, tested an innovative system of delivering data on worker performance to civil servants up the chain of command. The system channelled a number of forces for improved coverage in a polio vaccination drive: bonuses, monitoring, geographical mapping, and manipulation of workers’ intertemporal preferences. The project brought together officials from the Punjab health ministry, researchers from Evidence for Policy Design (EPoD) based at Harvard Kennedy School, and colleagues from the International Growth Centre, UC San Diego, and the University of Southern California.
Taking a close look at the project not only showcases an exciting new tool with applications far beyond public health; it reveals the critical importance of the exchange of quality information in optimizing service delivery in developing countries.
In the early 2000s, the world seemed on the verge of eradicating polio once and for all. Between 1988 and 2013 the number of cases went from 350,000 to 416, and the number of countries in which the disease was endemic went from 125 to 3. But in 2013–14 there was an increase in new cases. Of these, 85 percent were in Pakistan. According to the World Health Organization, cases were crossing the border from Pakistan into Afghanistan, and strains of the virus that originated in Pakistan were found as far afield as Egyptian sewers. It seemed that Pakistan’s faltering vaccination campaign was the chink in the world’s armour against the disease – troubling, because as long as one child has polio, there is a risk of renewed outbreak.
As EPoD’s Michael Callen wrote in a post on The Conversation, an academic journalism blog,
The reason Pakistan was having so much trouble didn’t come down to having enough doses of the vaccines or health workers to administer them – the country did. A key problem was that information about who was getting vaccinated wasn’t getting collected, and that the incentives health workers got didn’t actually motivate them to perform more vaccinations.
The health ministry conducted repeated vaccination sweeps without alleviating the disease. The result was that some children were immunized several times over and others, in small pockets isolated by geography or violence, not at all.
A metaphor from engineering comes in handy here: when one part of a machine causes a backup that threatens the functioning of the whole apparatus, it is called a “bottleneck.” The effects of one simple clogged artery can be catastrophic, and it is of utmost importance to pinpoint the problem and not mistake it for a migraine or a dizzy spell. In all kinds of design from machines to policies, it is important to identify bottlenecks and avoid mistaking their effects for a more pervasive, less localized problem.
Callen and Yasir Khan of the International Growth Centre had done a number of studies on how innovative management techniques and new technologies could open up information bottlenecks and improve outcomes in the public sector. In one study conducted with Saad Gulzar and Arman Rezaee, they had looked at ways to improve the deplorable attendance rate of employees in health centres in Punjab, and observed that attendance data was not making its way up the chain of command due to collusion among workers, and also due to outdated, paper-based record-keeping. They showed that public servants with a certain combination of traits, as measured by psychological personality tests, were less likely to collude and more likely to respond positively to improved attendance monitoring via smartphones. In this case, opening up an information bottleneck shows great promise to improve outcomes for millions, at a relatively low cost.
For some of the same reasons, Pakistan’s faltering vaccination programme can be viewed as a series of bottlenecks. Information on the performance of individual vaccinators was not reaching their managers; information on who has been vaccinated was not reaching the government. As a result, Pakistan itself was a bottleneck for the international campaign to end polio.
The first leg of the pilot project focused on helping get data about health worker performance to their managers, and helping managers use that information to improve performance.
In the status quo vaccination efforts, four or five roving teams of Lady Health Workers, or LHWs, worked to vaccinate a neighbourhood. They used paper maps like the one above, recorded their efforts on paper forms like the one below, and employed a protocol of writing in chalk on the walls of houses to indicate to other teams they had been there. For an independent monitor to check up on a team’s efforts, she would have had to physically follow their footpath through the neighbourhood and read the chalk marks.
It was a difficult job, the monitoring systems were weak and the pay low. Such factors conspired to create a situation where health workers often failed to achieve their targets, but reported that they had. Callen explains: “They weren’t being paid to meet the target, they were being told, You need to meet the target. So every single worker said they had met it. If you’re being explicitly paid to meet a target where no one can verify how much work is being done, then of course you’re going to say you’ve succeeded.” This was a situation ripe for P4P incentives, but this was impossible as long as information stagnated in antiquated, paper-based systems.
The research team and their counterparts at the health ministry developed a three-fold plan. With support from the BCURE programme, they distributed cheap smartphones to the health workers and constructed an online dashboard. This new data infrastructure allowed health workers to send geo-coded information about their activities to the dashboard where their managers could track their activities.
These two components enabled a third aspect of this intervention, whose evaluation was funded separately by the International Growth Centre and may represent the most original insight of the project – one that promises big efficiency gains for policymakers. The team tripled health workers performance-based bonuses as part of a new incentive structure that worked in a new way with the intertemporal preferences of the workers. A quick look at the concept may reveal its importance.
A major contribution of behavioural economics – the marriage of economics and psychology that put terms such as “nudge” and “fast versus slow thinking” into the popular lexicon – is that it acknowledges that people don’t always behave rationally (an assumption of classic economic models) and attempts to measure and predict their irrational behaviour.
People’s preferences often change over time in a way that is irrational – we are willing to pay more for a sofa set if the instalments start next January than if instalments start tomorrow. Similarly, health workers are more ambitious about the number of vaccinations they will accomplish next week than the ones they will do today. But procrastination in the face of distant targets or cramming to meet a deadline creates inefficiencies, inaccuracies, and reasons to cheat.
The policy-research team tested a system in which, in order to get that big bonus, the health workers used a custom-designed smartphone app to set their targets for days 1 and 2 of the vaccination campaign. The incentive structure expected workers to complete roughly 300 vaccinations over the campaign, and penalized them for moving tasks from day 1 to day 2. The workers recorded their goals, along with all attempted vaccinations, in the app. Information about actual vaccinations was aggregated in a central server and sent to the dashboard, which visualized the data for policymakers’ use, identifying whether or not the targets were being met. If a worker met the targets she set in advance for the first two days of the campaign, she got her bonus.
A key advantage is this: in making her bid, then recording her actual activity, the health worker let the app know how much she was willing to sacrifice to procrastinate. This allowed the team to tailor incentives individually to her, then apply those incentives in the second round of the campaign. The system gave the worker a degree of control over her activity (not to mention the chance for a bonus), and increased and evened out her effort in a way specific to her alone. All the while, it gave her manager the ability to track her and her co-workers’ progress toward vaccination goals in real time. If it worked, the app would be an exciting new tool for policymakers seeking to motivate better performance for workers in any governmental sector – one that automatically leverages the differences between workers for efficiency gains, rather than assuming all workers are the same, as traditional incentives structures do.
All of this, according to the team’s calculations, would cost around the same as existing systems, once implemented at scale. They piloted the system among 505 workers, divided into different treatments and control groups in a randomized trial.
This experiment yielded several notable results:
Influenced by the team’s idea, the government scaled up the technology under the banner of Punjab Information Technology Board (PITB) and implemented the smartphone application in the whole of Punjab. This was a very welcome move, as the PITB has scaled up other monitoring innovations presented by team members in the past – to good effect.
The BCURE pilot project had completed its goal of building information infrastructure within the Lahore health ministry, but the team is continuing its research under separate funding. And although the number of polio cases has dropped again, the disease remains endemic in Pakistan and Afghanistan – the last two remaining pockets in the world.
High-resolution data collected to incentivize workers may be applicable to another purpose. “In some regards, health officials really are operating in an information vacuum,” Callen explains. “They know which cities have had drives and which cities haven’t had drives, but they don’t know which neighbourhoods within the city have had drives. They know numbers of vaccinations, but they don’t have geospatial information.” In several upcoming vaccination drives, researchers will use data to make a map of a given area, understand why certain pockets are unvaccinated, and formulate a pay gradient that will incentivize workers to fill in those gaps. For example, the city-level map below draws the standard distance circle for each worker, showing that vaccinators tend to work within a small area and leave whole neighbourhoods unvisited.
The bottom map, showing a closer view of a specific neighbourhood, reveals a pattern of unvisited city blocks.
Callen and his team intend to use the maps, not only to encourage better coverage, but also to test how policymakers use research evidence. As part of the Assessment activity of the BCURE programme, Callen and EPoD co-director Asim Khwaja have been experimenting with presenting civil servants at Pakistan’s National School of Public Policy with research evidence and testing how they update their opinions. Laboratory exercises can only tell them so much, however. “What we want to do is figure out, how does providing policymakers with real information change how they take real action?” says Callen. “Imagine going to the guy in the health ministry whose job it is to guarantee vaccination, giving him a map and saying, Here’s where vaccinations have happened hundreds of times, here’s where they haven’t happened at all – now you redeploy your resources accordingly. Then measuring the outcome.”
Such a test could reveal the extent to which policymakers use research evidence in the real world, and suggest ways to increase such use.
"New policies and new information can be intelligently targeted at civil servants who are identified by personality tests as more intrinsically motivated. Such policies might provide useful complements to more traditional strategies - such as increasing worker monitoring or performance incentives - and provide the basis for additional, cost-effective policies for better public health."
Projects like the one conducted in Lahore can begin to fill in a problematic gap in development – the one between big datasets and their usability. The promise of big data, once freed, is undeniable: the world is seeing a remarkable revolution not only in the amount of data available, but also in the ease with which it can be rapidly and cheaply collected. Yet have sensibly argued that big data can only weigh in on big questions when accompanied by “small data,” usually in the form of targeted surveys, to add nuance.
By design, the app created for this pilot project passively collects data for the express purpose of being useful for policy. If such use of targeted use of data becomes widespread, it can provide another bridge between big and small data.
This pilot project – along with several others under the BCURE umbrella – have shown that gathering small data in developing countries can be done easily on the ground in real time, and the result can be improved implementation of programmes. This not only opens doors to more effectively identify problems: it also increases, perhaps substantially, the number of options available for dealing with them. The passive analysis of data is vital, but there are now new opportunities for the active design of systems and incentives based on the possibilities opened by new data technologies. Increasing the flow of data can not only add to what we know about the world, it can give us more options to improve it. ■
Text by V. McIntyre. Design by Angela Ambroz.