Update 07/12/16: The post below has been updated to more accurately represent some studies
This is one of our charity profiles, where we present our shallow, preliminary research on a potential, promising charity idea. We believe that this idea could be a potential contender for a GiveWell top charity, if further research confirmed the idea and if someone started the charity, executed it well, and resolved some of our outstanding questions and reservations.
Basic Idea Send SMS reminders for scheduled immunizations to improve childhood immunization coverage (IC) in low and middle-income countries (LMIC). Summary Cost-effectiveness: High -- Rough back-of-the-envelope cost-effectiveness estimates from ourselves and others suggests a comparable cost per DALY averted to top GiveWell charities . This estimate is heavily dependent on uncertain inputs for participant acquisition costs, IC increases attributed to SMS reminders and benefits of marginal immunizations. Strength of Evidence: High -- There is a considerable amount of studies, including systematic reviews, which suggest SMS reminders can increase IC across a wide variety of contexts. Counterfactual Scalability: Medium -- There appears to be no significant funders or organizations in this specific area, so it seems a new charity would initially operate in a relatively uncrowded environment. However, given the increasing popularity of mobile health (mHealth), the field may become more crowded. Ease of Testing: High -- It would be relatively easy to do an RCT on this specific charity idea because the intervention is low cost, has quick feedback loops, and individuals can be the unit of randomization. Flexibility: High -- It appears that the SMS infrastructure could be relatively easily used to promote other valuable behavior changes. Logistical Possibility: High -- Some of the main challenges to this specific charity idea seem to be determining optimal enrollment strategies, preferred locations and specific message details and content. These challenges seem less difficult than those associated with other specific charity ideas. Why We Think This Could be an Effective Opportunity SMS immunization reminders may be easier logistically, less crowded, more flexible, more testable, more scalable, as well as being relatively evidence-based and perhaps more cost-effective than the other specific charity ideas we’re considering. Immunizations have a strong evidence base for cost-effectively improving health (see GiveWell’s page on immunizations and DCP3) and a major reason for people with partial immunizations or lacking immunizations is a problem of demand (Rainey, et. al., 2009, Megiddo, et. al., 2014; see also The Wire Magazine and Live Mint Magazine) and a lack of knowledge in particular (Nath, Kaur, & Tripathi, 2015). This creates an opportunity to increase IC by increasing demand, leveraging the existing supply of vaccines. It’s possible demand can be increased cost-effectively via SMS reminders to comply with a specific immunization schedule. In order to implement this specific idea, a charity may complete some combination of the following: recruiting parents/guardians, recording age of child(ren), generating appropriate vaccination schedules, and sending SMS reminders at appropriate times. Our review of the literature indicated a wide variety of research that generally suggested SMS reminders were able to improve health outcomes. A systematic review found that 13 out of 14 studies showed a positive effect of SMS on some health-related behavior, such as tobacco cessation or diabetes self-management, though there were some methodological issues identified that undermined this result somewhat (Fjedsoe, Marshall, & Miller, 2009). A more recent review found that 5 out of 7 studies found an impact of mHealth SMS programs on health education outcomes (Hurt, et. al., 2016). Lastly, a Cochrane review of four randomised controlled trials involving 3547 participants also found positive increases in attendance at healthcare appointments (Car, et. al., 2012). However, the vast majority of studies cited in these mentioned reviews are from higher income areas than where we would like to implement this specific charity idea. When looking at literature directly connected to SMS reminders to improve IC in LMIC, we found five studies with a positive effect ranging from an increase of 8.7 to 17.5% percentage points in IC (Bangure, et. al., 2015; Haji, et. al., 2016; Uddin, et. al., 2016; Schlumberger, et. al., 2015; Eze & Adeleye, 2015; Brown, et. al., 2015) and two studies finding no effect (Gibson, 2015; Domek, et. al., 2016). We also found five upcoming studies that closely relate to the effect of SMS immunization reminders: JPAL (location: India, timeline: 2015-2017), JPAL (Mozambique, 2014), WHO (Pakistan, 2013), John Hopkins (Kenya, 2013-2014), and Bandim Health Project (Guniea-Basseau, 2016). We contacted an author of all these studies, asking if they had any shareable results. One author confirmed they did, and we have seen the results of the study, but we will wait until publication when we can review the full analysis and methodology before commenting further. We eagerly await the final results from these and other possible additional studies. Lastly, we examined research that fit only some of our criteria. Our preliminary conclusion is this evidence aligns with the view that SMS immunization reminders are a promising idea. Some evidence suggested that IC could be improved through a combination of SMS and conditional cash transfers (CCTs) (Wakhada, et. al., 2013) and through reminder systems other than SMS (Berhane & Pickering, 1993; Busso, Cristia, & Humpage, 2015) Two of the previously mentioned seven main studies report costs involved with the intervention. Haji, et. al. (2016) reported in Kenya total messaging cost of $0.27 per child for a +7%-point increase in IC of second pentavalent vaccine dose at 10 weeks of age and +13%-point increase in IC of third pentavalent vaccine dose at 14 weeks of age). Eze, et. al. (2015) (table 3) reports in Nigeria a cost of $0.15 USD per additional immunization of a child (original figure in Naira). Busso, et. al. (2015) (table 8) did a related intervention where they gave community health workers in rural Guatemala an updated list of children who needed vaccinations. They estimate that the cost per child who completed their full vaccination schedule due to the intervention is $7.53. We have not thoroughly vetted any of these numbers and hold some reservations about them. SMS immunization reminder costs can be split into initial user acquisition costs and marginal reminder costs. We probably would send three texts per immunization per user, though we could revise this based on pilot testing. A brief analysis of five different automated SMS providers suggests a cost of $0.001 to $0.06 per text, but the cost could vary depending on our method for acquiring users, country choice, and the desired amount of texts to be sent. This suggests marginal SMS costs of $0.003 to $0.18 per immunization reminder. Predicting 4% of our users would receive a counterfactual immunization means the marginal SMS costs per additional immunization are $0.075 to $4.50. The initial costs for acquiring a user are much less clear to us and we would hope to get a better sense of this during this specific charity idea’s piloting phase. SMS also seems adaptable to a variety of interventions. We found some evidence suggesting that SMS reminders worked in LMIC for increasing Vitamin A supplementation coverage (Thiaw, et. al., 2013), improving breastfeeding practices (Lee, et. al., 2016), and increasing utilisation of antenatal care (Watterson, Walsh, & Madeka, 2015). Note that this list isn’t exhaustive and we expect that there are other valuable behaviour changes that SMS’s could cause. Thus, SMS-related infrastructure could potentially be used for other health-related interventions, though we currently think they’re less cost-effective and have a lower strength of evidence than focusing on immunization reminders. Lastly, assuming we could recruit participants relatively easily and measure their immunization status, we could relatively easily randomize participants to determine if our system increases IC. The low cost per intervention and quick feedback loops makes a pilot study especially promising. A Possible Implementation Plan Broadly, we’d want to follow these steps for implementing this specific charity idea: Step 1: Determine areas where immunization supply is substantially higher than demand and where SMS immunization reminders seem likely to increase IC. Step 2: Investigate how to acquire reminder system users cost-effectively, experimenting with a few different approaches. We may experiment with cash incentives, partnering with advertising firms or peer-to-peer acquisition strategies. PATH’s mHealth Mobile Messaging Toolkit could be a helpful guide for rolling out some of these approaches. Step 3: Determine optimal number, timing, and content of messages. Step 4: Seek results relating to SMS immunization reminders and update accordingly. Step 5: Decide on a scaling process if it’s thought to be worth doing. Step 6: Partner with academics to run a RCT on the intervention. Step 7: Evaluate results of RCT and act accordingly. Who is Already Working in this Area? Multiple experts we spoke to agreed that mHealth suffers from “pilot-itis”, where most charities run a small pilot but never scale. None of the experts we talked to seemed to know why this is the case, though one person suggested it was because the government was not involved. As far as we could find, there are four organizations working in India that mostly provide SMS reminders, Vaccidate, eSwasthyaSeva, Immunize India, UNICEF, and mMitra. Vaccidate and eSwasthyaSeva seem small, internet-based, and focused on more wealthy Indian residents. Immunize India is larger (with over 700,000 users) but a large number of its signups are from the Indian middle class who are less in need. UNICEF is just running a pilot, which could very well suffer the same problems as all the other mHealth charities. mMitra is very small, only reaching about 42,000 people. When we talked to middle class locals in India, none of them reported receiving any sort of health reminder over SMS or any other medium. However, our impression is mHealth is becoming increasingly popular, so this idea may soon be expanded upon. For example, UNICEF is working with the Bangladeshi government to start a program to send reminders to do post-natal visits with a charity health worker (CHW) for both the CHW and the mother (mHealth Compendium, 2015, p79). Reservations To what extent do SMS reminders increase IC, after delays are taken into account? Some of the main studies report differences in IC only a few weeks after scheduled immunization date. For instance, Haji, et. al. (2016) and Bangure, et. al., (2015) report a 13% and 16% point increase in DPT3 coverage a few weeks after it being scheduled to be delivered at 14 weeks after birth. However, two systematic reviews identified a median delay of 6.2 and 6.3 weeks for DTP3 immunization in LMIC (Akmatov & Mikolajczyk, 2012; Clark & Sanderson, 2009). Studies that report differences in DTP3 IC only a few weeks after its scheduled delivery may give a misleading view of longer term IC increases attributable to the intervention because some children will receive DTP3 after the study finishes endline data collection. As a result, the counterfactual IC increases, say six months after scheduled immunization date, is likely less than the reported IC increases for some of these main studies but we aren’t sure to what extent. If the increase in IC attributable to SMS reminders six months after the scheduled immunization date was substantially less than the reported difference a few weeks or months after the scheduled immunization date this would significantly affect how promising we feel this specific charity idea is. Should we put less weight on these main studies? The ease of completing small scale studies on this intervention may make it particularly susceptible to publication bias and it’s worth noting that some of the main studies don’t appear to have been registered. Also, the lack of pre-analysis plan and independent re-analysis of the data makes it quite possible that inappropriate data analysis practices causes results to not replicate. What’s more, as the main studies are very recent, no reviews have commented on their quality or risk of bias and our own interpretation could miss some possible flaws. Will there be any limitations to sending texts? Potentially there are sizable limitations to sending reminder SMS’s at scale. Twilio, a cloud communications service that automates SMS sending mentions that, in India, the following limitations for sending automatic SMS:
People may be on a “do not call” list which makes them impossible to SMS unless they enable your texts to come through.
People might not be able to respond with SMS, though this may depend on the specific service.
SMSs can only be delivered between 9am and 9pm.
High rates of spam texts in India could also be a concern.
Remaining Questions
Will people be able to understand and act on our SMS’s?
How much will it cost to acquire a user and will those costs change with scale?
How difficult is it to expand this specific charity idea into another region or country?
What will it cost per counterfactual immunization received, once we acquire a user?
Is it cost effective to change our messaging or stop messaging those who miss vaccinations, despite our reminders?
How frequently do people change cell phone numbers? Will this pose a barrier to retention?
How will the forthcoming results of ongoing studies update our views?
Should we examine the evidence base more closely in light of potential problems with publication bias and problems with delay in immunizations?
Will other programs counterfactually serve populations we are interested in if we don’t act?
In the near future, will overall IC rise high enough that we quickly run out of opportunities to achieve impact through this specific charity idea?