 About Charity Science Health:
Charity Science Health was launched with the ambition of being the first direct poverty charity started by EAs to become GiveWell-recommended. Eight months into the project, we have received a GiveWell incubation grant and are expanding our team to increase our capacity. We are starting an internship program to help others gain experience with charity startups. This is intended to help them with starting one of their own. We have had multiple employees move on to found EA charities, and we intend to continue the trend. Impact The main sources of impact you will have from getting into this program will be:
Job Description Job responsibilities will be tailored according to the strengths of the candidate, their areas of potential growth, and the needs of Charity Science Health. This can range from doing cost-effectiveness analyses, designing a measurement and evaluation program, writing and editing charity entrepreneurship lessons, and anything in between. Requirements:
Benefits
Details
To apply, please send the following to joey@charityscience.com
1 Comment
This is a job position for a project the Charity Science / Charity Entrepreneurship team advises but which is not part of our organization. High impact charities are just waiting to be founded. This presents us with an opportunity to make a remarkable impact on the lives of others. Put another way, not pursuing these opportunities may come at a sizeable cost to the people who remain unhelped. Charity Science, an organization dedicated to promoting the causes that can best improve the world, has rigorously examined neglected cause areas that have the potential to join GiveWell’s top charities. They were evaluated in terms of cost-effectiveness, scalability, strength of evidence, ease of testing, flexibility, and logistical possibility. Charity Science concluded there are five promising charity ideas that are just waiting for an effective team to make possible. The Charity Science team has already begun to work on one of these charities. With the guidance and advice of Charity Science, I (Brendan Eappen) am assembling a team to found one of the other top charity ideas: Once a co-founder is identified, we would work together to identify which charity we could best implement. You can read more about each charity idea via the links above. Experts in the field, including GiveWell, 80,000 Hours and Charity Science all think this is an effective way to make a difference in the world. You can see a detailed model of the estimated impact here.  What we have to offer
What we need
What we want
Applying Your application should include two files: your resume and a document containing the following. Please download and complete the template linked here.
Start dates are flexible, but ideally we would decide whether or not to found the charity by early April. Please send questions or application materials to Brendan Eappen Eappen@alumni.harvard.edu by March 26, 2017 (extended). Applications will be reviewed on a rolling basis beginning on March 4, 2017. Want to learn more about who I am? Let’s chat! To give you a sense of what I’ve done, you can view my resume. If the right team is formed, I believe my investments in this charity startup will have a higher marginal impact and learning value than they would in an established organization. I’m excited to pursue this opportunity and look forward to reading your application! Brief history:
Charity Entrepreneurship was a project started by the Charity Science team with the intention of founding a highly effective charity (e.g., one that was competitive with the cost-effectiveness of current Givewell recommended charities). After six months of research into different priority program areas we found five ideas we think are worth starting a charity around. We launched Charity Science Health to start one of them. Charity Entrepreneurship in 2017: We consider Charity Entrepreneurship a success, but there were two large areas we could see further working being done on. The first area is the creation of other high impact charities based off our other four top ideas: Given the strength of the remaining ideas, it seems worthwhile to create additional charities following our work. Right now our team does not have the capacity to start more than a single idea at a time, but we have considered and tried a few ideas in this space to hire or inspire others, and excitingly it now looks as though a second charity might be founded based on CE research. An EA we have spoken with is going to create one of the top charity ideas with our support and advice. We think this is an incredibly high impact career and will soon post a job ad to join this EA to our CE website. The second area is a broadening of our research to include other cause areas: Our research was extremely focused on evidence-based poverty interventions. However, there are many other cause areas we think could be high impact and would like to conduct similar research in these other areas, such as expanding GiveWell’s shallow reviews, but focused on what charities could be started rather than funded. Our first step on this project is to do upfront crucial considerations research before being able to dive into comparing a more diverse range of cause areas. How to help out: The speed of Charity Entrepreneurship has slowed down from our initial six month burst because a majority of our team has moved to working on other projects (primarily Charity Science Health). We want to be conscious of not spreading our team too thin, but we could imagine completing the crucial considerations research and then scaling a team to do cause area research. Currently we are not hiring for this project, but we are looking to expand our pool of volunteers and unpaid interns from our other projects. Volunteering/interning is a great way to get to know our team and help us get a sense of your skills and abilities. When considering which charity idea would be the best one to create, we accounted for standard factors such as cost-effectiveness and the strength of the evidence that goes into the cost-effectiveness estimate. However, this analysis still left us rather uncertain, with a nagging feeling that we could be somewhere between mildly inaccurate and terribly wrong. What if we picked an intervention that turned out not to be very cost-effective or we picked an intervention that was effective generally speaking but beyond our ability to implement correctly?
The fear that we could be wrong drew us to our favorite metric for evaluating ideas -- flexibility. This is “keep your options open” applied to charity. How easily could we shut down this project and move to another project if this project turned out to not be very good?
The Value of Flexibility
We’ve been wrong a lot over the years, dramatically changing our beliefs, values, and career paths several times as we update on new information. It’s unlikely that the first thing you stumble upon will be the highest impact thing to do, any open minded person would experience a fair degree of change. While we’d like to think things are more stable now, chances are good that we will change our beliefs again as time goes on and that this will require yet another change.
Lastly, even if we don’t change our minds, the world itself is constantly changing. New laws and regulations could make our previous plans untenable. New opportunities could open up that didn’t exist when we first started putting our plan into action. New competitors could arise implementing our intervention just as well or better, rendering our work obsolete.
The more likely it is that change will occur and the more important you think the change will be, the more value you should put on flexibility.
Flexibility Over Robustness
Flexibility is the ability to change if circumstances change. Robustness is the ability to withstand change. For example, if there’s a flood in a village, a flexible population might move to another village. A robust population would have built flood-protecting walls around the village so that it’s not harmed by the flood. If there’s a disaster that prevents farming corn, a flexible population might switch to farming beans, whereas a robust population would have already known to genetically engineer their corn ahead of time to be immune to the disaster.
We definitely see the merit of robustness and would like to be as robust as we can. However, we see robustness as requiring stronger predictive ability about how things may go wrong plus the ability to create effective mitigation strategies, whereas flexibility can be reactive without needing to anticipate. In our examples, the village would have already needed to know to build flood-protecting walls and genetically engineered corn, which may have been hard to know in advance and costly to implement. For instance, maybe they worry about a flood so they build walls, but the disaster that actually strikes is a drought. Since we believe that anticipating and mitigating problems would be nearly impossible given that we face so many unknown unknowns, we think flexibility beats out robustness, at least for us.
Flexibility and The Lean Non-Profit
Flexibility is discussed a lot as a value for start-up for-profit companies. This philosophy, emphasized by Eric Ries’s book The Lean Startup, strongly encourages building just the minimum viable product (MVP), testing for user demand before scaling, and pivoting if your initial idea doesn’t work. The start-up world emphasizes that you are frequently wrong about the world (called “product-market fit”) and that you will fail many times before you succeed. Thus, the most successful start-up teams are the ones that are the most flexible.
We think this philosophy should equally apply to start-up non-profits too, though we seek “skills - capacity for impact” fit rather than “product-market fit”. While this idea has not taken off as much in the non-profit world, Luke Muelhauser wrote in 2013 about how the Machine Intelligence Research Institute was operating as a lean non-profit that emphasized MVPs, A-B testing, and pivoting. Since then, there have been two articles in the Stanford Social Innovation Review showing one particular experience implementing lean methodology in the non-profit world and another advocating for the benefits of pivoting and the lean methodology. Charity Science has also followed this model since 2013.
The Many Kinds of Flexibility
How can we increase flexibility? We found that flexibility can be increased in a wide variety of different ways. Here are some of them:
Intervention flexibility
A flexible intervention can be adjusted at a moment’s notice and doesn’t require that much upfront cost before seeing results.
Organizational flexibility
A flexible organization is able to quickly act differently based on new evidence.
Staff flexibility
While we already have flexible senior staff and management that can pivot from one project to another, we’ll also need to hire additional staff and ideally these people could fit into our flexible mindset. The following are a few ideas to increase our chances of coming into contact and hiring such people.
Funding flexibility
A lot of funding in the charity world is tied to particular ideas. But this is bad for the charity that wants to be able to change quickly, as changes will result in disruption of funds, which disincentivizes flexibility.
Other Areas
We think it is important to be flexible in other areas as well, such as with connections (e.g., people on the board, contacts in public and private sectors, relationships with academics) and with personal situations (e.g., location independence, being independently wealthy).
Conclusion
Overall, improving organizational flexibility by iterating quickly, building a culture that improves pivoting and open-mindedness, by putting a lot of work into hiring and skill training, and by building a broad, diversified funding base looks to be a relatively easy and high-value way of improving an organization’s potential for making an impact. Do you want to play a pivotal role in founding one of the most effective charities in the world? Join us and you will work on some of the most challenging problems, making decisions that will potentially influence hundreds of thousands of lives. This is a job where your day to day work can make a huge impact on the world.
The Charity Science team has spent hundreds of hours researching the most important poverty charities. Our top charity, SMS vaccine reminders, has the potential to increase immunization rates and decrease death and disease around the world. And this is just the beginning -- if we execute well, we could be in a good position to start other promising mobile health charities. This is a competitive job but we have non-traditional standards so we encourage a wide range of people to apply. We do not expect our applicants to fulfill all our needs and wants What we offer:
What we need:
What we want:
This position would start at a fairly senior level and the employee would be expected to be able to make hard calls and come up with intelligent ways to progress the project soon after joining the team and getting up to speed on the pre-existing research. Your application should include a (1) cover letter explaining what you bring to the team, your (2) resume and a (3) clear list of which of our needs and wants you meet. We do not expect our applicants to fulfill all our needs and wants, but we’d like to know how you fit before the interview. We’re flexible on start dates and all jobs start with a one month trial period to assess fit prior to extending a permanent offer. Send your application or questions to joey@charityscience.com by June 30th We have completed our second phase of research which was to narrow a list of about thirty charity ideas to a more concise list of five top prospects worth further exploration. Below is a chart comparing these five possibilities. The rankings are relative to each other, not absolute and based on a primary and time limited review of the data on these causes.  We have written a detailed summary for each of these options elaborating on their strengths and weaknesses:
It is difficult to compare charities at this level, especially when the metrics we use to measure their respective impacts are so different (e.g. research vs. direct benefit). However, we still feel as though we have a front runner among these possibilities. SMS reminders to encourage vaccinations is currently our top pick for an intervention to pilot. Though we are tentatively feel that our front-runner (SMS reminders) is the most promising intervention to proceed with, we have not closed the book on the other four options. We recognize that the difference in impact between the best and second best options could be very large, and thus feel that there can be “no stone unturned” with regards to making the right choice. To help make this the case, we are offering a one-time $500 prize to anyone who can significantly sway our decision by providing material that may challenge our conclusion thus far. How significant? We challenge you to change our top option by either weakening the case for SMS reminders or strengthening the case for another option. Why are we doing this? We need to make sure our decision is as airtight as possible before moving forward. Consider this an opportunity not only to get you (or a charity of your choice) $500 richer, but to make a hugely impactful contribution in the fight for global health. Email joey@charityscience.com for more information. This is one of our charity profiles, where we present our shallow, preliminary research on a potential, promising charity idea. We believe that this idea could be a potential contender for a GiveWell top charity, if further research confirmed the idea and if someone started the charity, executed it well, and resolved some of our outstanding questions and reservations.
Basic Idea Lobbying lower and middle-income country (LMIC) governments to legislate mandatory iron and folic acid fortification of a staple food (such as flour or salt). Summary Cost-effectiveness: High -- Iron fortification has been estimated to be $70 per DALY and appears to have a positive impact on a relatively large number of different metrics. Folic acid fortification is less cost-effective, with an estimate of $11K per life saved, but can be combined with iron fortification in the same efforts so there is little actual increase in cost. Strength of Evidence: Medium -- The case for iron and folic acid fortification are covered by several studies, including some systematic reviews. However, like with tobacco taxation, significant government lobbying would be needed and the effectiveness of such campaigns are less clear. Counterfactual Scalability: Medium -- There appear to be over a dozen countries and Indian states with large populations that aren’t currently being lobbied for fortification. However, there are a fair number of organizations focusing on this already, and it may be a matter of time before they focus on these areas. Ease of Testing: Low -- Performing an impact evaluation on a lobbying campaign is exceedingly difficult due to high interference from other campaigns, the all-or-nothing nature of lobbying, and an inability to effectively randomize. Flexibility: Medium -- While it appears we could move lobbying efforts to other areas as we gain more generalized skills and contacts, we would likely not build much infrastructure that is useful for areas outside lobbying. Logistical Possibility: Medium -- Lobbying poses significant logistical challenges and draws on skills we do not yet possess. Why We Think This Could be an Effective Opportunity Iron Iron deficiency is perhaps the most common micronutrient deficiency in the world (Baltussen, Knai, & Sharan, 2004). Fortifying food with iron could have immense benefits. A Copenhagen Consensus report notes that iron fortification is one of the top cost-effective interventions (Hoddinott, Rosegrant, & Torero, 2012, p33). The DCP-2 estimates that iron fortification results in $70 per DALY and costs $0.09-$0.12 per year per person treated (DCP2, p560)[1], though we haven’t yet vetted any of these estimates. Anemia is frequently linked to iron deficiency (and also to other factors). Increasing iron intake can decrease a significant portion of anemia cases (Guo, et. al., 2014; Gera, et. al., 2012; Zimmermann, et. al., 2004; Andersson, et. al., 2008; Sari, et. al., 2001; Lakshmy, et. al., 2007) which reduces fatigue, irritability, dizziness, shortness of breath, and other conditions. Increased iron is also possibly connected with decreased premature births (Pena-Rosas, et. al., 2012), increased income (Thomas, et. al., 2006; Niemesh, 2012), and decreased behavioral problems of children (Zhou, et. al., 2006). Iron has also been linked to decreased depression rates (Beard, et. al., 2008; Ballin, et. al., 1992; Rahn, et. al., 2008; Luca, et. al., 2008), though this effect is less well established than other conditions, as it’s mostly observational data. Lastly, it has also been linked to increased cognitive abilities of children (Baumgartner, et. al., 2012; Low, et. al., 2013), though this has been undermined by a more recent meta-analysis from Guo, Liu, and Qian (2015) and we’ve noted before that we have concerns with the importance of improving IQ. While many of these studies are for iron supplementation instead of iron fortification, the effects should be similar, since fortification also increases iron intake but at a lower dose (Hoa, et. al., 2005; Le, et. al., 2006). Iron fortification seems to us logistically more easy to do, because you can add it to a staple food people already get and eat, rather than create new distribution plans and behavior changes to get people to have and consume an iron supplement. This could be why some studies have shown iron fortification is also likely more cost-effective than iron supplementation (Baltussen, Knai, & Sharan, 2004; Le, et. al., 2006; Ramsay & Charles, 2015). Folic Acid Folic acid is a form of folate, also known as vitamin B9. Maternal folic acid deficiency caused over 300,000 annual neural tube defects (NTDs) in 2012 and 2013 (CDC, 2012; Pachón, et. al., 2013), though now the number appears to be below 280,000 annually (WHO, 2015). NTDs frequently result in death or severe lifetime mental and/or physical disability for newborns (Ibid.). Folic acid deficiency can be reduced through folic acid supplementation (De-Regil, Peña-Rosas, Fernández-Gaxiola, & Rayco-Solon, 2015). Folic acid fortification is less cost-effective than iron fortification, estimated at $1200 per NTD averted and $11K per life saved (Llanos, et. al., 2007). However, folic acid fortification can be done alongside iron fortification (Copenhagen Consensus, 2008; WHO, 2012) and one could lobby the government to do both forms of fortification with the same lobbying effort. This means additional folic acid fortification would likely have some additional benefit with very little additional cost, resulting in what would likely be a marginally better cost-effectiveness than iron fortification alone. A Possible Implementation Plan If we were to explore iron and folic acid fortification, we’d aim to hire a local expert with expertise in fortification and local politics. We’d aim to find a country that has a large population, a relatively centralized production of the staple food, doesn’t have other charities already working in that area, and has a stable government without any armed conflicts. Once we had identified a country that fit these criteria, would would talk to experts to understand if this country was neglected for a particular reason. It’s important to keep in mind that lobbying the government to make fortification mandatory is likely the best way to implement fortification. The government is able to pass along the costs of fortification to the consumer, which is a negligible $0.00063 per kg of flour (0.16% of current retail price) (Food Fortification Initiative FAQ). Other models, such as paying companies for fortification, cannot take into account these economies of scale and would have to end up paying a lot of money and undercut any potential for political change. Who is Already Working in this Area? We found a few big organizations working on this, such as the Micronutrient Initiative, which works primarily on lobbying for iron and folic acid fortification. While GAIN works primarily in salt iodization, they do have some iron and folic acid fortification programs. UNICEF works on iron fortification of flour, among many other things. Food Fortification Initiative does a variety of food fortification programs, including iron and folic acid fortification. Project Healthy Children works to encourage the implementation of mandatory food fortification programs by governments. Helen Keller International also works on iron fortification, among many other programs. Smarter Futures works on food fortification, including iron and folic acid. They are a partner of GAIN. Our understanding from talking to experts is that it is very common for these organizations to partner together, so any additional work we did would likely be in partnership with some of these groups. Reservations Lobbying The evidence for lobbying succeeding in LMIC is far weaker than the evidence for the impact of successfully implemented fortification. It’s often difficult to credit responsibility for legislative change to particular players in advocacy campaigns because multiple campaigns operate simultaneously and it’s unclear whether the government would have changed anyway, absent lobbying efforts. While there is considerable evidence that government implementation of fortification is highly cost-effective, there is a dearth of analysis on the return of investment of actual lobbying campaigns. We find the difficulty of testing this project to be a barrier to attempting an implementation, given our desire for rigorous impact evaluation and prioritizing against other charity ideas, as well as this making it difficult to receive feedback to refine our lobbying approach. However, depending on policy, this specific charity idea may be able to be completed at the state level before attempting it on the national scale. We’re also not enthusiastic about would could be the zero-sum and all-or-nothing nature of lobbying. It would very likely be necessary to hire local lobbyists to reach the needed policy makers, which could be expensive. This also creates flexibility concerns, as the contacts we build would likely not be useful for a charity other than lobbying. Moreover, if lobbying requires a high degree of specialization, it may not even be possible to move easily between different lobbying campaigns. Access to Fortification We would guess it is plausible that the people in most need of iron fortification would be less likely to buy their food from a centralized source; instead, being more likely to grow it on their own or trade with other locals. This would prevent these people from being reached by a centralized food fortification effort. However, we have not seen any literature one way or another on this effect and we are unsure how it would affect the cost-effectiveness for iron even if the effect was true. Long-term, Large-scale Track Record While many countries have already implemented large-scale mandatory iron fortification programs targeting wheat flour, there are very few studies that seek to evaluate these programs. More troublingly, evaluations that have taken place have found that in half the countries where large-scale iron fortification has taken place, iron deficiency did not decline in children, though it did for women (Pachón, Spohrer, & Serdula, 2015). However, most of the countries studied did not fortify according to WHO standards, which can dramatically impact the effectiveness of the iron (Ibid., p12). Ultimately, we’re unsure about how heavily to weigh this consideration. There are also no studies we could find that seek to verify the long-term effects of iron fortification. This means it remains possible that the positive effects found so far could be fleeting over a longer time-horizon. For one example, Protzko (2015) found evidence of a fade-out effect in raising IQ, and GiveWell raises concerns about fadeout effects in their review of salt iodization (GiveWell, 2014). Side-effects of Fortification There has been some concern that iron supplementation might negatively affect malaria rates, which is why the WHO recommends fortification to be done in conjunction with improved anti-malarial practices. However meta-analyses drawing on over 20 RCTs (Shankar, 2000; Gera, 2002) found no significant effect on malaria. This would be especially true for fortification, which is at much lower levels than supplementation. Our overall conclusion is to avoid high malaria zones whenever possible, but not to put too much weight on this concern. Another potential “side effect” is masking Vitamin B12 deficiency. B12 deficiency is often detected by the presence of anemia (NIH Fact Sheet, 2016) and the deficiency can cause long-term neurological damage (Lachner, Steinle, & Regenold, 2012), which means that a decrease of anemia through iron fortification could delay diagnosis. Iron supplementation may also result in excess iron, which can slow physical development (Ianotti, Tielsch, Black, & Black, 2006; Paricha, Hayes, Kalumba, & Biggs, 2013). It’s not clear whether iron fortification would carry the same risk because only small amounts are added to the staple food. While folate supplementation prevents cancerous tumors, folate can also increase the growth rate of existing cancerous tissue (Smith, Kim, & Refsum, 2008). This makes it unclear whether folate increases or decreases cancer risk (see discussion in Kim, 2007; Mason, et. al., 2007; Kim, 2008; Luebeck, et. al., 2008; Hirsch, et. al., 2008). Any rollout of iron and folic acid fortification may need to carefully target specific populations to reduce the risk of negative side-effects. Differences between Fortification and Supplementation Lastly, it’s worth noting that we’re suggesting a fortification program, but in the process we rely on evidence of the effect of supplementation (though we also do use evidence of fortification). As we discussed in the section on side-effects, the dosage of fortification is meaningfully lower than the dosage involved with supplementation, which can involve changes in effects. We have not yet thought seriously about how much, if at all, this would undermine the case for fortification. Remaining Questions
Footnotes [1]: DCP has had flaws in the past which overestimated cost-effectiveness (GiveWell, 2011) which could make the true estimate worse. Additionally, the DCP estimate does not account for effects on income, depression, and irritability, which could make the true cost-effectiveness estimate better. This is one of our charity profiles, where we present our shallow, preliminary research on a potential, promising charity idea. We believe that this idea could be a potential contender for a GiveWell top charity, if further research confirmed the idea and if someone started the charity, executed it well, and resolved some of our outstanding questions and reservations.
Basic idea Incentivize positive behavior with conditional cash transfers (CCTs) in lower and middle-income countries (LMIC). Summary Cost-effectiveness: Low-High -- Cost-effectiveness will largely be determined by specific behaviour incentivised and the extent to which the incentive is successful. The added benefits of the transfer on income will also play a part in overall cost-effectiveness. Strength of Evidence: Medium -- A significant number of studies have explored a variety of potential behaviors that could be incentivized. However, the variation in CCTs -- in incentive delivery and amount, and which action is incentivized -- calls for a larger amount of research. Counterfactual Scalability: Medium -- There are no major funders or organizations in this specific area. However, because of the growing popularity of cash transfers, the field may become more crowded. Ease of Testing: High -- It is relatively easy to perform a randomized controlled trial for a CCT, though ease may vary according to intervention specifics. Flexibility: Medium -- It seems limited work would be required to shift a CCT program incentivizing one behavior to incentivizing a different behavior. A conditional cash transfer charity also seems like it would be able to update on new information at a faster rate and to a greater extent than charities in most other intervention areas. Logistical Possibility: Medium -- Some challenges associated with this specific charity idea are identifying an ideal targeting approach, a reliable and convenient cash transfer system, and a preferred monitoring method. Why We Think This Could be an Effective Opportunity CCTs entail giving someone a cash incentive for completing a certain activity. CCTs can incentivize behavior that improves wellbeing while also improving well-being by increasing wealth. GiveDirectly, a GiveWell top charity, has a well-evidenced, high-impact intervention through unconditional cash transfers (UCTs), where money is unconditionally given to some of the world’s poorest (see GiveWell’s review of GiveDirectly). A CCT seems like it may be able to achieve higher cost-effectiveness than UCTs by successfully incentivizing highly valuable health behaviors. However, the beneficial effects of the cash transfer may be much lower than in the UCT since the ability to target the poorest is compromised. There are numerous behaviors that could be incentivized and many different incentive sizes and delivery methods. Additionally, it is easy to randomly assign participants to different conditions, including a control condition. Thus, there’s much room to experiment with CCTs, which makes it a flexible, testable and possibly cost-effective intervention class. CCTs for Healthcare Utilization Akresh, et al. (2012) reported that CCTs to incentivize quarterly childhood growth monitoring resulted in significantly more clinic visits than an UCT control group. Rai (2014) reported that Kenya’s traditional birth attendant encouragement program increased the number of clinic births and Lim, et. al (2010) reported a similar result for a large scale Indian program. Lagarde, Haines, and Palmer’s (2009)’s Cochrane review also suggested CCTs could be used to increase health care utilization. However, we’re concerned about the quality of some clinics (Fraker, et. al., 2013, Chapter 3) which may provide minimal health services, though we think this concern could be somewhat mitigated by selecting clinics that score well on evaluations (see ICDS, 2015; Lee, Madhaven, & Bauhoff, 2016). CCTs for Healthcare Worker Performance There are some areas of standard healthcare worker performance that if incentivised correctly might increase health outcomes. For example, Banerjee, Duflo, and Glennerster (2008) reported as many as 35% of healthcare workers were absent on a given day, while Chaudhury, et. al. (2006) estimated a 43% Indian medical worker absenteeism rate. Also, workers that are present may be low skilled, with one survey suggesting 67% of Indian healthcare providers had no formal medical qualifications (Das, et. al., 2012). There are some concerns CCTs based on patient outcomes create perverse incentives (rejecting sick patients, gaming hospitalization admission dates, etc.) but we think there are reasonable ways this can be avoided. Some apparently successful trials involved performance pay among a fixed beneficiary population (Miller & Babiarz, 2013). Other trials have focused on incentivizing community health workers, who generally lack medical training, to refer patients to hospitals under certain conditions (Rai, 2014). Another interesting program adopted a mystery shopper method, paying based on observed quality of healthcare given to people pretending to be customers (Wilson, Morris, & Gilbert, 2012). However, systematic reviews have suggested the general state of evidence for “pay for performance” in LMIC to be relatively low (Witter, Fretheim, Kessy, & Lindahl, 2012; Grittner, 2013; Gopalan, Mutasa, Friedman, & Das, 2014; Das, Gopalan, & Chandramohan, 2016). CCTs for Increasing Immunizations Immunizations have a strong evidence base for cost-effectively improving health (see GiveWell’s page on immunizations and DCP3) and a major reason for people with partial immunizations or lacking immunizations is a problem of demand (Rainey, et. al., 2009, Megiddo, et. al., 2014; see also The Wire Magazine and Live Mint Magazine) and a lack of knowledge in particular (Nath, Kaur, & Tripathi, 2015). This creates an opportunity to increase IC by increasing demand, leveraging the existing supply of vaccines. While inadequately trained health workers could be addressed with CCTs for healthcare worker performance, immunizations could also be increased through CCTs providing a cash incentive to increase immunization demand (see Barham & Maluccio, 2009; Banerjee, Duflo, Glennester, & Kothari, 2010; Chandir, et. al., 2010; Sato, 2015). A Possible Implementation Plan We think that a good way of implementing CCTs would be to explore the range of CCT implementations further before making an initial commitment to a particular form. If we were to do this idea we lean towards piloting many different CCT approaches, varying transfer size, transfer mechanisms, enforcement mechanisms, participant recruiting strategies, and the behavior promoted. This seems relatively feasible given Haushofer & Shapiro (2013) were able to experiment with recipient gender (wife vs. husband), transfer timing (lump-sum transfer vs. monthly installments over 9 months), and transfer magnitude (USD 404 vs. USD 1,520) in one study. Who is Already Working in this Area? CCT programs are most frequently implemented by governments. Evans, Holtemeyer, and Kosec (2015) note that CCT programs are present in almost every Latin American country and in over a dozen African countries. For example, by 2011, CCTs had spread to 18 countries in Latin America, with as many as 129 million combined beneficiaries (Stampini & Tornarolli, 2012, p3). The biggest non-profit we know of implementing CCTs is New Incentives, which incentivizes prevention of mother to child transmission (PMTCT) of HIV by incentivizing birth in clinics, antiretroviral adherence, and HIV screenings for newborns. GiveWell concluded that “there appears to be strong independent evidence that CCTs and PMTCT are effective programs” and that “New Incentives' PMTCT CCT program could potentially be in the same range of cost-effectiveness as our top charities” (GiveWell review of New Incentives). VillageEnterprise provides CCTs for entrepreneurs, giving $150 to each three-person new business founded by someone in extreme poverty with no prior business experience. TrickleUp also seems to provide CCTs for entrepreneurs, though they have a more complex model. Reservations Logistics The logistics of running a CCT may be challenging. Lahariya, et. al. (2011) found that poor planning, lack of public awareness, and the risk of corruption were non-trivial challenges to implementing a CCT. Dr. Bhargava notes lack of public awareness, targeting errors, execution gaps, payment errors, inadequate dispute resolution, inadequate service quality, and unclear graduation processes as the “Seven Deadly Sins in Conditional Cash Transfer Programs”. Additionally, a more simple problem may just be availability of bank accounts to transfer cash, which may be a problem in India (The Wall Street Journal, 2012; Indian Express, 2015; TechVibes, 2015) and other countries. Warping Incentives We also think a badly done CCT could warp incentives negatively, incentivizing the wrong behavior and producing net negative results. For example, Singh, et. al. (2015) found that low remuneration for community health workers could undermine their intrinsic motivation and foster discontent. a CCT for healthcare visits could incentivize frivolous clinic visits solely so people could get into the program. Also, a CCT for healthcare performance could lead to hospitals denying treatment to patients who would be most likely to reduce patient care metrics. It could be difficult to craft and enforce an incentive structure to lessen the possibility of these unintended effects. We are also concerned about the possible tendency of cash incentives to erode intrinsic motivation for the action in question, which could have negative long-term effects. For example, This effect of incentives on intrinsic motivation has been found by multiple meta-analyses (Tang & Hall, 2006; Marianne & Marteau, 2013; Cerasoli, Nicklin, & Ford, 2014), but it’s unclear if it would generalize to CCTs for health behaviours in LMIC. We would like to survey the literature more and even see long-term studies of the effects of CCTs to assess this concern’s validity. Cost-effectiveness Another potential worry is CCTs may take too much of a middle-ground approach and end up not be as cost-effective as either direct work on the behavior incentivized or an unconditional cash transfer. For example, using incentives to improve immunization rates may be more expensive than other methods to increase immunization rates, like SMS reminders. The cost of the transfer as well as the compliance monitoring to ensure the transfer is merited may get expensive, relative to just administering the program. More precise cost-effectiveness estimates would have to be made on particular CCT programs in order to assess this effect. There may also be degraded impact from the transfer itself, since it is possible (though not backed up one way or another by any studies we know of) that those with access to a particular CCT intervention may be more wealthy than those without access to that intervention, which would decrease the impact of the wealth transfer by a significant amount[1]. We’re not sure how much the relative cost-effectiveness of the incentivized behavior would be relative to the wealth transfer, though we expect generally that uptake in the behavior to be more impactful. Remaining Questions
Endnotes [1]: We roughly approximate the impact of a wealth transfer by how much it increases the log (base 2) of the income of the person receiving the wealth, on the principle that wealth has a log-relationship with happiness (Stevenson & Wolfers, 2008). However, we admit that there is a weakness in this approximation since it is not clear whether this doubling effect applies to very low levels of income. However, if the doubling effect is true and if an extremely poor person targeted by a UCT earns $1/day but a poor person targeted by a CCT earns $4/day, a UCT transferring $100 would have ~3x more impact than the CCT. This is one of our charity profiles, where we present our shallow, preliminary research on a potential, promising charity idea. We believe that this idea could be a potential contender for a GiveWell top charity, if further research confirmed the idea and if someone started the charity, executed it well, and resolved some of our outstanding questions and reservations.
Basic idea Conduct high-quality research establishing which global poverty interventions are the most promising. Summary Cost-effectiveness: Low-High -- While it may be possible to estimate the historical cost-effectiveness of global poverty research, we are not aware of any estimates and we would be concerned about whether they would generalize to our specific case. It’s possible for research to be highly cost-effective but there are far too many unknowns. Strength of Evidence: Low-Medium -- The case for the impact of global poverty research is intuitively strong. However, the causal chain is long enough that it ought to be established by empirical evidence before it’s thought to have a strong strength of evidence. Counterfactual Scalability: Low-High -- There are a considerable number of competent organizations conducting global poverty research. It may be better to support them than start another organization in this space. However, we feel there’s possibly a large quantity of valuable high-quality studies that can be done. Scaling this research organization could be difficult and there’s some risk that no one acts upon research we complete. Also, scale matters less for this kind of organization as it could be possible to have a very large impact with just a few number of studies. Ease of Testing: Low -- It would be very difficult to measure the impact of research, let alone quantify impact on endline metrics. Flexibility: Medium-High -- A research organization can more easily change its research focus than a direct charity can change its programs. However, a research program would still be limited to doing research. Logistical Possibility: Medium -- We feel that producing high-quality research may be more challenging than implementing a specific charity idea. We also may lack the formal academic credentials required to implement a research organization. Additionally, there may be a difficult lobbying component to ensuring others take our research into account. Why We Think This Could be an Effective Opportunity While some scholars have attempted to empirically quantify the impact of developed world medical research (e.g., HERG, OHE, & RAND, 2008), we are not aware of anyone who has quantified the impact of global poverty research. However, we see the case for such research as intuitively promising. GiveWell writes in their list of charities they’d like to see that they would be excited to see “[c]harities that collect or generate information and data relevant to [their] recommendations.” For example, GiveWell sees more information on large-scale bednet distributions implemented by groups other than the Against Malaria Foundation (AMF), salt iodization programs, and tetanus immunization programs as valuable, but seem not to know of organizations collecting this data. They also write about a desire for more randomized control trials (RCTs) pertaining to their priority programs that could potentially be more cost-effective than their current top charities. Similarly, our initial intervention research identified some potentially promising ideas that we ultimately eliminated because they had a poor evidence base. This problem could be solved with more high-quality global poverty research. The principal benefit of performing global poverty research is likely the chance of finding a charity more cost-effective than GiveWell’s current top-rated charity, AMF. As of the end of 2015, AMF has an estimated cost-effectiveness of $2838 per life saved (see GiveWell’s analysis) and receives $22.8M from Good Ventures (see GiveWell’s blog post), which results in an estimated ~8000 lives saved. However, if our research could identify an underfunded intervention with a cost-effectiveness of $2000 per life saved, $22.8M could save ~11,000 lives. Our research could then be taken to have contributed toward saving an additional 3000 lives. If we assume a typical “gold standard” RCT costs $500K[1] and that two such RCTs would be needed to establish an existing pretty close charity as more cost-effective than AMF, and if we assume that we hit on the right charity 20% of the time, it would have taken $5M to save those 3000 lives, which works out to an estimated ~$1500 per life saved. Of course, the assumption of a 20% success rate of discovering charities with a cost-effectiveness of $2000 per life saved or lower is a complete guess and could be wildly inaccurate. Moreover, there still would need to be a lot of additional work and due diligence on the part of GiveWell, or some similar funder, and our research would only be a part of that. A Possible Implementation Plan A research organization could interview those founding (e.g., Evidence Action, us) or reviewing (e.g., GiveWell, the Global Innovation Fund) effective charities and compile a list of studies that would most benefit these reviewers/founders. This list could then be prioritized by other research organizations so that the highest-priority studies are done first. Replications It’s well known that replication research is a neglected but very important area of research. We’d love to see replications of studies that significantly inform views of priority programs. For example, replications of studies analyzing cost-effectiveness of deworming interventions could verify its impact. We think external validity is a concern when attempting to scale an intervention that has impact evidence in one particular area to significantly different areas. For this reason, we’d also be interested in experimental replications of studies in different contexts. Research on highly promising interventions While researching particular interventions, we found a lot of unanswered questions that would be amenable to future research. One such example is conditional cash transfers (CCTs). The large variation in CCTs -- in incentive delivery and amount, and which action is incentivized -- create an immense opportunity for research. Data on intervention costs Commonly global poverty research aims to correlate an intervention to a positive outcome and it seems usually there’s less investment in researching the cost-effectiveness of that program. When reviewing literature for interventions, we found that research elaborating on intervention costs to be quite useful in advancing our understanding of the intervention’s relative potential. While GiveWell has provided some considerations against more investment in cost-effectiveness estimates, outlining problems with these estimates being wrong, highly sensitive to assumptions, and not reality-checked, better reporting of costs could help correct these three problems. More endline metrics We have written previously about the dangers of measuring “incomplete metrics” and have argued our concerns that measuring improvements in IQ could be one of these metrics. We’d like to see more studies focus on endline metrics that are directly connected to well-being, such as income, health, and subjective well-being. More high-quality “gold standard” studies GiveWell has laid out a case for focusing on more “gold standard” studies that are well-designed and well-executed, which help fight against the problems of drawing misleading conclusions based on other research that all tend to have the same flaws. While conducting high-quality studies is difficult, we think there are some relatively simple techniques that will ensure increased study quality. Pre-registration can mitigate inappropriate data analysis, soliciting peer feedback can improve the experimental design, and large sample sizes can reduce the chance of an underpowered study. Who is Already Working in this Area? Many organizations already provide high quality studies, such as the Institute for Health Metrics and Evaluation, IDinsight, the Center for Global Development, the World Health Organization, the Center for Disease Control, Cochrane, the Campbell Collaboration, the Abdul Latif Jameel Poverty Action Lab, Innovations for Poverty Action, and the International Initiative for Impact Evaluation. Other organizations help curate, summarize, and synthesize these studies, such as Copenhagen Consensus, the Disease Control Priorities Project, Our World in Data, and GiveWell. Though there are many global poverty research organizations, we see the potential value in further studies as enough to justify there’s ample room for more research and data collection in the area of global poverty. Reservations Our biggest reservation is the large number of unknowns in committing to research. It seems that this process involves a substantial gamble on finding a “white whale” of a highly cost-effective intervention or charity with very slow feedback loops. Not only might this task be very difficult, it may be impossible. We also put some weight on the idea that it could be easier to produce high-quality research just by implementing a particular intervention. For any charity idea we select to implement, we already plan to produce pilot studies and eventual RCTs as the idea allows. Additionally, we’re nervous about how we’d acquire funding for starting and scaling our activities. Certainly we could hope to get funding from those who would potentially consume our research, such as GiveWell or the Open Philanthropy Project. However, we think there could still be concerns for seeking large amounts of funding, as a conversation between GiveWell and Development Media International notes that “[t]here are not many funders that support this type of [research] work [because m]ost foundations prefer funding projects that maximize the number of lives saved, rather than evidence-gathering work”. Lastly, we are also unsure of whether people would be willing to act on our research once we produce it. From speaking with experts, we have found that many large aid departments in different countries may be slow to update practices based on new evidence, if they update at all. It seems that for as much effort as we put into producing high-quality studies, we may have to put equal or more effort into lobbying relevant decision makers to utilize the findings, which could be very hard to do. Remaining Questions
Endnotes [1]: This is a high-end guess based on a few figures: Replication grants from 3IE are $15K at maximum plus $10K to the author to prepare their study for replication; IPA RCTs range from $50K to $500K; and 3IE funds up to $100K for a systematic review. Update 07/12/16: The post below has been updated to more accurately represent some studies
This is one of our charity profiles, where we present our shallow, preliminary research on a potential, promising charity idea. We believe that this idea could be a potential contender for a GiveWell top charity, if further research confirmed the idea and if someone started the charity, executed it well, and resolved some of our outstanding questions and reservations. Basic Idea Send SMS reminders for scheduled immunizations to improve childhood immunization coverage (IC) in low and middle-income countries (LMIC). Summary Cost-effectiveness: High -- Rough back-of-the-envelope cost-effectiveness estimates from ourselves and others suggests a comparable cost per DALY averted to top GiveWell charities . This estimate is heavily dependent on uncertain inputs for participant acquisition costs, IC increases attributed to SMS reminders and benefits of marginal immunizations. Strength of Evidence: High -- There is a considerable amount of studies, including systematic reviews, which suggest SMS reminders can increase IC across a wide variety of contexts. Counterfactual Scalability: Medium -- There appears to be no significant funders or organizations in this specific area, so it seems a new charity would initially operate in a relatively uncrowded environment. However, given the increasing popularity of mobile health (mHealth), the field may become more crowded. Ease of Testing: High -- It would be relatively easy to do an RCT on this specific charity idea because the intervention is low cost, has quick feedback loops, and individuals can be the unit of randomization. Flexibility: High -- It appears that the SMS infrastructure could be relatively easily used to promote other valuable behavior changes. Logistical Possibility: High -- Some of the main challenges to this specific charity idea seem to be determining optimal enrollment strategies, preferred locations and specific message details and content. These challenges seem less difficult than those associated with other specific charity ideas. Why We Think This Could be an Effective Opportunity SMS immunization reminders may be easier logistically, less crowded, more flexible, more testable, more scalable, as well as being relatively evidence-based and perhaps more cost-effective than the other specific charity ideas we’re considering. Immunizations have a strong evidence base for cost-effectively improving health (see GiveWell’s page on immunizations and DCP3) and a major reason for people with partial immunizations or lacking immunizations is a problem of demand (Rainey, et. al., 2009, Megiddo, et. al., 2014; see also The Wire Magazine and Live Mint Magazine) and a lack of knowledge in particular (Nath, Kaur, & Tripathi, 2015). This creates an opportunity to increase IC by increasing demand, leveraging the existing supply of vaccines. It’s possible demand can be increased cost-effectively via SMS reminders to comply with a specific immunization schedule. In order to implement this specific idea, a charity may complete some combination of the following: recruiting parents/guardians, recording age of child(ren), generating appropriate vaccination schedules, and sending SMS reminders at appropriate times. Our review of the literature indicated a wide variety of research that generally suggested SMS reminders were able to improve health outcomes. A systematic review found that 13 out of 14 studies showed a positive effect of SMS on some health-related behavior, such as tobacco cessation or diabetes self-management, though there were some methodological issues identified that undermined this result somewhat (Fjedsoe, Marshall, & Miller, 2009). A more recent review found that 5 out of 7 studies found an impact of mHealth SMS programs on health education outcomes (Hurt, et. al., 2016). Lastly, a Cochrane review of four randomised controlled trials involving 3547 participants also found positive increases in attendance at healthcare appointments (Car, et. al., 2012). However, the vast majority of studies cited in these mentioned reviews are from higher income areas than where we would like to implement this specific charity idea. When looking at literature directly connected to SMS reminders to improve IC in LMIC, we found five studies with a positive effect ranging from an increase of 8.7 to 17.5% percentage points in IC (Bangure, et. al., 2015; Haji, et. al., 2016; Uddin, et. al., 2016; Schlumberger, et. al., 2015; Eze & Adeleye, 2015; Brown, et. al., 2015) and two studies finding no effect (Gibson, 2015; Domek, et. al., 2016). We also found five upcoming studies that closely relate to the effect of SMS immunization reminders: JPAL (location: India, timeline: 2015-2017), JPAL (Mozambique, 2014), WHO (Pakistan, 2013), John Hopkins (Kenya, 2013-2014), and Bandim Health Project (Guniea-Basseau, 2016). We contacted an author of all these studies, asking if they had any shareable results. One author confirmed they did, and we have seen the results of the study, but we will wait until publication when we can review the full analysis and methodology before commenting further. We eagerly await the final results from these and other possible additional studies. Lastly, we examined research that fit only some of our criteria. Our preliminary conclusion is this evidence aligns with the view that SMS immunization reminders are a promising idea. Some evidence suggested that IC could be improved through a combination of SMS and conditional cash transfers (CCTs) (Wakhada, et. al., 2013) and through reminder systems other than SMS (Berhane & Pickering, 1993; Busso, Cristia, & Humpage, 2015) Two of the previously mentioned seven main studies report costs involved with the intervention. Haji, et. al. (2016) reported in Kenya total messaging cost of $0.27 per child for a +7%-point increase in IC of second pentavalent vaccine dose at 10 weeks of age and +13%-point increase in IC of third pentavalent vaccine dose at 14 weeks of age). Eze, et. al. (2015) (table 3) reports in Nigeria a cost of $0.15 USD per additional immunization of a child (original figure in Naira). Busso, et. al. (2015) (table 8) did a related intervention where they gave community health workers in rural Guatemala an updated list of children who needed vaccinations. They estimate that the cost per child who completed their full vaccination schedule due to the intervention is $7.53. We have not thoroughly vetted any of these numbers and hold some reservations about them. SMS immunization reminder costs can be split into initial user acquisition costs and marginal reminder costs. We probably would send three texts per immunization per user, though we could revise this based on pilot testing. A brief analysis of five different automated SMS providers suggests a cost of $0.001 to $0.06 per text, but the cost could vary depending on our method for acquiring users, country choice, and the desired amount of texts to be sent. This suggests marginal SMS costs of $0.003 to $0.18 per immunization reminder. Predicting 4% of our users would receive a counterfactual immunization means the marginal SMS costs per additional immunization are $0.075 to $4.50. The initial costs for acquiring a user are much less clear to us and we would hope to get a better sense of this during this specific charity idea’s piloting phase. SMS also seems adaptable to a variety of interventions. We found some evidence suggesting that SMS reminders worked in LMIC for increasing Vitamin A supplementation coverage (Thiaw, et. al., 2013), improving breastfeeding practices (Lee, et. al., 2016), and increasing utilisation of antenatal care (Watterson, Walsh, & Madeka, 2015). Note that this list isn’t exhaustive and we expect that there are other valuable behaviour changes that SMS’s could cause. Thus, SMS-related infrastructure could potentially be used for other health-related interventions, though we currently think they’re less cost-effective and have a lower strength of evidence than focusing on immunization reminders. Lastly, assuming we could recruit participants relatively easily and measure their immunization status, we could relatively easily randomize participants to determine if our system increases IC. The low cost per intervention and quick feedback loops makes a pilot study especially promising. A Possible Implementation Plan Broadly, we’d want to follow these steps for implementing this specific charity idea: Step 1: Determine areas where immunization supply is substantially higher than demand and where SMS immunization reminders seem likely to increase IC. Step 2: Investigate how to acquire reminder system users cost-effectively, experimenting with a few different approaches. We may experiment with cash incentives, partnering with advertising firms or peer-to-peer acquisition strategies. PATH’s mHealth Mobile Messaging Toolkit could be a helpful guide for rolling out some of these approaches. Step 3: Determine optimal number, timing, and content of messages. Step 4: Seek results relating to SMS immunization reminders and update accordingly. Step 5: Decide on a scaling process if it’s thought to be worth doing. Step 6: Partner with academics to run a RCT on the intervention. Step 7: Evaluate results of RCT and act accordingly. Who is Already Working in this Area? Multiple experts we spoke to agreed that mHealth suffers from “pilot-itis”, where most charities run a small pilot but never scale. None of the experts we talked to seemed to know why this is the case, though one person suggested it was because the government was not involved. As far as we could find, there are four organizations working in India that mostly provide SMS reminders, Vaccidate, eSwasthyaSeva, Immunize India, UNICEF, and mMitra. Vaccidate and eSwasthyaSeva seem small, internet-based, and focused on more wealthy Indian residents. Immunize India is larger (with over 700,000 users) but a large number of its signups are from the Indian middle class who are less in need. UNICEF is just running a pilot, which could very well suffer the same problems as all the other mHealth charities. mMitra is very small, only reaching about 42,000 people. When we talked to middle class locals in India, none of them reported receiving any sort of health reminder over SMS or any other medium. However, our impression is mHealth is becoming increasingly popular, so this idea may soon be expanded upon. For example, UNICEF is working with the Bangladeshi government to start a program to send reminders to do post-natal visits with a charity health worker (CHW) for both the CHW and the mother (mHealth Compendium, 2015, p79). Reservations To what extent do SMS reminders increase IC, after delays are taken into account? Some of the main studies report differences in IC only a few weeks after scheduled immunization date. For instance, Haji, et. al. (2016) and Bangure, et. al., (2015) report a 13% and 16% point increase in DPT3 coverage a few weeks after it being scheduled to be delivered at 14 weeks after birth. However, two systematic reviews identified a median delay of 6.2 and 6.3 weeks for DTP3 immunization in LMIC (Akmatov & Mikolajczyk, 2012; Clark & Sanderson, 2009). Studies that report differences in DTP3 IC only a few weeks after its scheduled delivery may give a misleading view of longer term IC increases attributable to the intervention because some children will receive DTP3 after the study finishes endline data collection. As a result, the counterfactual IC increases, say six months after scheduled immunization date, is likely less than the reported IC increases for some of these main studies but we aren’t sure to what extent. If the increase in IC attributable to SMS reminders six months after the scheduled immunization date was substantially less than the reported difference a few weeks or months after the scheduled immunization date this would significantly affect how promising we feel this specific charity idea is. Should we put less weight on these main studies? The ease of completing small scale studies on this intervention may make it particularly susceptible to publication bias and it’s worth noting that some of the main studies don’t appear to have been registered. Also, the lack of pre-analysis plan and independent re-analysis of the data makes it quite possible that inappropriate data analysis practices causes results to not replicate. What’s more, as the main studies are very recent, no reviews have commented on their quality or risk of bias and our own interpretation could miss some possible flaws. Will there be any limitations to sending texts? Potentially there are sizable limitations to sending reminder SMS’s at scale. Twilio, a cloud communications service that automates SMS sending mentions that, in India, the following limitations for sending automatic SMS:
Remaining Questions
This is one of our charity profiles, where we present our shallow, preliminary research on a potential, promising charity idea. We believe that this idea could be a potential contender for a GiveWell top charity, if further research confirmed the idea and if someone started the charity, executed it well, and resolved some of our outstanding questions and reservations.
Basic Idea Lobby low and middle-income countries (LMIC) to increase tobacco taxes. Summary Cost-effectiveness: High -- Studies claim this to be one of the most cost-effective interventions, with $/DALY figures more cost-effective than estimates for most other interventions. (However, these cost-effective estimates shouldn’t be taken literally and expect the true cost-effectiveness to potentially be much lower.) Strength of Evidence: Medium -- The case for tobacco taxation is covered by several studies, including some systematic reviews. However, we are concerned there is a lack of research on key lobbying related factors, such as counterfactuals, average campaign time, and the base rate of successful campaigning. Counterfactual Scalability: High -- If this was successful in one country, it seems like we could easily adapt the infrastructure to other countries. Moreover, there is only a few large organizations and two large funders working in the area, suggesting large room to grow. Ease of Testing: Low -- Performing an impact evaluation on a lobbying campaign is exceedingly difficult due to high interference from other campaigns, the all-or-nothing nature of lobbying, and an inability to effectively randomize. Flexibility: Medium -- While it appears we could move lobbying efforts to other areas as we gain more generalized skills and contacts, we would likely not build much infrastructure that is useful for areas outside lobbying. Logistical Possibility: Medium -- Lobbying poses significant logistical challenges and draws on skills we do not yet possess. Why We Think This Could be an Effective Opportunity There’s strong evidence that increasing tobacco taxation reduces tobacco consumption and that reduced tobacco consumption results in improved health outcomes (International Agency for Research on Cancer, 2011; Jha, et. al., 2013). Without cessation increases, tobacco use may account for some 10 million deaths per year by 2030, with most deaths occurring in LMIC (Ibid.). For example, in India, smoking accounts for nearly 40 percent of tuberculosis deaths among middle-aged males, or about 120,000 deaths annually. (Jha 2012). WHO (2011) estimates a 10% price increase in tobacco causes a 4% reduction in tobacco consumption. Jha, et. al. (2013b) estimates that increasing taxes to the World Health Organization (WHO) recommended level of 70% of total sale price in all LMIC would result in an 11-27% decrease in smoking mortality and therefore save tens of millions of lives. There are also numerous pieces of evidence suggesting that government implementation of tobacco taxation is highly cost-effective. Savedoff and Alwang, (2015), writing for the Center for Global Development, price tobacco taxation as $3 - $70 per DALY averted. Jha, et. al., (2013), writing for the Copenhagen Consensus, estimates a 4000% return, with a $500M/yr tobacco control program averting more than one million deaths annually. Jha, et. al., (2013) also states that tobacco taxation is the most important and most cost-effective portion of the tobacco control campaign. A systematic review of 84 studies also found that taxation was among one of the more effective tobacco control initiatives (Thomas, et. al., 2008). Experts we spoke to at tobacco lobbying campaigns and evaluative organizations (e.g., the DCP3) also broadly agreed that this was a strong area. A Possible Implementation Plan The most plausible route we identified was hiring local lobbyists and tobacco control experts to implement tobacco taxation legislation at the national level. Our starting point would be consulting the Framework Convention on Tobacco Control, which provides tax policy specifics, refutations of tobacco industry talking points, and guides for lobbying policy makers. The WHO Framework for Tobacco Control also contains additional information. We think it is best to focus tobacco taxation campaigns on lobbying LMIC because the costs of lobbying are significantly lower than the developed world and there is less existing lobbying efforts in place. There are many LMIC that fall beneath the WHO policy prescriptions for tobacco control which recommend taxation make up 70% of cigarette costs. Campaign for Tobacco-Free Kids (CTFK), a large non-profit working in the area, has prioritized their campaigns in most LMIC with a high burden of disease from tobacco that are also not facing violent internal conflict. If we were to work in one of those countries it would make sense to partner with them. We might be interested in working where CTFK isn’t, so that we can test our counterfactual impact free from the involvement of other campaigns. We very weakly think that Ethiopia may be a promising place to start, based on some preliminary research. CTFK has not prioritized Ethiopia, perhaps because of their low overall smoking rates. However, Ethiopia has a large population and they recently banned smoking in public places, which suggests political will may exist for further tobacco control. Who is Already Working in this Area? Our research suggests that Bloomberg Philanthropies and the Bill and Melinda Gates Foundation are currently the dominant funders in this area. Callard (2010) estimated no more than $240M is spent annually on tobacco control, worldwide, but this could be out of date due to more recent grants and GiveWell has argued that this study makes fairly aggressive assumptions (GiveWell report on tobacco control). Between 2007 and 2015, Bloomberg Philanthropies reportedly committed $600 million to combat tobacco use worldwide (Bloomberg press release, 2015), which amounts to $75M/yr. In 2008, the Bill and Melinda Gates Foundation pledged $125M over five years ($25M/yr) to tobacco control, including a $24M grant to Bloomberg (BMGF press release, 2008). CTFK has several tobacco control initiatives, including a focus on taxation. PSI in India ran Quitline from 2011 to 2013, which was a telephone help line primarily offering counselling to help people quit tobacco addiction. They encouraged the Indian government to continue the campaign, which, according to PSI, the government is planning to do but has not (Ibid.). In 2015, PSI in India also started a campaign to reduce exposure to second-hand smoke and tobacco use among minors (Ibid.). WHO offers technical assistance for governments trying to implement tobacco control. Resource Centre for Tobacco Free India is an advocacy and research group that has implemented various tobacco control initiatives, including taxation. The Economics of Tobacco Control Project, based in South Africa, creates new research and train NGOs about tobacco control initiatives, including taxation. Reservations The evidence for lobbying succeeding in LMIC is far weaker than the evidence for the impact of successfully implemented tobacco taxation. It’s often difficult to credit responsibility for legislative change to particular players in advocacy campaigns because multiple campaigns operate simultaneously and it’s unclear whether the government would have changed anyway, absent lobbying efforts. While there is considerable evidence the government implementation of tobacco taxation is highly cost-effective, there is a dearth of analysis on the return of investment of actual lobbying campaigns. We find the difficulty of testing this project to be a barrier to attempting an implementation, given our desire for rigorous impact evaluation and prioritizing against other charity ideas, as well as this making it difficult to receive feedback to refine our tobacco taxation approach. However, depending on local tax policy, this specific charity idea may be able to be completed at the state level before attempting it on the national scale. We’re also not enthusiastic about would could be the zero-sum and all-or-nothing nature of lobbying. We may not only have to be effective enough to lobby the government and beat the status quo but we may have to be more effective than the pro-tobacco interests lobbying the government. It seems quite possible that we could spend a significant amount of resources lobbying and not accomplish any actual legislative outcomes. According to some experts we spoke to, legal attacks on lobbying efforts from tobacco companies are also somewhat common. It would very likely be necessary to hire local lobbyists to reach the needed policy makers, which could be expensive. This also creates flexibility concerns, as the contacts we build would likely not be useful for a charity other than lobbying. Moreover, if lobbying requires a high degree of specialization, it may not even be possible to move easily between different lobbying campaigns. Funding could be a concern, especially if we were seeking a diversity of sources. While this intervention has large support from the global health community, there aren’t many large funders in this specific area. This could, however, suggest that the area is neglected. Remaining Questions
Everything takes longer
One of the biggest things we noticed while in India was that it takes large amounts of time even to accomplish fairly routine tasks relative to trying to do them in Canada. This no doubt was in part due to our unfamiliarity with the country, but based on what we saw, we think this could also be true even for natives, though to a lesser extent. For example, basic tasks like getting a vaccine or getting a cell phone set up were far cheaper but took far longer than doing the equivalent task in Canada. For example, getting a vaccine required considerable time and many different small steps for payments and waits (getting the appointment, for getting the consultation, prescription, getting prescription filled, getting injection, paying for each step). Mobility is really important One thing we noticed was that mobility was far more important in the locations we visited in India than almost anywhere in Canada. Sidewalks were narrow and paved inconsistently. It was hard for even the most able bodied individuals to cross many streets. Whenever we considered the impact of a health problem that impaired mobility it seemed like it would have a far greater impact in India than in Canada. Cellphones get a lot of texts Our cellphones got a very large and consistent number of advertising texts (5-10 a day) starting from the first day we got local cellphones. We could not find good data on whether this was average or abnormal but it made us more cautious about SMS interventions than we were originally. If we do any intervention based on SMS, we’d have to find a way to establish authority and make it clear that any health messages sent were not merely advertising. People mostly hang out in their own wealth brackets Unsurprising like in most countries there are pretty large deviations between different wealth brackets. When we talked to most individuals about their connections to others it was very often in the same wealth bracket and general socio-economic status. Pollution is really bad The pollution was shockingly bad. We had a sense of this from the statistics going in, but it was a whole different experience to feel it in person. At nighttime burning garbage and wood cook stoves made the air extremely polluted. Even far outside of large cities there was still the feel of some pollution in the air. There were many different languages spoken There was a huge variety of languages spoken in India that we encountered. Some of the people we met were trilingual and said they still had major translation problems in some places. The ability to speak fluent English was extremely correlated with wealth, although most people we encountered could speak a few English sentences. A surprisingly large number of signs and written language that we encountered were in English. This helped remind us that we would really need to have a good sense of what specific area we are working in before hiring staff or we would risk having major language problems. Different newborn health practices We learned there’s a tradition in many parts of India to not take a newborn or the mom out of the house until months after birth. This can cause problems because this holds true even if the newborn is sick and needs care. We found quite a few differences like this that could affect different programs we were considering. People self reported money and jobs as more important than health Quite a few times we asked impoverished people what they most desired or what they thought would improve their happiness most. The most common answer we got was the desire for employment or a better job. People's happiness seemed to vary a lot depending on their job and it also came up a lot in general conversations. The second and third most common things we heard were money and food. We almost never heard about health issues or loss of family members due to health concerns. We are not sure what causes this or if it would be different in different areas. The smallest city we stayed in was still over 100,000 people and we suspect this could affect the answers quite a bit. People we talked to seemed to think life was getting better Many different people we talked to commented about how their personal life was getting better or mentioned that life in India is generally improving. India has experienced huge income and health gains historically, but it was interesting to see how many people had actively noticed and appreciated the improvements. --- This blog post had been edited and updated based on new information sent to us about the relationship between IQ and income. We feel as though the evidence is more positive than we originally wrote but still not enough to change our conclusion that there is insufficient evidence of a connection between IQ and other positive life outcomes for us to value increasing IQ as a potential metric. ---
When looking at interventions in the developing world, you have the opportunity to improve a lot of things, from education, to job performance, to income, to reduced disease burden, to improved eyesight, to improved intelligence, and more. But how good are these things? In “Six Examples of Measuring Incomplete Metrics …and How to Fix Them”, we shared a concern over “incomplete metrics” in non-profits, like web traffic or growth in budget size, that don’t get at the actual good produced by the non-profit. But when thinking about how to robustly define what is good, we realized that avoiding incomplete metrics might be harder to avoid than we thought. When looking at increasing salt iodization, which is a GiveWell priority program rated to have “[r]easonably strong evidence of effectiveness”, we saw an opportunity to improve IQ. And this got us wondering -- how good is improving IQ? Could this be an incomplete metric or does it actually have a connection to the kind of good that we value? Is IQ and end in itself? In the book Utilitarianism, Mill said that “It is better to be a human being dissatisfied than a pig satisfied; better to be Socrates dissatisfied than a fool satisfied”. This argument has been taken to mean that intelligence is a good from a utilitarian perspective, insofar as intelligence is a key contributor to a life worth living. Thus even if your life is miserable, it would be better, on Mill’s view, than a life of only simple pleasures. Other utilitarian authors would have disagreed with Mill. Bentham, whom Mill is arguing against, put forth a view that all pleasures are a part of the same “hedonic calculus” and thus it would be possible, at least in principle, for a satisfied pig to have a happier life than a sufficiently dissatisfied human being. After all, an opposite view is that “ignorance is bliss”. Ultimately, we take this to be more of a value judgement and less of a question of empirical fact. Do you value intelligence for the sake of intelligence alone? Or do you only value intelligence as a means to an end, claiming the existence of empirical evidence that ties intelligence to a more satisfied life or an enhanced ability to create satisfying lives for others through intelligent works? While looking for effective charity is hopefully an empirical matter, a lot of questions of what interventions are best depend on judgement calls about values that we believe cannot be fundamentally grounded in empirical research alone. We think reasonable people can disagree and this can change their opinion of which charity is the most effective, even if one looked at exactly the same empirical evidence. Thus, we on the Charity Entrepreneurship team want to make it clear what we do and don’t value and which value judgement calls we are making and how they affect our final judgements about which charity ideas we recommend become reality. Effects of IQ on things that matter The value judgement that our team makes is that IQ is not a good in itself and that instead we want to find empirical research that substantially connects IQ to the outcomes we do care about, which are mainly that of improving psychological well-being. When GiveWell talked with Dr. James Flynn, Emeritus Professor of Political Studies and Psychology at the University of Otago, he argued that a gain of four IQ points would be enough to allow someone to function independently as opposed to needing special assistance, that IQ improves job performance even on the menial jobs common among the extreme poor, and that gains in IQ would reduce the prevalence of intellectual disability. Improving education and job performance seems good, but also just kick the can down the road. Sure, if someone does well in school, do they then go on to have an increase in well-being? If someone is better able to function at their job, do they go on to have an increase in well-being? Presumably there could be a complex chain of impact, where an increase in IQ boosts job performance, which boosts income, which boosts the ability to provide food for oneself and one’s family, which boosts well-being. But this chain is complex enough that it could be reasonably doubted and empirical evidence would have to establish it. IQ and subjective well-being Empirical evidence does show some connection between intelligence and well-being. Academics have mainly looked at the connection between IQ and “subjective well-being” (SWB), which is one way to capture well-being that we like as a metric, but it has pros and cons that we intend to elaborate on in a future post. When looking at the benefits of IQ, we’re open to finding connection between IQ and any other metric that we find important, like DALYs, SWB, or income. Cross-country studies find a connection between IQ and country-wide life satisfaction (Salahodjaev, 2015). However, it is less clear if there is an effect on the individual level. Diener and Fujita (1995) found a 0.17 correlation between IQ and subjective well-being in their developed world subjects after controlling for other factors, and they suggested that IQ fits in with a much broader view of “personal resources” that is much more highly predictive of SWB. Moreover, Diener and Fujita argued that personal resources would play a very different in a developing world context where immediate needs (e.g., food) are more dire. Additionally, a 40-year longitudinal study in Luxembourg compared cognitive ability as measured at age 12 and compared it to SWB as measured at age 52 and found no connection between childhood cognitive ability and adulthood life satisfaction when controlling for childhood socioeconomic status, though they did find weak effects on satisfaction with individual life components (e.g., satisfaction with health or satisfaction with finances) (Chmiel, et al.2012). The macro-IQ, micro-IQ paradox A meta-analysis puts this conflicting picture in perspective. Veenhoven and Choi (2012) analyzed 19 studies and found no effect of intelligence on life satisfaction on the individual level, but analyzed four studies focused on the connection between national intelligence and national happiness and found a significant correlation, even after controlling for economic development. They speculated that this could be caused by intelligent people having higher expectations for themselves, additional uncontrolled factors beyond economic development (as controlling for economic development is tricky and other factors could be at play, like culture), or that intelligence could just add indirectly to the quality of society without impacting individuals directly. After all, higher IQ has been connected to more efficient government institutions (Kanazawa, 2009), reduced violent crime (Bartels et al, 2010), higher levels of interpersonal cooperation (Jones, 2008), and increased trust in society (Carl and Billari, 2014). Together, these effects could create a long-run benefit for intelligence in societies, even if there is no benefit at the individual-level. IQ and income So if IQ perhaps doesn’t matter much for individual SWB, we could look at its affect on income (a metric we think is pretty connected to SWB and happiness). It seems intuitively plausible a higher IQ should result in both increased job performance and increased educational ability, which results in better access to jobs and thus better income? Some evidence does suggest this. Psacharopoulos and Velez (1992) conducted an econometric analysis in rural Columbia that found a half standard deviation change in IQ lead to a 3.5% direct change in wages. Literacy scores show a reasonably strong connection to income across thirteen developed nations, even when controlling for some non-school factors (Hanushek and Woessmann, 2008). It is possible that those in the developing world may not see a benefit from improved IQ simply because they don’t have an opportunity to put improved IQ to use. Banerjee and Duflo (2006) found that, as we would expect, the jobs in the developing world are very different than the jobs in the developed world. Most jobs are agricultural, and the non-agricultural jobs are typically either doing general labor or sales (ibid.), which might not make use of improvements to IQ. For example, Jolliffe (1998) found that cognitive skills raised non-farm income and total income in Ghana, but not farm income, which suggests that education may not be as helpful to the many of those focused on agricultural jobs. Additionally, Vijverberg (1999) found that the effects of education on the self-employed in Ghana were weak or nonexistent. A similar pattern can also be found in the United States, where a large-scale study found that the returns to IQ were moderate and positive across all occupations, but were lower for manual jobs (e.g., farming) than non-manual jobs (e.g., banking) (Kuncel, Hezlett, and Ones, 2004). However, in the developed world, even manual jobs often involve operating complex machinery, so the external validity to the developing world is questionable. IQ and education Another area for question is how appropriate it is to control for years of education when looking at the connection between IQ and income, since IQ may easily cause more years of education, which then causes higher income. This is what Glewwe, Huang, and Park (2015) found in rural western China -- a high connection between IQ and educational attainment, but no connection between cognitive skills at an earlier age and wages at a later age, after controlling for educational attainment. Boissiere, Knight, and Sabot (1985) found in Kenya and Tanzania that the return on education was moderate across both manual and non-manual workers (though higher for non-manual workers), but that the returns to reasoning ability was mostly nonexistent after (a) amount of education and (b) the presence of basic numeracy and literacy were controlled for. Other studies find a connection between education and income too, though they don’t mention the role of IQ. Asadulah (2006) found statistically significant effects of primary education on wages in Bangladesh, especially for females and those in urban settings. Duflo (2001) found a connection between education and wage for primary education in Indonesia, but found no effect beyond age 7. Lastly, Alderman,et.al.(2009) found that there was an effect of education on wage in rural Pakistan that was the result of increased cognitive ability and not the credentialism of having a degree. One concern with this avenue, as pointed out by Behrman, Ross, and Sabot (2002), Duflo (2001) and GiveWell’s review on education, (2009), is that even if studies do establish a connection between education and income (or even IQ and income), these studies mostly focus on uncontrolled correlations and therefore exhibit a selection bias by highlighting only the participants who can access and maintain their education, which is usually indicative of other resources (e.g., sufficient family income and support) that could also, independently, promote wages and income. This also means we must take individual context into account. Even if education in the developing world has some connection to wage, it may not have a connection for the “poorest of the poor” who are most likely to not be able to go to school, have a low quality school, or not be in a position to get a job that lets them take advantage of their education. Furthermore, these children are also most likely to be victims of malnutrition or other health problems that would keep them from gaining anything from school even if they did attend. For these reasons, special care is needed to pay attention to the external validity of studies to the populations we may be trying to serve. Overall, while we think it’s likely that IQ improves income in the developed world, there is not as clear as case for this in the developing world as we would like to see based on our desire for a strong strength of evidence. We’re concerned that mere observational studies of the impact of IQ may end up conflating IQ with general resources, like family finances, which can also improve educational attainment and future income. Ideally, we would like to see a randomized controlled trial that isolates the impact of improving IQ on life outcomes -- however, we have not been able to find one yet. We intend soon to write up how we assess strength of evidence, what our criteria is for how much strength of evidence is necessary, and what claims we do think have sufficient strength of evidence. What about intellectual disability? One aspect we did not mention is reducing the prevalence of intellectual disability. Dr. Flynn told GiveWell that “if the IQ of a population were normally distributed with a mean of 100 and a standard deviation of 15, then a four point increase in the IQ of the population would reduce the prevalence of intellectual disability from about 2.3% to 1.2%.” While the main effect of improving IQ of non-disabled children may not increase income, it is quite clear that there are sizable impacts on wages from establishing basic literacy and numeracy (Boissiere, Knight and Sabot, 1985; Hanushek and Woessmann, 2008). It’s possible that we might find impact among reducing the level of intellectual disability, however we’re still unsure about the severity of the intellectual disability that is being averted and whether it would make the difference between whether someone becomes literate and numerate or not. One aspect we were curious about, however, was whether reducing intellectual disability had any effect on well-being directly. A survey in England found that mean overall SWB was only slightly lower among the intellectually disabled, as compared to the rest of the population (Emerson and Hatton, 2008) and similar findings were also found via a survey done in Kentucky (Sheppard-Jones, 2003). Of course, this could be wildly unrepresentative of experiences in the developing world, where culture and institutions of care can be very different than in the developed world, so we would take the effects of improving income more literally than these effects. So does IQ matter? Overall, we remain unconvinced that increasing IQ is as worthwhile a pursuit as other metrics, like increasing income or decreasing the burden of disease (as measured in DALYs), both of which we intend to talk about at length in the future. We think, intuitively, it is plausible there is a connection between IQ and well-being. However, empirical studies point to the idea that this connection may only bear out on the macro-level and not on the micro-level. This makes the overall effect size of IQ-improving interventions incredibly unclear and complex to study. For example, we wouldn’t know to what degree iodization of salt contributes to improved social institutions and reduced crime, making it very difficult to form sufficiently robust cost-effectiveness estimates. We also think it is quite possible there is a connection between IQ and income in the developing world. However, existing studies suggest that this may happen by improving educational attainment, which might not work in contexts where educational access is harder to come by. Furthermore, all the studies of IQ and income we found in the developing world are observational studies, which opens up the possibility that any observed effect of IQ is the result of the influence of uncontrolled additional variables, like family resources, that cause both increases in IQ and increases in educational attainment. Overall, researching the impact of interventions is already hard enough without worrying about the difficulty of knowing the effect of the intermediary metric on outcomes we care about. Therefore, we remain skeptical that improvements to IQ will lead to robust, concrete beneficial outcomes for the extremely poor people that we would focus our efforts on. If you want to create an effective charity, how do you know if it is effective?
A very common answer is doing detailed measurement and evaluation. But this is harder than it sounds. While research is hard to get right in general, the difficulty starts just with the measurement… which metric should you measure? The Case of the Hard Worker Imagine that you work for an EA organization. You work really hard. In fact, last week you completed 154 pomodoros of work. Was it effective? It’s hard to know without knowing what the pomodoros were actually spent on. Sure, your inputs matter some and it’s possible that more pomodoros means more good happening. But instead you need to measure your outputs. Time spent is just a means to accomplish certain high-impact goals. How much good did you accomplish with those pomodoros? The Case of the Growing Organization Imagine that you’re the CEO of an EA organization. You were created in 2011 and you had a budget of $30K. Now it’s 2016 and you have a budget of $300K. That’s 10x growth in five years! That’s amazing! ...But is it effective? The amount of budget your organization spends, like the amount of time your employees work, are only a means to an end. Again, what matters is the results of your budget and what you accomplish with them. It’s possible to do more good with $30K than $300K. The Case of the Fundraiser Imagine that I’m a fundraiser for GiveWell top charities. How do I know whether I’m effective? What if I told you that I was able to send out over one million cold contact fundraising emails in 2015. Would you think I’m an effective fundraiser? Sure, it’s likely that emails lead to donations and that more emails likely means more donations. But without tracking any sort of metric of whether people are (a) opening the email, (b) clicking the donate link, and (c) actually making a donation through that link, it’s impossible to tell. Only by measuring actual donations made can we truly measure or impact. The Case of the Charity Club Imagine I created a college group called “Charity Club” where we had monthly meetings about donations and career choice. In 2015, we got 100 new members and held ten meetings with an average attendance of thirty. ...How do I know if this project was effective? Sure people attending meetings about donating to effective causes is likely a good thing. But attending meetings about donations is not good in itself… instead what matters is people actually making effective donations. So we instead should measure the actual donation habits of club members. But what if club members were just more likely to donate, entirely irrespective of the existence of the club? Ideally we would measure counterfactual impact by randomly only taking some of the interested people into the club and measure their donation habits relative to the people who weren’t allowed in the club. ...But chances are that this RCT is not very practical. The Case of the Immigration Reform Imagine that I think improving immigration in the US is important for economic growth and the welfare of immigrants. So I set up an advocacy website that encourages people to write to their congressperson and encourage immigration reform. ...How do I know if this website is effective? One way is that I can measure web traffic. More traffic should be good, right? But what if I get a lot of visitors but no one follows through and writes to their congressperson? Okay, that’s bad, so maybe I should measure the number of letters to congress people that get delivered. That does measure our influence over the public process, but what if the petitions get ignored? How do we know our petitions lead to legislation change? What if the legislation would have changed anyway? What we really want to measure is counterfactual legislation change. To do this, we construct an RCT where we randomly select some legislators to be targeted and some not to be and then we see whether the targeted legislators are more likely to sponsor immigration reform than the non-targeted legislators. While web traffic -- or even the amount of letters sent -- is a positive thing and might contribute to more immigration reform, it could easily not be connected to immigration reform. Only measuring the right thing helps us check. The Case of the Developing World Charity Imagine you’re the executive director of a charity that does unconditional cash transfers to the global poor. How do you know you’re doing good? The problem is there is nothing inherently good about people having more money than they used to. Money is just a means to an end. So to know how much good we are doing, we need to see what is happening with the money that we give. What are people spending on? Does the money actually make them happier? To do this, we need to measure the effects of giving money, hopefully with an RCT. Why might charities focus on incomplete metrics? In each of the above examples, we consider the metric being measured to be incomplete, or that the metric actually needs to be investigated further before we can clearly connect it with positive impact. But why might charities focus on these incomplete metrics? It is easier Some metrics are way easier to measure than others. It’s far easier to measure the web traffic to your advocacy website than to do an RCT on your legislative impact. It’s even much easier to measure web traffic than to measure the actual amount of petitions sent. Thus increases in web traffic get cited a lot as a criteria for success, even when it may not be connected to the charity's real goals. It looks more impressive Reporting on several metrics looks more impressive because you are showing more data even if the data is not reflective of the good your charity is doing. Additionally, the more metrics you have, the easier it is to cherry-pick the ones that are going well and play down the ones that are not going as well. This kind of practice makes your organization more appealing to donors and members, even if it is ultimately an illusion. Charities are unsure what the important metrics are This is true particularly with younger charities, as well as charities with less of a clear focus. When the goals are not clear, this will often result in reporting on several unhelpful metrics or missing very important ones. Ways to avoid this mistake Think of the number one most important metric This is hugely important as it makes clear what your organization is really aiming to do, and how you will measure it. Letting the public know your most important metric also allows them to focus on what really matters. Be sure of the connection between your metric and to real good happening in the world Even with a straightforward metric, you have to make sure that it really translates into good getting done. With money raised you would need to look at the charities you are moving money to, and make sure they will accomplish good with extra donation. For website traffic, you would have to make sure website traffic really correlates with actions that you really want to achieve and furthermore make sure those actions correlate with more impact. Think if it would be possible to cheat this metric Is it possible to be “game” this metric, thus making it less valuable? For example, if I wanted to gain a bunch of website traffic, it would be quite easy for me to invest in non-targeted online ads or just directly buy “views” to my website. Although this would boost my website traffic, it’s very unlikely to cause any real good on the metric I really care about. Watch out for counterfactuals An easy mistake to make is to measure metrics that have many possible causes. Given that many organizations are working towards the same goals, it is necessary to be able to isolate the impact your organization is having when compared to the wider movement. Be cautious of longer causality chains Consider an unconditional cash transfer charity. Their “chain to impact” looks like this: We give grants of unconditional cash transfers to the global poor → the global poor spend the money on what they desperately need → They are happier because they could afford a basic necessity or invest in their future → Good is achieved. Furthermore, we’re pretty confident that each link on this causal chain because there are multiple studies supporting each link. Now consider an organization that fundraises for the unconditional cash transfer charity and cites web traffic as their metric of success: website traffic → People are then more interested in donating → More people go to the cash transfer charity website to learn more → More money is donated → The transferred cash is spent on basic necessities or investments → People are happier → Good is achieved Not only is this chain longer, but there is also a huge problem in the assumption that website traffic results in more donations. While we can easily track website visits, it’s very difficult to track how many of these visits translates into more donations, and it’s easy for the metric to get cheated by getting large amounts of "lower quality" traffic. Generally the more steps you have the more confidence you need to have in each of the steps working. Focusing on the right metrics in Charity Entrepreneurship For folks like us interested in creating the most effective charities, we also need to be careful about metrics. We care most about having the largest counterfactually positive impact on global well-being (for both humans and nonhuman animals). We don’t want to look at any incomplete metrics, like the number of people we help, the size of our budget, or how many people read our blog posts. But well-being isn’t a very precisely defined metric and very few RCTs look at this. Instead, there are many more precisely defined metrics that we could measure, such as the impact of our charity on improving the length of life, on reducing the burden of disease and disability, on improving income, improving subjective well-being, etc. After a lot of research we have opinions on each of these metrics. While each of the metrics is a lot more nuanced than the ones in our examples and none of them are clear examples of incomplete metrics, we do think that some of these metrics are more complete than others relative to measuring the ultimate goal of global well-being and we plan on writing up detailed thoughts on them soon. When we decided to do this project many of our advisors suggested we spend time in the developing world to get a stronger sense of some factors that we wouldn’t from reading the statistics. Only one member of our team had spent significant time in the developing world and we thought there could be considerable learning value from spending some time there.
One way we came up with to take advantage of this learning value was living in India while doing our, mostly online, charity and intervention research. This way we could slowly get a soft sense of many different factors over time through a sort of passive learning. We learnt a lot about India in the first two weeks. We got a more visual sense of cellphone/smartphone penetration rates -- very high even in slums -- and saw the importance of mobility in walking through the Indian streets -- the streets and walkways are often rugged and only partly paved, making travel harder on anyone with a disability. But as time progressed, we hit less and less of the “passive” learning value activities, and instead we suspected that the highest value remaining learning activities would come from more deliberate and time consuming activities, such as surveying certain populations about our top charity options or running specific micro pilots. This would come at the considerable cost of reducing our online research hours. We don’t feel this is a good tradeoff at this point in our research. We have also found some cons of living in India. Most importantly, we felt a subtle but measurable drain on our weekly productivity. While it is hard to tell the full cause of this, we felt the weight of poorer sleeping and living conditions, higher levels of air pollution/sickness, regular internet and power outages, and general logistical time costs of living in an unfamiliar country all likely contributed. It became clear to us that spending further time learning in India would come at a significant cost to running our research program. After spending about two months in India we did a plan re-evaluation and we decided that it wasn’t worth the costs to remain in the developing world at this time, though we’re definitely opening to returning at a later point, perhaps for a shorter time span and with a more specific plan of what we would like to accomplish. We also think it could be quite valuable to focus on a more specific area where we could hire a translator and get a sense of the more specific cultural factors we would expect to interact with when creating a charity there. We recognize that our internal research is greatly outpacing our current external publication of that research and that people may have questions and feedback that could help us do better research. While we do intend to write up all of our thoughts at length over the coming months, we thought the easiest way to bridge the gap in the meantime was to hold an open Skype call about our research.
We have recently published some summaries of our research from the first two months, as well as our intervention spreadsheet and soon plan on publishing our promising charity ideas for future research. We would love to hear from anyone who has questions or comments on any of these topics. The format would be a brief summary of our research with the majority of the time open for questions. The call will be held Thursday March 24th at 7pm PST. To join the call, dial 1-302-202-1119 and enter the conference code 654083. If you would like to ask a question you can email it, before or during the call to joey@charityscience.com or press “5*” during the conference to queue to ask a question. We will post a recording of the call afterwards so that anyone interested can listen. Two months ago, we set out to identify possibilities for future GiveWell top charities, with the intention to start one ourselves. We started out by looking at the GiveWell priority program list and added a few more interventions we thought could be promising as we conducted our initial research. We ended up with a list of 28 target interventions to research further. After two months of research, we have now reviewed all our target interventions, ranking each on eight key criteria (from low to high) and producing overall scores (out of ten). Here are our results:  We researched each intervention for about fifty hours each, producing over 900 pages of notes. We already explained the interventions that didn’t make the cut and why. Over the next month, we will be posting many blog posts explaining our thinking on these criteria and outlining our research on six interventions that did make the cut. We will also eventually start posting our ideas about which concrete charities we think may be high value and why. After completing our first two months of research across thirty intervention areas, we have now selected numerous specific charity ideas from eight broad intervention areas for further research. Twenty-two intervention areas have been ruled out for further research, as we don’t think they offer as strong options as these other eight intervention areas.
We have passed our March 1st deadline for our first section and have not quite completed all we aimed to do. We have two large changes from our original goals and outlined steps. The first change is we moved our deepening knowledge step to our next research phase. We found that as we did the research a deepening knowledge step on a whole intervention did not seem as productive as spending more time on our top specific charity ideas. This was particularly true because within the 40 hours of intervention research, we often had much more specific ideas we wanted to pursue at a deeper level. The second big change of plans regards the publishing of our research. Our original plan was to have all of our research notes published in real time to allow for feedback from the broader effective charity community. We found publishing our research to be extremely time consuming, even with the help of our volunteer team, as a huge amount of our researcher time had to be spent improving and making the reports more understandable to someone who wasn’t familiar with the intervention area. We also ended up spending a lot of time polishing and improving reports for interventions we were very confident we were not going to do future research on. This felt like a waste as our main goal is to find the most promising interventions instead of fully explaining our reasoning for the ones we ruled out. So although we planned on doing both of these aspects before March 1st we now plan on moving the deep dive section to our specific charity ideas research and to publish higher level and more board information instead of all of our notes. Specifically we plan on publishing our summary spreadsheet, and a short paragraph explanation for the main reason we ruled out the interventions we did. We expect these to be published in the next few weeks. We plan on and already have started using some of the time gained from these moves on more research into some areas that were not originally on our list,like NCDs in the developing world and to get more specific feedback from domain experts.  We knew when coming to India that a huge percentage of the population was vegetarian and many different cultures in India had respect for animals significantly different from that seen in the West. However, we had no idea how animals would be treated on a day to day basis but wanted to keep an eye out and try to get a sense of what an animal's life is like and particularly how farm animals might be treated in India. That said, the following are informal observations based mostly on walks and travelling we have done for purposes not related to animal welfare. We can imagine our views changing over the rest of the time we are here.
Charity Entrepreneurship has been running for about one month so far with intentions to complete our shallow intervention research in two months (by the start of March). In this time we have also moved to India, which slowed down our research considerably over the first week. We feel as though we are roughly on pace, although we might not fully complete this section until mid-March. The picture below is a chart of our current progress. We would love to hear about any intervention areas not listed here that seem promising to research. Our list originally came from GiveWell, and as our research has progressed we have made a few additions. The order in which things have been researched do not reflect our views on their promisingness. (Green means complete, yellow means in progress, red means not yet started)  Our team is now in India, and one of the first things we did was arrange a tour of a large slum in New Delhi. We did this through PETE, a local nonprofit that provides free educational programs,. This tour was very informative and we wanted to write up a quick post on some of the biggest things we learned.
Published below are our processes and research questions that we use as a guide when researching priority programs. We have made it a Google document so that interested individuals can comment on specific sections.
We are extremely interested in feedback on all elements of this document as it has a large influence on the first broad phase of our project. We have dramatically improved this document as we have progressed, in large part due to getting feedback from our advisors. We expect that it will be improved by further feedback as well. We wanted to publish a quick update of a summary timeline and broad phases of our project. All these numbers are tentative, and we expect they will change as we conduct the research and get a better sense of each step. Throughout this whole process there will also be ongoing logistical and meta-work to support these main activities.  There are many different ways and levels to help out at Charity Entrepreneurship. The higher the level of involvement, the more difficult it will be to apply for. Our strongest employees have often started at a low level of involvement and consistently demonstrated that they could handle higher levels of responsibility. For our highest level of involvement we would almost always require someone to start off at a lower level first to test out whether they are a good fit for our organization. The below table offers some guidelines for the different roles and expectations of each level of involvement. To get involved, please contact us.  |
|
RSS Feed
Charity Entrepreneurship is a project of Charity Science
