OUR RESEARCH PROCESS 2023-2024
This is a summary of our general research process for finding cost-effective interventions. The process inevitably varies across different cause areas that we evaluate. This version has minor updates from the 2021-22 version available here.
This process is also fluid and ever-changing; we test each stage as we go and adapt accordingly.
OUR GOALS
Our primary goal is to identify the most high-impact intervention opportunities.
Our secondary goals are:
Providing support to future incubatees through high-quality research reports and implementation advice.
Seeking additional wins (supporting other organizations, keeping track of high-impact interventions, identifying potential program participants).
Ongoing improvement of the research process.
OUR DECISION CRITERIA
We determine policy interventions to be particularly high-impact based on the following 5 targets (applied in priority order):
1. Substantially better than our bar to beat – E.g., for global health policy we set a bar of 5x more cost-effective than AMF and other top GiveWell evaluated charities.
2. Evidence quality – Evidence should be high quality and robust to uncertainty.
3. Limiting factors – There should be minimal ways that a new charity could fail.
4. Variation – The overall set of opportunities identified should vary in type.
5. Maximally impactful – Highest expected value.
THE PROCESS
Our research process comprises seven stages, which were preceded by strategic process design. We time-cap each stage to produce two-three high-quality recommendations that will be implemented through our yearly Incubation Program.
STAGE 0: PROCESS DESIGN AND TOPIC LEARNING
Goal 1: Designing the decision-making process for the cause area.
Time spent: ~30 Hours
Implementation:
-
Defining crucial parameters for decision-making
-
Research into available metrics
-
Creating/updating templates for each step and method
-
Establishing a benchmark idea to compare against
Goal 2: Gaining background knowledge
Time spent: ~30 hours per person
Implementation:
-
Broad reading and research discussed with the team
-
Cross-applicable research presented to the team
STAGE 1: GENERATING IDEAS
Goal: Creating a list of ~150-400 intervention ideas in our target cause area
Time spent: ~ 50 Hours
Implementation:
-
Generating ideas based on:
-
Contacting experts
-
Contacting key organizations
-
Broad reading and cross-applicable research
-
CE Team’s input
-
-
Put ideas into key categories
-
Remove any obviously low-quality ideas
STAGE 2: QUICK PRIORITIZATION
Goal: Narrowing down ideas list to 25% (~50) most promising interventions
Time spent: ~120 Hours
Implementation:
-
Researching top considerations that might eliminate sets of ideas
-
Organizing ideas (e.g., merging similar ideas)
-
First independent rating, taking 2-12 minutes (Researcher 1)
-
Second independent rating, taking 2-12 minutes (Researcher 2)
-
Additional research
-
​​Review meeting and final decision-making
STAGE 3: SORTING IDEAS
Goal: Narrowing down ideas list to 25% (~15) most promising interventions
Time spent: ~130 Hours
Implementation:
-
Assessing strength of evidence for the intervention (40 min. per idea)
-
Looking for evidence (e.g., RCTs, recommendations from effectiveness-focused organizations, macro/country-level data on effects, consensus among experts, existing effective implementations)
-
Assessing quality of evidence
-
Cutting 20% of least promising ideas
-
-
Paths to failure weighted factor model (15 min per idea)
-
Each team member individually scores each idea on a number of factors related to how likely the idea is to fail to make a successful charity
-
-
Quantitative modeling (30 min. per idea)
-
CEA/BCR estimates
-
Comparing to the benchmark interventions
-
STAGE 4: EXPERT INTERVIEW AND CRITICAL UNCERTAINTY ANALYSIS
Goal: Arriving at a ranked list of the top ~10 ideas for the most complex research stage
Time spent: ~150 Hours
Implementation:
-
At least 1 expert interview on each idea
-
Identify 1-6 crucial considerations or cases against each idea
-
Spend 1.5h evaluating each of the identified crucial considerations.
STAGE 5: DEEP DIVE
Goal: Selecting top 2-4 charity ideas that we recommend to be started through our 2022 Incubation Program, by producing high-quality research reports on the top 5-10 ideas.
Time spent: 700-900 Hours
Implementation:
-
Background reading
-
Mapping a causal chain of the intervention and theories of change
-
Geographic assessment weighted factor model
-
Stakeholder mapping
-
Interviewing experts
-
Quality of evidence assessment (academic evidence, case studies, historical data, macro-level data trends, theoretical evidence, assessing the strength of evidence for each step in the causal chain)
-
Assessing the potential for a new charity (neglectedness, limiting factors, ease of implementability, scalability)
-
Cost-effectiveness analysis
-
Externalities/crucial considerations
-
Expert review
-
Final decision meeting
STAGE 6: WRITE-UP, IMPLEMENTATION REPORTS AND FEEDBACK
Goal: Creating polished reports and 2-3 implementation notes on the most promising interventions in order to get new charities up and running.
Time spent: ~5-30 Hours
Implementation:
-
Work with the outreach team to polish the reports ready for publication
-
Write up additional thoughts and reading lists and contacts lists and any additional research that could be used to inform new charity founder working on the top ideas.
-
Give feedback on and improve the research process.
If you are interested in seeing a longer, internal, unpolished version of the process, please contact Morgan at morgan@charityentrepreneurship.com​