OUR RESEARCH PROCESS 2021-2022
This is a summary of our general research process for finding cost-effective interventions. The process inevitably varies across different cause areas that we evaluate. Cause areas we have researched recently using this process are: international development policy (2021), animal welfare (2021), health security (2022) and large-scale global health and development (2022).
This process is also fluid and ever changing, we test each stage as we go and adapt accordingly.
OUR GOALS
Our primary goal is to identify the most high-impact intervention opportunities.
Our secondary goals are:
Providing support to future incubatees through high-quality research reports and implementation advice.
Seeking additional wins (supporting other organizations, keeping track of high-impact interventions, identifying potential program participants).
Ongoing improvement of the research process.
OUR DECISION CRITERIA
We determine policy interventions to be particularly high-impact based on the following 5 targets (applied in priority order):
Substantially better than our bar to beat – E.g., for global health policy we set a bar of 5x more cost-effective than AMF and other top GiveWell evaluated charities.
Evidence quality – Evidence should be high quality and robust to uncertainty.
Limiting factors – There should be minimal ways that a new charity could fail.
Variation – The overall set of opportunities identified should vary in type.
Maximally impactful – Highest expected value.
THE PROCESS
Our research process comprises seven stages, which were preceded by strategic process design. We timecap each stage to produce two-three high-quality recommendations that will be implemented through our yearly Incubation Program.
STAGE 0: PROCESS DESIGN
Goal: Designing the decision-making process for the cause area.
Time spent: ~150 Hours
Implementation:
-
Defining crucial parameters for decision-making
-
Research into available metrics
-
Creating templates for each step and method
-
Establishing a benchmark of recommendation
-
Test run
STAGE 1: TOPIC LEARNING
Goal: Gaining background knowledge
Time spent: ~ 100 Hours
Implementation:
-
Broad reading and research discussed with the team
-
Cross-applicable research presented to the team
STAGE 2: GENERATING IDEAS
Goal: Creating a list of ~200-350 intervention ideas in our target cause area
Time spent: ~ 50 Hours
Implementation:
-
Generating ideas based on:
-
Contacting experts
-
Contacting key organizations
-
Broad reading and cross-applicable research
-
CE Team’s input
-
-
Put ideas into key categories
-
Remove any obviously low-quality ideas
STAGE 3: QUICK PRIORITIZATION
Goal: Narrowing down ideas list to 25% (~60) most promising interventions
Time spent: ~190 Hours
Implementation:
-
Researching top considerations that might eliminate sets of ideas
-
Organizing ideas (e.g., merging similar ideas)
-
First independent rating, taking 2-12 minutes (Researcher 1)
-
Second independent rating, taking 2-12 minutes (Researcher 2)
-
Additional research
-
Review meeting and final decision-making
STAGE 4: SORTING IDEAS
Goal: Narrowing down ideas list to 25% (~15) most promising interventions
Time spent: ~130 Hours
Implementation:
-
Assessing strength of evidence for the intervention (30 min. per idea)
-
Looking for evidence (e.g., RCTs, recommendations from effectiveness-focused organizations, macro/country-level data on effects, consensus among experts, existing effective implementations)
-
Assessing quality of evidence
-
Cutting 20% of least promising ideas
-
-
Identify implementability/execution difficulty and limiting factors (30 min. per idea)
-
Quantitative modeling (30 min. per idea)
-
CEA/BCR estimates
-
-
Comparing to the benchmark interventions
STAGE 5: SHALLOW INVESTIGATIONS
Goal: Arriving at ~5-10 top ideas for the most complex research stage
Time spent: ~150 Hours
Implementation:
-
5-10 hour research reports on each intervention
-
Expert interviews
-
Identify and assess crucial considerations
-
Look for evidence against intervention
STAGE 6: DEEP DIVE
Goal: Selecting top 2-4 charity ideas that we recommend to be started through our 2022 Incubation Program, by producing high-quality research reports on the top 5-10 ideas.
Time spent: 700-900 Hours
Implementation:
-
Background reading
-
Mapping a causal chain of the intervention and theories of change
-
Geographic assessment weighted factor model
-
Stakeholder mapping
-
Interviewing experts
-
Quality of evidence assessment (academic evidence, case studies, historical data, macro-level data trends, theoretical evidence, assessing the strength of evidence for each step in the causal chain)
-
Assessing the potential for a new charity (neglectedness, limiting factors, ease of implementability, scalability)
-
Cost-effectiveness analysis
-
Externalities/crucial considerations
-
Expert review
-
Final decision meeting
STAGE 7: IMPLEMENTATION REPORTS
Goal: Creating 2-3 implementation reports on the most promising interventions in order to get new charities up and running right after the program.
Time spent: ~5-30 Hours
Implementation:
-
Additional research used to inform a successful start and implementation by Incubation Program alumni.
If you are interested in seeing a longer, internal, unpolished version of the process, please contact Morgan at morgan@charityentrepreneurship.com