Our Website Backgrounds (2).jpg

INFORMED CONSIDERATION

This is one of the documents on Charity Entrepreneurship’s 2019/2020 research process. ​A summary of the full process is here.

SUMMARY

In this document, we explain why and how CE uses informed consideration (IC) as part of its research process. IC consists of using a nonpredetermined and nondirected research process to evaluate an idea, which allows taking into account forms of evidence and considerations that may be neglected by other methodologies. It is particularly useful because some lines of reasoning are more theoretical or subjective in nature but still make up crucial considerations that could make a charity idea worth pursuing or not. However, it is not the only method we rely on because its output is generally formed by a single individual, which can significantly bias the results. 

CE uses IC at three stages of our research. At the first stage (idea sort), each intervention is assessed for twenty minutes based on all the methodologies, including IC. At the second stage (prioritization report), two hours will be spent on each intervention using this method, but only for our health policy research area. Finally, IC is one of the four methods used for the eighty-hour assessment of each of the top interventions (intervention report). Concretely, lead researchers will use this method by doing broad research, trying to answer crucial considerations or intuitive questions related to factors that might dramatically affect the potential for an idea. Then, they are asked to assign an intuitive estimate of how promising a charity idea is.

Table of contents:

1. What is informed consideration
2. Why is it helpful
3. Why it is not our only or endline perspective
4. How much weight we give to informed consideration
5. How CE generates informed consideration
6. Different lengths of informed consideration-based estimates
7. Summary
8. Deeper reading

 

1. WHAT IS INFORMED CONSIDERATION?

 

IC is the closest methodology we use to common sense, a priori reasoning, or intuition-based methodologies. It involves using a nonpredetermined research process and takes into account soft forms of evidence and information that do not fit into a cost-effectiveness analysis (CEA), expert view (EpV), or weighted factor model (WFM). IC is generated by the lead researcher on a given report, and questions are asked that are more philosophical and broad in scope. It also allows space for undirected research that is not systematic in the same way that a WFM would suggest. It is described in a narrative form to help the reader get a sense of the information and intuition that the research team has on the issue after conducting substantial research.

Informed consideration example (Charity Entrepreneurship - 2019)

informedcns_orig.png

2. WHY IS THIS A HELPFUL METHODOLOGY?

 

Many weak arguments can often be stronger than a single strong argument. Sometimes an intuition can differ substantially from a calculated estimate even when both are based on the same information at the outset. Some lines of reasoning are more theoretical or subjective in nature but still make for crucial considerations that make a charity idea worth pursuing or not. 

Reasons this is a helpful tool (in order of strength) 

  • Information that does not fit into other systems

  • Information that would otherwise be lost

  • Divergent conclusions 

  • Many perspectives 

  • Multiple references classes

  • Speed

  • Flexibility across ideas

  • Transparency of existing intuitions and heuristics 

  • Cross-domain knowledge is generated 



Captures information that does not fit into other systems: The biggest advantage of IC is that they take into account large pieces of information that are not easily captured by other systems. For example, maybe we know that a nonexpert group has a very strong view on an intervention, or there are some wisdom of the crowds or Chesterton’s fence arguments in favor of an intervention. As formal models, including ours, do not offer a clear way to include this sort of information even to a small degree, IC gives both time and space to outline this sort of evidence. 

Captures information that would otherwise be lost: Some information might fit into a different system but might not have been clearly captured. For example, if you have several quick conversations at a conference, you might get a sense of a given issue but not know exactly which individual or combination of individuals this sense came from. Information like this can easily get lost when creating deep formalized reports, but it is still important data and can lead to differences in researcher-intuition- vs. model-based results.  

Allows space for more divergent conclusions: Many of the most valuable research contributions have come from divergent or original thinking. Detailed models do not often encourage divergent conclusions or thinking when applied to an intervention or issue. For example, IC would more easily spot if an intervention makes sense only when paired with a different intervention instead of being performed in isolation.  

Takes into account many perspectives: Because IC includes both information that does not fit into other systems and information that would otherwise be lost, it ends up taking a large number of data points into account.

Takes into account multiple reference classes: One of the largest challenges in research is picking an appropriate reference class to compare the current intervention to. Often when comparing something it’s easy to get stuck considering a single or limited number of reference classes, whereas softer comparisons often use a broader range of classes.

Is very quick: IC is a much faster system to use to go deeper on a specific issue. When there is not a prespecified question list or methodology, a researcher with the same amount of time can more quickly read and synthesize information than if they are seeking specific answers to specific questions. It is also one of the quickest ways to get a broad overview of a topic. 

Has a high degree of flexibility across ideas: Some ideas require more time to be put into certain aspects compared to others. For example, an area that affects the total human population might require more philosophical consideration than a simpler intervention that merely makes already existing people less sick. Thanks to IC’s lower level of structure, it can be used to direct more time toward idea-specific concerns.

Formalizes already existing intuitions/heuristics: One goal of written IC is to make more transparent the intuitions and heuristics that are used. Almost every important evaluative decision relies on intuitions or best guesses at key points, but having these considerations discussed and elaborated on makes it clearer where they come from, compared to a system such as a CEA where the full explanation might be “best guess.” 

Cross-domain knowledge is generated: Many times, hours used during IC are used for cross-applicable questions that might not be the most important questions for any single charity idea but are important for a wide range of charity ideas. 

 

3. WHY IT IS NOT OUR ONLY OR ENDLINE PERSPECTIVE

 

IC has a lot of breadth-based advantages; however, it is far from our only source of information and compared to the four other perspectives holds relatively minor weight in our endline perspective. Many of the same concerns that exist for experts also apply to IC but on a more dramatic scale. IC is generally formed by a single individual, making standard human biases a substantial concern. 
 

  • Nonsystematicity 

  • Lack of pre-committed methodology

  • Nonnumerical 

  • Dependence on researcher strength

  • Cognitive bias

  • Overweighting personal experience 

  • Intractable issues

  • Subjective individual judgment call 

  • Counting the same information multiple times​


Nonsystematicity: One of the biggest concerns with IC is that it is nonsystematic. Given the huge amount of data, conflicting experts, and possible angles from which to consider a complex issue, it is often easy for a researcher to find a long list of citations broadly supporting his or her views even if the views are wrong. IC is particularly susceptible to this concern. 

Lack of pre-committed methodology: The best studies conducted in science generally have a pre-analysis plan. There are many resources on why this is a good policy and improves research quality. Some forms of research have clearer ways to pre-commit to certain methodologies or sources. IC offers much more limited pathways than this when compared to more structured sets of questions for research used in the WFM.

Nonnumerical: IC is generally soft and nonnumerical, which can easily lead to factors holding inappropriate amounts of weight based on salience and general human weaknesses with numbers. It also makes many of the assumptions less clear because terms are softer such as “it seems probable,” which does not give the reader a clear sense of the percentage-based update or confidence of the claim. 

Cognitive bias: IC has considerable likelihood of cognitive biases to the point of them being almost guaranteed. Some of these mental heuristics reflect useful but hard-to-trace-back knowledge; however, it is well documented that biases can consistently negatively affect decision-making in a wide range of ways, and IC is the most vulnerable system we use in this respect. 

Dependence on researcher strength: IC leaves the bulk of the time distribution and research decisions to the individual researcher. This could lead to considerable quality differences between reports and researchers, making comparative results less useful. For example, a generally skeptical researcher might focus on flaws with these hours, and a more optimistic one might focus on implementation strategies. This could lead a reader or researcher to have an unreflective conclusion of what would have happened if a different research lead had been running the project.  

Overweighting personal experience: Overweighting personal experience is a cognitive bias that is likely more severe for IC than for EpV. If someone has tried an idea or methodology and it did not work for them in the past, this is extremely likely to affect their judgment of similar ideas even when it should not (e.g., if you assume your odds of success are ten percent, and you tried an activity and it did not work, your update should be almost nothing rationally but generally will be substantial intuition-wise).  

Intractable issues: Given this methodology’s focus on theoretical issues, it would be easy for a considerable amount of time to be spent on an issue that is not tractable. Some issues will be important but will eventually fall outside of the scope of an eighty-hour review. In addition, many issues that will be considered in this section cannot be moved forward even with several years of work. 

Subjective individual judgment call: A final weakness of this methodology is that it leaves a lot more up to subjective judgment calls. This means that if your values or epistemology are largely different from those of our lead researcher, the results will be less cross applicable than the expert interviews or CEAs would be. It is also not easy to, for example, change a single input and see the difference in results in the same way as with the CEA or weighted quantitative models. 

 

4. HOW MUCH WEIGHT WE GIVE TO INFORMED CONSIDERATION

We see IC as an important perspective but are also keenly aware of its many documented weaknesses. It is one of our four methods, but we expect it to hold between one-fifth and one-tenth weighting, with considerable variation depending on the specific charity idea and cause area. This is far less than CEAs, EpV, or WFM, beating only prior views (our fifth perspective). We expected IC to be comparatively stronger in areas where there are many philosophical considerations with an unclear consensus on the answers. ​

 

5. HOW CE GENERATES INFORMED CONSIDERATION

 

IC has quite a few unique elements compared to our other systems. ​

5.1. BROAD RESEARCH

 

In theory: We have written about the use of broad research when understanding a cause area in the past. Broad research in this context is a similar concept but applied to a more specific intervention. For a first sense of an intervention, it is easy to rely on a few sources or types of sources for the bulk of your information. For example, it would be easy to read many studies and books written by academics but gather limited information from field workers who often have a very different perspective. The point of broad research is to, somewhat slowly, get a sense of the area from multiple angles. 

In practice: This involves watching related documentaries, reading related books, going through overview sources such as Wikipedia, and trying to look at the issue from as many different introductory perspectives as possible. The exact way that hours are used in broad research will vary depending on the cause area and where the most information can be found but often involves many different methods and sources.

Example: If one were conducting broad research on malaria-preventing bed nets, one might read a book on the historical progress of the fight against malaria; watch a short documentary on a specific village that is combating malaria; read the Wikipedia pages on malaria, bed nets, and other related interventions; read blog posts from half a dozen different authors writing on the topic, and listen to a podcast interviewing a bed net producer. 

Questions to consider: The main questions to consider in the broad research phase include:

  • Have I looked at all the related summary material that is available on evaluation websites, CE’s website, or cross-cutting organizations?

  • Have I considered at least four different clusters with different views on this issue and read some resources from each?

  • Have I reviewed information from a number of different media (e.g., studies, news articles, Wikipedia, documentaries, interviews)?

 

5.2. UNDIRECTED RESEARCH

 

In theory: When IC is compared to our other methodologies, it is clear that it is the least directed. The research questions are far less detailed, and much is left up to the researcher’s interpretation of what to prioritize and how many hours to spend in each area. This reduces the systematicity of the intervention but opens up the ability to spend more hours in key areas that were unresolved from the more systematic research conducted. 

In practice: This means there are a few guiding questions for the IC write-up and an expected consistent end product. However, the specific sources and hour spread will vary considerably between cause and interventions, and cross-area uniformity is far less expected.

Example: During the broad research phase, a key consideration is found; it seems this single consideration could determine whether the intervention is impactful or not. The researcher decides to spend the bulk of his or her time deep-diving this consideration because it seems both trackable and pivotal to an eventual charity recommendation in the area. 

Questions to consider:

  • What are the factors least likely to be considered deeply enough by our other methodologies?

  • What types of research have been leading to the best progress on this charity idea?

  • What issues do I feel like our team needs to consider more?

 

5.3. CRUCIAL CONSIDERATIONS

 

In theory: Crucial considerations are considerations that could significantly affect an intervention. In some cases a consideration will be both important and cross-cutting throughout many charity ideas and thus will be worth an independent supporting report. However, in many cases an issue will be worth more consideration than our systems give it but less than a full independent report.

In practice: When researching an intervention, a consideration that is very important to the conclusion will present itself that will require dedicated hours. Often it will be worth trying to brainstorm multiple possible crucial considerations and determining an hour split between them (depending on their importance and expected progress on them) before diving into one.

Example: When considering an online cognitive-behavioral therapy (CBT) application, one of the biggest questions might be whether online therapy has evidence of working and, if so, in what contexts. This issue is likely large enough to be worth spending some unstructured research hours on but not large enough to warrant a separate supporting report.

Questions to consider:

  • What are the top crucial considerations in this area?

  • How likely am I to make progress on this consideration given the hours I have?

  • Does this affect enough charity ideas to be worth a separate supporting report?

 

5.4. THEORETICAL CONSIDERATION

 

In theory: Certain questions require more introspective or philosophical consideration to make progress on. Writing down your thoughts can be a way to mediate this sort of thought in a more considered and transparent way. Chains of logic can sometimes lead to surprising conclusions when given room for consideration.

In practice: In practice, some questions will be ethical or philosophical in nature, and there will not be a clear answer. Many relevant questions will not have written resources to rely on, and the researcher will have to determine how to make progress on the issue.

Example: Is a smaller amount of harm spread across a larger number of individuals a permissible trade for a reduction of a large amount of harm spread across a smaller number? At what point do most people agree with the trade-offs, and at what point is there controversy? Could I conduct a quick survey to get a sense of these numbers?

Questions to consider:

  • What ethical views would change our conclusion on this?

  • Are there any jumps in logic or reasoning in our research?

  • What evidence could falsify or change my conclusion on this?

 

5.5. INTUITIVE CONSIDERATION

 

In theory: Often intuition can pick up on things that more formal systems might miss. It can be affected by anything from heuristics from other areas to subtle body language that an expert had but did not make it into the notes.  

In practice: We have intuitive feelings about everything that constantly affect our judgment; making these feelings and views explicit and trying to see if they are relevant information or not is an important part of the process. We can also flag for external reviewers what areas might be affected by intuition. 

Example: A certain number in the CEA result just seems far too high; it seems to affect and trivialize everything else. I should really research that key number more to make sure it holds up on a double check.

Questions to consider:

  • Does my intuition feel uncomfortable with any of the conclusions reached on this report? If so, why should I put more time into that spot?

  • Is there any area or question I am intuitively shying away from?

  • Could I take some of these conclusions further than I currently do?

 

6. DIFFERENT LENGTHS OF INFORMED CONSIDERATION-BASED ESTIMATES

 

6.1. FIVE MINUTES

Five minutes is not enough time to do even the smallest amount of broad research for each charity idea; however, it is enough time to do a minor amount of broad research on the cause area as a whole and then make an intuitive estimate on how promising a charity idea is. For example, if there are 300 ideas in a space, twenty hours could be spent doing unstructured broad reading in the cause area. You do not need to take polished notes on this research because the only published product will be the list of ideas. Afterward, work through the list spending one minute on each idea. For each idea ask yourself a question—“What is the probability that we will recommend this as a charity idea?”—and rate it based on that. 

Broad research questions to consider:

  • Has CE written any reports in related areas? 

  • Have any charity evaluators (GiveWell, Animal Charity Evaluators, Copenhagen Consensus Center), cross-cutting organizations (Rethink Priorities, Wikipedia, World Health Organization, Abdul Latif Jameel Poverty Action Lab (J-PAL), Institute for Health Metrics and Evaluation), systematic reviews and meta-analysis organizations (Cochrane), or think tanks (Center For Global Development, The International Initiative for Impact Evaluation (3ie), Department for International Development) written any overviews of the topic? 

  • Who is considered the respected source in the area? What are they focusing on, and what have they written on the topic?

  • It is up to the researcher’s discretion what questions are going to bring most value to inform their judgment. 

 Expected outcomes

  • A score out of ten for each of the ideas

Advice:

  • Before rating the idea, do a calibration exercise 

  • Organize your twenty hours beforehand to make sure you spend enough time on important considerations. Spend time listing your biggest questions and try to make sure you keep enough time for each. An example of such an exercise can be found below. Process:

1. Set up the time cap
2. Compile list of possible sources
3. Guess how useful they might be on a scale of zero to ten
4. Estimate how long it might take to read each source
5. Order by expected usefulness and sum the expected time to decide which sources to read (best ratio of time to usefulness)
6. Read all of those sources within the time cap

6.2. TWO HOURS

 

In the two-hour model, the researcher has enough time to do unstructured research on the given charity idea. Some of this time will be spent finding sources—ideally overview sources—and some of the time on reading and absorbing them. This knowledge will build upon the twenty hours of cause area knowledge that has already been acquired. The bulk of this time will be spent on unstructured research with the last roughly thirty minutes spent on contemplation and written description (one paragraph) of why an idea is ranked more or less promising. This stage will only happen for one of the four cause areas we are researching. 

Questions to consider:

  • What factor might rule out this intervention quickly?

  • Are there any overview books/deeper write-ups that might summarize the area (e.g., a section of a textbook)? 

  • Are there specific write-ups or blogs that focus on this area? Can you read a sample of them to get a sense of the broader space?


Expected outcomes

  • One paragraph on why the idea was ranked more or less promising

 

6.3. TWENTY HOURS

IC happens at two different stages in our research process; the first is right at the start to attain a deeper view of both the broad research and the key crucial considerations that might dramatically affect the intervention. This will take about ten hours of the IC process and will result in a one-to-five-page summary of the critical considerations, mitigations, and overall feel of the intervention. The second set of ten hours will be spent near the end of the research report, allowing the researcher to contemplate all the data they have obtained from building a CEA, conducting systematic research, and talking to experts. Part of this process will be writing summaries and updating learnings from all the other areas as well as giving some time to introspective thinking and theoretical considerations that need further work. The expected output is also one to five pages of writing about different considerations and contemplations. These can be less structured and less cited than other parts of the full report and should include intuition and gut checks on various assumptions, even if it is unclear where the intuition comes from. This is also the chance to include points that might not be clear from the other models but would be clear if someone conducted the full scope of research themselves. 

Questions to consider

  • What would every person in this field have read or heard about?

  • Consider if there are any crucial considerations or other factors that could lead to elimination of the idea. Steelman back and forth to see if the elimination criteria are strong enough. 

  • What are the predominant considerations against doing this intervention?

  • What are the predominant considerations for doing this intervention?

  • How does it seem from the reading you have done, past experience, and people you have talked to?

  • What do we not know?

    • What further research should be done if we were to go deeper with this approach?

    • What are the open or remaining questions on the issue?

    • How much more research might it take (if any) to feel confident? 

  • What do experts seem to broadly think of this area?

  • How does this intervention broadly compare to the other intervention areas?

  • 1. Imagine it’s a year from now (the furthest possible future), and we have tried this intervention and it has failed. 2. Write down why you think it failed (remember to consider both internal and external factors). 3. What actions could you take to prevent this hypothetical failure mode? 4. Imagine that you are applying them. 5. Iterate on steps 1–4: Imagine that even with your new failsafes, the plan still fails. Are you shocked? How much?  5. Do the same for a time horizon of five years. 


How IC compares to other methods used (timeline of an 80-hour report) 

  • Ten hours – Broad undirected reading and CC (IC)

  • Sixteen hours – Directed research (WFM)

  • Ten hours – Finding and talking to experts (EpV)

  • Twenty hours – CEA creation (CEA)  

  • Four hours – Directed research (WFM) 

  • Ten hours – Summary writing and internal contemplation (IC) 

  • Ten hours – Showing endline report to experts (EpV)


Expected outcomes

  • First section: two-to-five-page summary of crucial considerations considered and resulting research

  • Second section: one to three pages of internal contemplation as well as the writing of the summary report ​

 

7. SUMMARY

 

By the time a charity is recommended, it will have been thoughtfully considered multiple times at different points in the process. Crucial conditions and theoretical questions will have been researched; broad and indirect research will have been carried out. In many ways this part of the process is most similar to how the bulk of research is conducted, with few formal systems set up beforehand and the lead researcher’s endline views carrying a lot of the conclusion’s weight. ​

 

8. DEEPER READING

 

1) When can you trust your gut
2) Sequence thinking vs cluster thinking
3) Importance of crucial considerations
4) Many weak arguments
5) Benefit of broad understanding
6) Sources of Power: How People Make Decisions
7) The Power of Intuition, and Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making
8) Thinking, Fast and Slow by Daniel Kahneman