INFORMED CONSIDERATION

2. WHY IS THIS A HELPFUL METHODOLOGY?

 

Many weak arguments can often be stronger than a single strong argument. Sometimes an intuition can differ substantially from a calculated estimate even when both are based on the same information at the outset. Some lines of reasoning are more theoretical or subjective in nature but still make for crucial considerations that make a charity idea worth pursuing or not. 

Reasons this is a helpful tool (in order of strength) 

• Information that does not fit into other systems

• Information that would otherwise be lost

• Divergent conclusions 

• Many perspectives 

• Multiple references classes

• Speed

• Flexibility across ideas

• Transparency of existing intuitions and heuristics 

• Cross-domain knowledge is generated 

Captures information that does not fit into other systems: The biggest advantage of IC is that they take into account large pieces of information that are not easily captured by other systems. For example, maybe we know that a nonexpert group has a very strong view on an intervention, or there are some wisdom of the crowds or Chesterton’s fence arguments in favor of an intervention. As formal models, including ours, do not offer a clear way to include this sort of information even to a small degree, IC gives both time and space to outline this sort of evidence. 

Captures information that would otherwise be lost: Some information might fit into a different system but might not have been clearly captured. For example, if you have several quick conversations at a conference, you might get a sense of a given issue but not know exactly which individual or combination of individuals this sense came from. Information like this can easily get lost when creating deep formalized reports, but it is still important data and can lead to differences in researcher-intuition- vs. model-based results.  

Allows space for more divergent conclusions: Many of the most valuable research contributions have come from divergent or original thinking. Detailed models do not often encourage divergent conclusions or thinking when applied to an intervention or issue. For example, IC would more easily spot if an intervention makes sense only when paired with a different intervention instead of being performed in isolation.  

Takes into account many perspectives: Because IC includes both information that does not fit into other systems and information that would otherwise be lost, it ends up taking a large number of data points into account.

Takes into account multiple reference classes: One of the largest challenges in research is picking an appropriate reference class to compare the current intervention to. Often when comparing something it’s easy to get stuck considering a single or limited number of reference classes, whereas softer comparisons often use a broader range of classes.

Is very quick: IC is a much faster system to use to go deeper on a specific issue. When there is not a prespecified question list or methodology, a researcher with the same amount of time can more quickly read and synthesize information than if they are seeking specific answers to specific questions. It is also one of the quickest ways to get a broad overview of a topic. 

Has a high degree of flexibility across ideas: Some ideas require more time to be put into certain aspects compared to others. For example, an area that affects the total human population might require more philosophical consideration than a simpler intervention that merely makes already existing people less sick. Thanks to IC’s lower level of structure, it can be used to direct more time toward idea-specific concerns.

Formalizes already existing intuitions/heuristics: One goal of written IC is to make more transparent the intuitions and heuristics that are used. Almost every important evaluative decision relies on intuitions or best guesses at key points, but having these considerations discussed and elaborated on makes it clearer where they come from, compared to a system such as a CEA where the full explanation might be “best guess.” 

Cross-domain knowledge is generated: Many times, hours used during IC are used for cross-applicable questions that might not be the most important questions for any single charity idea but are important for a wide range of charity ideas. 

 

3. WHY IT IS NOT OUR ONLY OR ENDLINE PERSPECTIVE

 

IC has a lot of breadth-based advantages; however, it is far from our only source of information and compared to the four other perspectives holds relatively minor weight in our endline perspective. Many of the same concerns that exist for experts also apply to IC but on a more dramatic scale. IC is generally formed by a single individual, making standard human biases a substantial concern. 
 

• Nonsystematicity 

• Lack of pre-committed methodology

• Nonnumerical 

• Dependence on researcher strength

• Cognitive bias

• Overweighting personal experience 

• Intractable issues

• Subjective individual judgment call 

• Counting the same information multiple times​

Nonsystematicity: One of the biggest concerns with IC is that it is nonsystematic. Given the huge amount of data, conflicting experts, and possible angles from which to consider a complex issue, it is often easy for a researcher to find a long list of citations broadly supporting his or her views even if the views are wrong. IC is particularly susceptible to this concern. 

Lack of pre-committed methodology: The best studies conducted in science generally have a pre-analysis plan. There are many resources on why this is a good policy and improves research quality. Some forms of research have clearer ways to pre-commit to certain methodologies or sources. IC offers much more limited pathways than this when compared to more structured sets of questions for research used in the WFM.

Nonnumerical: IC is generally soft and nonnumerical, which can easily lead to factors holding inappropriate amounts of weight based on salience and general human weaknesses with numbers. It also makes many of the assumptions less clear because terms are softer such as “it seems probable,” which does not give the reader a clear sense of the percentage-based update or confidence of the claim. 

Cognitive bias: IC has considerable likelihood of cognitive biases to the point of them being almost guaranteed. Some of these mental heuristics reflect useful but hard-to-trace-back knowledge; however, it is well documented that biases can consistently negatively affect decision-making in a wide range of ways, and IC is the most vulnerable system we use in this respect. 

Dependence on researcher strength: IC leaves the bulk of the time distribution and research decisions to the individual researcher. This could lead to considerable quality differences between reports and researchers, making comparative results less useful. For example, a generally skeptical researcher might focus on flaws with these hours, and a more optimistic one might focus on implementation strategies. This could lead a reader or researcher to have an unreflective conclusion of what would have happened if a different research lead had been running the project.  

Overweighting personal experience: Overweighting personal experience is a cognitive bias that is likely more severe for IC than for EpV. If someone has tried an idea or methodology and it did not work for them in the past, this is extremely likely to affect their judgment of similar ideas even when it should not (e.g., if you assume your odds of success are ten percent, and you tried an activity and it did not work, your update should be almost nothing rationally but generally will be substantial intuition-wise).  

Intractable issues: Given this methodology’s focus on theoretical issues, it would be easy for a considerable amount of time to be spent on an issue that is not tractable. Some issues will be important but will eventually fall outside of the scope of an eighty-hour review. In addition, many issues that will be considered in this section cannot be moved forward even with several years of work. 

Subjective individual judgment call: A final weakness of this methodology is that it leaves a lot more up to subjective judgment calls. This means that if your values or epistemology are largely different from those of our lead researcher, the results will be less cross applicable than the expert interviews or CEAs would be. It is also not easy to, for example, change a single input and see the difference in results in the same way as with the CEA or weighted quantitative models. 

 

4. HOW MUCH WEIGHT WE GIVE TO INFORMED CONSIDERATION

​

We see IC as an important perspective but are also keenly aware of its many documented weaknesses. It is one of our four methods, but we expect it to hold between one-fifth and one-tenth weighting, with considerable variation depending on the specific charity idea and cause area. This is far less than CEAs, EpV, or WFM, beating only prior views (our fifth perspective). We expected IC to be comparatively stronger in areas where there are many philosophical considerations with an unclear consensus on the answers. ​

 

5. HOW CE GENERATES INFORMED CONSIDERATION

 

IC has quite a few unique elements compared to our other systems. ​

​

5.1. BROAD RESEARCH

 

In theory: We have written about the use of broad research when understanding a cause area in the past. Broad research in this context is a similar concept but applied to a more specific intervention. For a first sense of an intervention, it is easy to rely on a few sources or types of sources for the bulk of your information. For example, it would be easy to read many studies and books written by academics but gather limited information from field workers who often have a very different perspective. The point of broad research is to, somewhat slowly, get a sense of the area from multiple angles. 

In practice: This involves watching related documentaries, reading related books, going through overview sources such as Wikipedia, and trying to look at the issue from as many different introductory perspectives as possible. The exact way that hours are used in broad research will vary depending on the cause area and where the most information can be found but often involves many different methods and sources.

Example: If one were conducting broad research on malaria-preventing bed nets, one might read a book on the historical progress of the fight against malaria; watch a short documentary on a specific village that is combating malaria; read the Wikipedia pages on malaria, bed nets, and other related interventions; read blog posts from half a dozen different authors writing on the topic, and listen to a podcast interviewing a bed net producer. 

Questions to consider: The main questions to consider in the broad research phase include:

• Have I looked at all the related summary material that is available on evaluation websites, CE’s website, or cross-cutting organizations?

• Have I considered at least four different clusters with different views on this issue and read some resources from each?

• Have I reviewed information from a number of different media (e.g., studies, news articles, Wikipedia, documentaries, interviews)?

 

5.2. UNDIRECTED RESEARCH

 

In theory: When IC is compared to our other methodologies, it is clear that it is the least directed. The research questions are far less detailed, and much is left up to the researcher’s interpretation of what to prioritize and how many hours to spend in each area. This reduces the systematicity of the intervention but opens up the ability to spend more hours in key areas that were unresolved from the more systematic research conducted. 

In practice: This means there are a few guiding questions for the IC write-up and an expected consistent end product. However, the specific sources and hour spread will vary considerably between cause and interventions, and cross-area uniformity is far less expected.

Example: During the broad research phase, a key consideration is found; it seems this single consideration could determine whether the intervention is impactful or not. The researcher decides to spend the bulk of his or her time deep-diving this consideration because it seems both trackable and pivotal to an eventual charity recommendation in the area. 

Questions to consider:

• What are the factors least likely to be considered deeply enough by our other methodologies?

• What types of research have been leading to the best progress on this charity idea?

• What issues do I feel like our team needs to consider more?

 

5.3. CRUCIAL CONSIDERATIONS

 

In theory: Crucial considerations are considerations that could significantly affect an intervention. In some cases a consideration will be both important and cross-cutting throughout many charity ideas and thus will be worth an independent supporting report. However, in many cases an issue will be worth more consideration than our systems give it but less than a full independent report.

In practice: When researching an intervention, a consideration that is very important to the conclusion will present itself that will require dedicated hours. Often it will be worth trying to brainstorm multiple possible crucial considerations and determining an hour split between them (depending on their importance and expected progress on them) before diving into one.

Example: When considering an online cognitive-behavioral therapy (CBT) application, one of the biggest questions might be whether online therapy has evidence of working and, if so, in what contexts. This issue is likely large enough to be worth spending some unstructured research hours on but not large enough to warrant a separate supporting report.

Questions to consider:

• What are the top crucial considerations in this area?

• How likely am I to make progress on this consideration given the hours I have?

• Does this affect enough charity ideas to be worth a separate supporting report?

 

5.4. THEORETICAL CONSIDERATION

 

In theory: Certain questions require more introspective or philosophical consideration to make progress on. Writing down your thoughts can be a way to mediate this sort of thought in a more considered and transparent way. Chains of logic can sometimes lead to surprising conclusions when given room for consideration.

In practice: In practice, some questions will be ethical or philosophical in nature, and there will not be a clear answer. Many relevant questions will not have written resources to rely on, and the researcher will have to determine how to make progress on the issue.

Example: Is a smaller amount of harm spread across a larger number of individuals a permissible trade for a reduction of a large amount of harm spread across a smaller number? At what point do most people agree with the trade-offs, and at what point is there controversy? Could I conduct a quick survey to get a sense of these numbers?

Questions to consider:

• What ethical views would change our conclusion on this?

• Are there any jumps in logic or reasoning in our research?

• What evidence could falsify or change my conclusion on this?

 

5.5. INTUITIVE CONSIDERATION

 

In theory: Often intuition can pick up on things that more formal systems might miss. It can be affected by anything from heuristics from other areas to subtle body language that an expert had but did not make it into the notes.  

In practice: We have intuitive feelings about everything that constantly affect our judgment; making these feelings and views explicit and trying to see if they are relevant information or not is an important part of the process. We can also flag for external reviewers what areas might be affected by intuition. 

Example: A certain number in the CEA result just seems far too high; it seems to affect and trivialize everything else. I should really research that key number more to make sure it holds up on a double check.

Questions to consider:

• Does my intuition feel uncomfortable with any of the conclusions reached on this report? If so, why should I put more time into that spot?

• Is there any area or question I am intuitively shying away from?

• Could I take some of these conclusions further than I currently do?

 

6. DIFFERENT LENGTHS OF INFORMED CONSIDERATION-BASED ESTIMATES

 

6.1. FIVE MINUTES

​

Five minutes is not enough time to do even the smallest amount of broad research for each charity idea; however, it is enough time to do a minor amount of broad research on the cause area as a whole and then make an intuitive estimate on how promising a charity idea is. For example, if there are 300 ideas in a space, twenty hours could be spent doing unstructured broad reading in the cause area. You do not need to take polished notes on this research because the only published product will be the list of ideas. Afterward, work through the list spending one minute on each idea. For each idea ask yourself a question—“What is the probability that we will recommend this as a charity idea?”—and rate it based on that. 

Broad research questions to consider:

• Has CE written any reports in related areas? 

• Have any charity evaluators (GiveWell, Animal Charity Evaluators, Copenhagen Consensus Center), cross-cutting organizations (Rethink Priorities, Wikipedia, World Health Organization, Abdul Latif Jameel Poverty Action Lab (J-PAL), Institute for Health Metrics and Evaluation), systematic reviews and meta-analysis organizations (Cochrane), or think tanks (Center For Global Development, The International Initiative for Impact Evaluation (3ie), Department for International Development) written any overviews of the topic? 

• Who is considered the respected source in the area? What are they focusing on, and what have they written on the topic?

• It is up to the researcher’s discretion what questions are going to bring most value to inform their judgment. 

 Expected outcomes

• A score out of ten for each of the ideas

Advice:

• Before rating the idea, do a calibration exercise 

• Organize your twenty hours beforehand to make sure you spend enough time on important considerations. Spend time listing your biggest questions and try to make sure you keep enough time for each. An example of such an exercise can be found below. Process:

1. Set up the time cap
2. Compile list of possible sources
3. Guess how useful they might be on a scale of zero to ten
4. Estimate how long it might take to read each source
5. Order by expected usefulness and sum the expected time to decide which sources to read (best ratio of time to usefulness)
6. Read all of those sources within the time cap