EXPERT VIEW
This is one of 10 documents on Charity Entrepreneurship’s research process.
A summary of the full process is here.
A summary of the full process is here.
SummaryIn this document, we explain why and how CE uses expert views (EpV) as part of its research process. Garnering expert views consists of speaking to experts who may have broad, domain-focused, or specific knowledge of the field. It is particularly useful because experts can often give a broad overview of a topic, allowing researchers to gain a comprehensive understanding of an idea. However, it is not the only method we rely on because human judgment often suffers from cognitive biases.
CE uses expert views at three stages of our research. At the first stage (idea sort), each intervention is assessed for twenty minutes based on all the methodologies, including expert views. At the second stage (prioritization report), two hours will be spent on each intervention using this method, but only for our family planning research area. Finally, expert views are one of the four methods used for the eighty-hour assessment of each of the top interventions (intervention report). Concretely, the lead researchers will use this method by holding one-hour interviews with broad experts at the initial stage, domain experts at the second stage, and a mix of experts with various levels of knowledge, including specific knowledge, for the last stage of the research. Table of contents:
1. Who are experts 2. Why is this a helpful methodology 3. Why they are not our only endline perspective 4. How much weight we give experts 5. Our process for speaking to experts 5.1. Selecting 5.2. Contacting 5.3. How to speak to experts 5.4. Recording 5.5. Reviewing as potential mentors 5.6. Process 5.7. Brief survey 5.7.1. Cause-neutral questions 5.7.2. Mental health questions 5.7.3. Animal questions 5.7.4. Family planning questions 5.8. Note-taking and summarizing 5.9. Synthesis 5.10. Expert review 6. Different levels of expert depth 6.1. Five minutes 6.2. Two hours 6.3. Twenty hours 6.4. External expert data 7. Summary 8. Deeper reading 1. Who are experts?Speaking to experts is a common way to gain a lot of information about a topic quickly. Experts can synthesize an informally large amount of knowledge into layman’s terms that are much easier to understand than a meta-analysis or other form of formal synthesis. When we speak about experts we are referring to three different groups of individuals:
Specialist experts will often be highly versed but in a very specific situation or content area. For example, a fish disease specialist would fall under specialist domain experts. They can provide a piece of the picture but often not a broad comparison. If your goal is to start a charity that helps the most fish, they would not be able to compare disease to transportation issues, and often specialist experts would not even offer a guess on it. However, they would be able to provide highly specific information about disease rates in a given species and situation that you have identified as promising. Domain experts are experts who have a sense of a single area. They might know about many different possible factors that affect a single type of fish but would not be able to compare a fish-based intervention to a chicken-based one. Heads of nonprofits in a given area would be good examples of domain experts. Broad experts can provide comparisons across different domains, for example, a funder who supports half a dozen different fish organizations might have a strong sense of how disease compares to transportation even if she does not have the same detailed sense of specific diseases as the specialist expert. We see all of these experts as very helpful but in very different situations. Example conversation notes (GiveWell and Charity Science Health) 2. Why is this a helpful methodology?Experts are in many ways the broadest source of information. They rarely give specific conclusions but a broad overview of a large field, covering a lot of ground and have views that are often easy to explain in terms of conclusion but hard to explain in terms of the factors that went into their formulation.
Reasons they are a helpful resource (in rough order of strength)
Utilizes a large number, and a variety, of evidence sources: Experts have formed views using a wide range of sources of information. They often form their perspectives based on a number of studies, conversations, personal experiences, and other sources. These diverse sources are combined into a single view. This has a number of advantages including making their conclusions more robust and grounded. Apply common sense filters: When you are new to a field you do not have a strong common sense filter; however, experts have often seen many projects come and go and have a strong sense of things that will be more successful or impactful. Experts often have a sense in their field of what things might go wrong in a project or lead it to failure. These filters can be helpful and informative in prioritization among areas and making long-term plans. Quickly assess weaknesses: As a result of being able to talk directly to an expert and lay out specific situations and combinations of ideas, it is easier to identify flaws in reasoning or possible areas in which an idea could fail. This information would often be hard to find from informal research or even deeper, more systematic research. Are insensitive to single-number model errors: One of the biggest concerns with many-multiplication models such as CEAs is that a single error such as a mistyped number can have a large effect on the model. Experts on the other hand are rarely overly affected by a single model or a single number and tend to be slow to update on shocking conclusions. They more intuitively and directly apply “extraordinary claims require extraordinary evidence” heuristics. Use new sources of information: Experts often have connections and knowledge about what resources are worth considering for further research or which other experts are worth talking to. This methodology, when applied, thus lends itself to finding more information and getting a clear path of who to talk to next or what resources to read. Experts sometimes have access to studies or other research that is not yet available or easily accessible in the public domain. Well-positioned experts can often have access to information a full year before it is publicly available. They also often have details on what studies are being worked on and will be completed in the near future. Are a respected source of information: Talking to and getting viewpoints from experts is a respected and even expected tool to use when researching an area. It is also common practice in many fields including charity evaluation. Offer multisession engagement: Experts are one of the few sources of information that can directly engage in back-and-forth discussion, which means that after speaking to them with a more basic idea, you can converse again about the changes or advancements that have happened in your thinking. Can directly compare possible strategies: Experts can be given highly specific plans and compare different elements of them much more quickly than a more formal model like a CEA can. For example, if you are considering three interventions in three different countries with three different partner organizations, the number of permutations quickly becomes overwhelming for a formal model. Experts, however, can compare multiple iterations and suggest which combination seems to have the highest impact or is the most promising to research further. Provide field-level convergence: Experts can give a sense of whether many individuals within a field have a fairly unified view on something (e.g., if all three experts you speak to agree on a topic) or if there is a variety of views on a topic (e.g., three experts give three different answers). If an area has a high level of convergence, it is good to get these conclusions, and if it does not, that leaves open more areas that should be considered or researched. 3. Why they are not our only endline perspectiveDespite experts being a helpful source of information, they are not our only endline perspective. When viewing an evidence table, there is a reason why EpV is often near the bottom.
Experts have a number of weaknesses that have been demonstrated to negatively affect their judgment, and studies have shown in some areas, such as predicting the future, that “many of the experts did worse than random chance, and all of them did worse than simple algorithms.” These concerns limit experts’ usefulness and make us confident that they should not be the only perspective used. Many of our biggest concerns with experts are cognitive biases that cross-apply to the vast majority of human judgments. Not all the following concerns apply to every expert, but they are generalized concerns that will apply to a large number of experts.
Unequal application of rigor: A major concern with experts is equal application of rigor. Given all the information currently available, a motivated actor can find evidence supporting almost any viewpoint. Thus a fairly weak argument could hold a lot of weight in an expert’s view if he or she has not considered it skeptically or if it fits a prior worldview the expert holds. Similarly, if an expert does not like an idea, he or she finds it easy to be significantly more critical of it than would be justified compared to other ideas or viewpoints he or she holds. This rigor concern makes it highly challenging to take expert conclusions without a deeper sense of how they, for example, react to any new idea. Inconsistent and unclear epistemology: Another factor that makes expert judgment less strong a source of evidence is the relative rarity of a formal or consistent epistemic system. Experts often have views about how to weigh different types of evidence but few have thought about this problem explicitly, and very few have publicly laid out how they would compare and integrate different pieces of evidence into their endline viewpoints. Cognitive bias: There are a number of cognitive biases that affect humans. Experts are fundamentally just more informed humans and thus generally suffer from the same biases. Some evidence suggests that experts can be affected even more strongly by some biases than the general population. One mitigating factor is that if multiple experts are spoken to, their biases will not necessarily overlap, and their average quality of judgment tends to do better than that of a single expert. There are hundreds of biases that can affect judgment and decision-making, but some that seem particularly relevant to experts when considering charity ideas are: Anchoring: Is when an individual depends too heavily on an initial piece of information offered (considered to be the “anchor”) when making decisions (1). Experts can often anchor on a specific idea for a charity early in a conversation or before the conversation has even started. Many experts will have projects they have already supported or invested time into, and these existing projects will generally be compared to any new idea with a high level of comparative skepticism regarding competing or different ideas. Groupthink: Is a psychological phenomenon that occurs within a group of people in which the desire for harmony or conformity in the group results in an irrational or dysfunctional decision-making outcome. Group members try to minimize conflict and reach a consensus decision without critical evaluation of alternative viewpoints by actively suppressing dissenting viewpoints and by isolating themselves from outside influences (2). In the case of charity ideas, if an idea has not been previously tested or considered by experts in the field, often they will be more inclined to dismiss the idea than they would if the same concept was presented by someone connected to their ingroup. Although this is a useful heuristic for experts to use, it can make them underweight new ideas relative to more established ones, particularly if the new ideas are generated using an intelligent process (e.g., CEAs). Illusion of control: Is the tendency for people to overestimate their ability to control events; for example, it occurs when someone feels a sense of control over outcomes that he or she demonstrably does not influence (3). This connects closely to experts having difficulty discerning effects from randomness or noise. The way this connects to charity ideas is that often experts will put more weight on personal experiences they have had; for example, if idea A has worked in the past, idea A will always work, and if idea B failed in the past, idea B is likely to fail. These are often assumptions held without careful consideration of the environmental factors that are different or the nonresults-focused factors; for example, a higher chance of failure might be worth it if the win is several times larger. Lack of transparency in argument generation: Experts have formed their views using a considerable number of sources and experiences. A byproduct of this is the great difficulty of tracking down the basis for a given viewpoint. This can make it very challenging to confirm or disprove a given idea or even know how much weight it should be given. This is not the fault of the expert but is a flaw inherent in expert-based information. Memory concerns: Evidence has demonstrated that memory is a fallible tool, but generally when speaking to an expert there is a high level of reliance on the expert’s memory. A remembered version of a study or conversation could be significantly different than the original. It is also hard or impossible to detect these memory effects given the lack of transparency. Limited specificity: Many experts are unwilling to give specific estimates such as a percentage-based chance of success. Experts are often unwilling to make claims that could be used in other methodologies such as CEAs, particularly if those claims cannot be anonymized. Lack of decisiveness: Similar to the specificity concerns, experts are often unwilling to make decisive claims even when taking a neutral or unsure stance would have its own ramifications. This is often taught practice in academia and can be a good habit when it comes to truth-seeking, although it can impair comparison among different options. 4. How much weight we give expertsExperts are quite an important source of information, and a lot of fundamental information comes from EpV as interpreted by others. However, experts suffer from many human biases that more algorithmic systems are less affected by. Experts are one of our five perspectives, the others being: WFM, team intuition, CEA, and our prior view. We expect our experts to generally hold more than one-fifth weight but likely not more than one-third of the weight of the model. Given the considerable variation depending on the specific charity idea and cause area, we expect experts to be stronger in areas where it is harder to get solid numbers and commonsense intuitions can be used as an effective guide.
5. Our process for speaking to experts5.1. SelectingWho do we count as an expert, and how do we speak to experts? We have two main processes to generate expert names. Our list of experts is generally created opportunistically instead of systematically.
The types of experts we end up speaking to in practice fall into the following categories.
5.2. ContactingWhen contacting an expert for the first time, we generally use an email similar to the following.
Dear _____ _____ person recommended that I speak to you because of your background in _____ OR I am researching ______and I read your paper on _____ (which was very _____! (compliment, like interesting or well done). Based on this, I thought you might know the answers to a few questions about the topic. I am a research associate at an organization that researches and funds new nonprofits which put that research into action (our website here). Previously we worked mainly in global poverty and animal welfare, but we have expanded to mental health so we can make this issue a higher priority in the global sphere. We are based out of London and so far have work underway in Oxford, London, the US, India, and Asia. Some questions I have are: ____ and ____. Would you happen to have the time to jot down some quick answers to the above, or maybe discussing it via Skype might be easier? We’d really love to have your input and research inform our funding decisions and what charities to work on in the sphere. Best regards, ______ 5.3. How to speak to expertsExperts are ultimately just people like anyone else, so most standard conversational rules apply to them. A few elements to highlight are:
5.4. RecordingAt the start of the interview, we ask the expert if they are comfortable having the interview recorded. This means that we don’t have to take notes during the interview so we can entirely focus on the questions and their answers to them, and it also makes summarizing the interview easier because we can go back and listen to their answer to x question again, for example.
To be safe, we usually record in two different ways.
5.5. Reviewing as potential mentorsIt is critical to use the research team’s extensive interactions with researchers to identify additional mentors. At the same time, the additional effort for a CE researcher should be limited. Hence, the review of researchers as potential mentors should be quick and fully integrated into the existing process that outlines conversations with researchers. Otherwise, the likelihood of poor retention or poor data quality is high.
5.6. Process
5.7. Brief SurveyCE Expert Assessment
The survey is kept intentionally short to ensure consistent application. It will be automatically graded and return a suitability score. Questions to ask
5.7.1 Cause-neutral questionsA little bit about the project: I am surveying <cause area> experts to get a sense of what would be the best areas to research and launch charity startups in.
At the end:
5.7.2. Mental Health QuestionsCustomized cause area questions:
There are some broad areas of mental health we are considering. For each area, we consider two questions. 1) How effective the area seems generally (for starting new charities) (below average - average - above average - the best intervention) 2) What might be the most promising specific things to do within an area. For example, one of the broad areas is therapy, and one specific area that some people think is promising is online apps for lower-income countries.
Customized person questions: These will be customized questions to ask a specific person. For example, a question to ask someone who mainly works in global health:
5.7.3. Animals QuestionsCustomized cause area questions:
There are some broad areas of animal advocacy we are considering. For each area we have two questions we are considering: 1) How effective does an area seem (for starting new charities)? 2) What might be the most promising interventions within an area? For example, within the area of food technology, one intervention that may be promising is lobbying governments to ensure fair labeling of plant- and cell-based products?
Customized person questions: These will be customized questions to ask a specific person. For example, a question to ask someone who works with funders/works for a fund:
5.7.4. FAMILY PLANNING QUESTIONSBroad questions
Comparative questions
Funding
5.8. Note-taking and summarizingFor every conversation, we ask the expert if we can take notes and share them, either named or anonymized as input from one of several experts we have interviewed on the topic. We offer to send them a copy of any notes we take so they can comment if they feel we misunderstood anything. We also offer to send them a copy of our full report if they are interested in seeing other experts’ views or other synthesized research we conduct on this topic. If they are interested, we would love their feedback on the full report.
Conversation notes are summarized into an easily readable document and then sent back to the expert for confirmation that we did not misunderstand or misrepresent anything, similar to the GiveWell conversation notes. In an experiment using Otter (an automatic transcription service) to record the conversation vs. manual conversation notes (making notes whilst listening to the audio), we found that the time taken for the automatic transcription to be edited into a readable form is slightly longer than the time taken to manually write up conversation notes. 5.9. SynthesisAlthough the bulk of the expert report will be the conversation notes, the project lead will have to synthesize these thoughts into a one-page easy-to-read summary. This can include both narrative explanation of concepts that came up multiple times in conversation notes or table-based data with rough quantification on what experts thought. An example of this can be seen here. The section above the “additional details” is more reflective of the detail that would be helpful to have for a specific idea expert synthesis.
5.10. Expert reviewOur final utilization of experts is when our full report is nearing conclusion. We send out our full report to any expert that indicated an interest in seeing the endline results and ask for any feedback they have.
6. Different levels of expert depth6.1. Five minutesProcess
Speaking to an expert for five minutes is not possible because even finding and contacting them would take close to this amount of time. However, one broad expert conversation can cover a lot of ground. If there are 300 interventions in a given cause area and each is given five minutes, this adds up to twenty-five hours of total time. Of this, seven hours would need to be used to find and contact experts, two hours to prepare the most important questions that would give helpful information across an intervention area, ten hours to interview the top five who respond, and finally eight hours to synthesize the notes from speaking to them and scoring interventions based on the responses. This results in around five minutes per intervention in a given area or about five experts for the area. Interviews at this level of depth will be used in our research agenda phase. We will contact five broad experts to help us narrow down our long list of ~300 <cause area> intervention ideas. For this level of depth, we will use an automatic transcription service (Otter seems best). We will send this transcription to the experts for review following our Skype call, explaining that this is an automatic transcription that has not been edited and will not be published. Expected outcome
Advice
Lessons learned The categorization of interventions into a broad category (which you will ask experts to rate) is a highly important part of this process that we should have put more thought into for three main reasons.
Potential solutions
What we think would be best to do in the future: Incorporate both a Google form survey and the Skype interview into the EpV (e.g., Skype with experts and ask open-ended questions to get a sense of their values etc., and after the Skype send them a Google form survey and get them to rank subcategories of interventions on there, explaining what sort of intervention would fall into these categories). 6.2. Two hoursProcess
Given the generalized information that has been gained from the five-minute process, the most helpful expert to speak to next would be a single domain expert. This interview would take one hour including prep, contacting the expert, and summarizing the notes afterward, as well as one hour speaking directly to the expert about the key questions that would be hard for a broader expert to answer. If an expert can cover more than one cause area, more time can be used to prep for his or her interview. Over a given cause area this would result in one expert per domain or five to thirty experts in total. This stage will only happen for family planning this year. In other cause areas, we will use a different two-hour methodology. Interviews at this level of depth will be used to help us understand a more specific charity idea such as what a promising country or approach might look like. These interviews, which will often be with domain experts, will help us determine, for example, what country would be most promising to run a conditional cash transfer (CCT) program for intrauterine devices (IUDs) in. For this level of depth, we will manually write up conversation notes (because these notes will be published, they need to be higher quality than for the other levels of depth). We will send this conversation summary to the experts for review before publishing. We will also invite these specialist experts to review the report as a whole. Expected outcome
Advice
6.3. Twenty hoursProcess
At the twenty-hour level, roughly the same process would be used as for the five-minute level. Of this time, two hours would need to be used to find and contact experts, as well as two hours preparing the most important questions that would give helpful information across an intervention area, six hours interviewing the top five that respond, four hours synthesizing the notes from speaking to them and scoring interventions based on the responses, and six hours sending the results to the experts and getting their feedback on both their personal notes and the report as a whole. This would result in five experts spoken to about a single intervention. These expert interviews will often be with specialist experts, such as speaking to an animal advocate in Taiwan or speaking to a fish disease specialist to help us determine whether paying farmers to use vaccines to treat diseased tilapia in Taiwan seems like a promising intervention for farmed fish. One of the experts contacted will be a person leading an organization implementing similar interventions in the same region as the recommended charity. The goal of this interview is to find out if they could be influenced to change their program to a more cost-effective one (recommended by CE). Such a change would alter where we plan to allocate resources. For example, influencing a charity fortifying flour with iron to add folic acid would alter the score of a folic acid fortification intervention, and would lead to starting a tobacco taxation charity instead. For this level of depth, we will manually write up conversation notes (because these notes will be published, they need to be higher quality than for the other levels of depth). We will send this conversation summary to the experts for review before publishing. We will also invite these specialist experts to review the report as a whole. Timeline relative to other methods (eighty-hour report)
Expected outcome
Advice
6.4. External expert dataExperts we interview are not the only source of expert data we use. If there are previously written interviews, conversation notes, or other direct sources of expert data, we also include these in our expert report. Data like these would be searched for in the directed research phase of the project but would be included in the evidence section of data.
7. SUMMARYBy the time a charity is recommended in an area, we will have spoken to five broad experts in the cause area, as well as six domain experts or specialist experts. Some of these experts will also have reviewed the overall report and given comments or suggested improvements. We also will have taken into account any publicly available expert surveys or summaries of other related conversation notes.
These experts are spoken to using a consistent methodology. Our conversation notes, as well as our summarized interpretation of the conversations, are published in a single section but clearly differentiated from each other. 8. Deeper reading1) External resources on how to be a good interviewer
2) Evidence on good forecasting practices from the Good Judgment Project: An accompanying blog post 3) The Black Swan: The Impact of the Highly Improbable 4) Expert Political Judgment: How Good Is It? How Can We Know? 5) Future Babble: Why Expert Predictions Fail and Why We Believe Them Anyway 6) Scientists can sense which studies are weak |
About UsCharity Entrepreneurship (CE) is a project of Charity Science Foundation of Canada, a foundation registered in Canada (charity number 80963 6236 RR0001). CE supports its incubated charities through a fiscal sponsorship with Players Philanthropy Fund (Federal Tax ID: 27-6601178), a Maryland charitable trust with federal tax-exempt status as a public charity under Section 501(c)(3) of the Internal Revenue Code. Contributions to CE are tax-deductible to the fullest extent of the law.
Privacy Policy: You can read our Privacy Policy here Terms of Use: You can read our Terms of Use here |
ConnectContact usPlease use our contact form.
|