Hi, I am Kieran. I joined the Charity Science team in June 2015 after volunteering since the previous February. Hope you're enjoying the new website. Charity Science recently evaluated more than 20 different fundraising methods to determine which methods would be the most promising to experiment with in the third quarter of 2015. This prioritization task resulted in slightly more than a hundred pages of research about fundraising methods, which will be helpful for other groups considering a wide range of fundraising options. When researching, reviewing and evaluating these methods, something that persistently proved useful was the cluster thinking approach. In essence, cluster thinking involves approaching empirical questions from numerous reference frames or mental models and synthesising this cluster of views into one’s opinion.
We attempted to apply this approach when researching fundraising methods by asking researchers to look for and explain reasoning from multiple angles. These angles included but were not limited to:
Hard research numbers or data
An expected value calculation
What other charities/experts generally think of this method
The extent that other charities use this strategy, and how effective Charity Science will be at competing with them
The soft sense the researcher had for this fundraising technique
It was not always possible to represent each angle and in different reports the same angle was sometimes weighted quite differently. For instance, in one report the expected value calculation may be based off pure estimates, whereas in another they may be shaped by several sources of information. In addition to having researchers consider a number of different outlooks when completing their research, they themselves also represented a set of slightly different perspectives. This stemmed from their differing levels of involvement with Charity Science, experience in fundraising, and familiarity with research. However, that didn’t translate into all points proffered being equal and the conclusion reached just an average of them. This wasn’t an exercise in pure “philosophical majoritanism.” Yet nor was a view completely ignored in this decision making process either. One of the reasons we encouraged research informed by cluster thinking approach was because, as we have previously written, there is no good science on fundraising. Further, there appear to be systemic issues with the available information as it tends to be informal and anecdotal in nature, seemingly pervaded by publication bias, and rarely acknowledges, let alone attempts, to answer questions of cross applicability… but these are issues for another time. What matters here is that robust expected value estimates couldn’t be made. Instead, constructed expected value estimates were often heavily contaminated with uncertainty. For instance, 95% confidence intervals could span orders of magnitude without even accounting for Knightian uncertainty. This didn’t prevent us from completing expected value estimates during the research, rather, it caused some researchers and all reviewers to adjust the epistemic weight assigned to them. Within the cluster thinking approach, this meant that expected value estimates were just one line of reasoning that was integrated into our overall conclusions about a fundraising method. For a related piece that goes into more depth about Bayesian updating in light of uncertainty in making expected value calculations, see why expected value estimates can’t be taken in literally even when they are unbiased. An advantage of treating expected value estimates in this way is that it prevents one poorly constructed or mighty powerful link in the chain from dominating our concerns. For instance, even though some fundraising methods possesses a chance for enticingly high potential returns, namely High Net Worth Individual acquisition and their stewardship, this wasn’t enough to swamp all other considerations. Similarly, any potential remarkably poor counterfactual estimates involved in other expected value estimates will not derail a conclusion that draws on many weak arguments instead of one relatively strong one. After incorporating elements of cluster thinking at the individual research level and at the individual review stage, we also sought to apply it in the final evaluation stage. To do this, all Charity Science staff and some board members were invited to individually evaluate each fundraising technique. We asked that each individual didn’t express their preferences as this would likely influence the ratings of others. After all who were involved had reached their independent conclusions, we then compared divergent items and explored areas of disagreement. We were aware of the distinction between fox and hedgehog style thinking throughout this process and attempted to emulate the fox type, because evidence suggests its predictions are more accurate. From start to finish, this process went for ~2 months and wasn’t consistently worked upon. Our conclusion was that the fundraising methods most promising were legacy fundraising, niche marketing, and online advertising. In future posts, we’ll likely be writing more about each method.