I have a tool for thinking I call “steelman solitaire” that I have found comes to much better conclusions than doing “free-style” thinking, so I thought I should share it with more people. In summary, it consists of arguing with yourself in the program Workflowy, alternating between writing a steelman of an argument, a steelman of a counter-argument, a steelman of a counter-counter-argument, etc. (I will explain steelmanning later in the post; in brief, it is the opposite of a strawman argument. Steelmanning means trying present the strongest possible version of an opposing view.) In this blog post I’ll first explain the broad steps, then list the benefits, and finally, go into more depth on how to do it.
The broad idea
Strawmanning means presenting the opposing view in the least charitable light, often so uncharitably that it does not resemble the view that the other side actually holds. The term of steelmanning was invented as a counter to this; it means taking the opposing view and trying to present it in its strongest form. This has sometimes been criticized because often the alternative belief proposed by a steelman also isn’t what the other people actually believe. For example, there’s a steelman argument that states that the reason organic food is good is because monopolies are generally bad and Monsanto having a monopoly on food could lead to disastrous consequences. This might indeed be a belief held by some people who are pro-organic, but a huge percentage of people are just falling prey to the naturalistic fallacy.
Nonetheless, while steelmanning may not be perfect for understanding people’s true reasons for believing propositions, it is very good for coming to more accurate beliefs yourself. If the reason you believe you don’t have to care about buying organic is because you believe that people only buy organic because of the naturalistic fallacy, you might be missing out on the fact that there’s a good reason for you to buy organic because you think monopolies on food are dangerous.
However, and this is where steelmanning back and forth comes in, what if buying organic doesn’t necessarily lead to breaking the monopoly? Maybe upon further investigation, Monsanto doesn’t have a monopoly? Or there are multiple organizations who have copyrighted different gene edits so there’s no true monopoly?
The idea behind steelmanning solitaire is to not stop at steelmanning the opposing view. It’s to steelman the counter-counter-argument as well. As has been said by more eloquent people than myself, you can’t consider an argument and counter-argument and consider yourself a virtuous rationalist. There are very long chains of counter^x arguments, and you want to consider the steelman of each of them. Don’t pick any side in advance. Just commit to trying to find the true answer.
This is all well and good in principle but can be challenging to keep it organized. This is where Workflowy comes in. Workflowy allows you to have counter-arguments nested under arguments, counter-counter-arguments nested under counter-arguments, and so forth. That way you can zoom in and out and focus on one particular line of reasoning, realize you’ve gone so deep you’ve lost the forest for the trees, zoom out, and realize what triggered the consideration in the first place. It also allows you to quickly look at the main arguments for and against. Here’s a worked example for a question.
Tips and tricks
That’s the broad-strokes explanation of the method. Below, I’ll list a few pointers that I follow, though please do experiment and tweak. This is by no means a final product.
Acknowledgements. I’d like to thank Spencer Greenberg for both inspiring the original idea with Clearer Thinking’s Belief Challenger tool and for coming up with a much better name for the concept than my original “steelmanning back and forth”.
Conducting research is essential for identifying high-impact interventions and assessing the effectiveness of existing strategies. Unfortunately, the historical impact of research within the animal welfare space has been limited by poor quality studies, a lack of experienced researchers, and organizations’ lack of responsiveness to findings. In this report, we consider various approaches to animal welfare research and their prospects of being translated into positive impact for animals.
This report identifies nine crucial considerations regarding the funding and talent landscape of the animal welfare movement. These considerations are used to evaluate 16 potential paths to using research to enhance the impact of the movement. The report analyzes the interrelationships between the actions of researchers and NGOs and summarizes some historical evidence regarding the influence of research on the effectiveness of activist movements.
Of the approaches considered, we think that research that informs existing corporate campaigns is among the highest impact paths for research. For example: should activists prioritize demanding that corporations keep chickens cage-free, slaughter chickens less painfully, or nutritionally fortify their chicken feed? Research that helps corporate campaigners identify which of these asks makes a bigger difference to the welfare of the animals could double or triple the impact of the considerable resources invested in these campaigns.
Another important finding was the importance of a research agenda that influences funding landscapes, which would potentially direct money from lower to higher impact interventions and shift the incentives under which organizations operate. This would involve surveying larger and more well-informed funders to determine what types of research would further improve their funding decisions and then conducting that research. This methodology could be used in parallel with many other research strategies.
Some additional paths that we considered include research coordination, randomized control trials on animal welfare interventions, deeper reports and summaries on interventions, and improving impact evaluations for existing animal organizations. These were assessed to have less potential for impact.
The full report includes a summary of our research, taking into account the uncertainties associated with different types of campaigns.
Did you know that Charity Entrepreneurship can help you start
an effective charity for animals?
If you want to be the first to know when we will start accepting applications to our incubation program, subscribe to our mailing list.
Should we minimize the suffering felt next year or speed up neglected welfare improvements? A simple model
In my work as a research analyst for Charity Entrepreneurship, I have been assessing possible animal welfare campaigns. The first thing I found is that finding the correct welfare asks for corporate or government campaigns is really hard. The scarcity of information in animal advocacy relative to other cause areas means that accurate decisions are harder to make. Establishing reliable metrics to assess ideas is essential to avoid wasting time. One way of doing this is to think about how to approximate the endline metric we are trying to maximise: counterfactual impact.
When trying to make the world a better place, what we really want to measure is the counterfactual impact. Counterfactual impact is the increase in utility that your actions actually cause in the world. If you are choosing to do X action, the actual impact that can be attributed to your action is the difference in utility between the world where you do X and the world where you do not. This is very difficult to measure, as your actions have flow-through effects, which have flow-through effects, and so on. Every action you take ripples out through the butterfly effect to change all of history. The way we account for this is by discounting these flow-through effects based on uncertainty until effects with an unknown direction are discounted to 0. Peter Hurford has written a more fleshed-out discussion of different ways to handle these flow-through effects.
Given that counterfactual impact is what we are pursuing, I have considered two possible strategies for selecting welfare asks: Maximizing the value of counterfactual speed-up and maximizing the welfare increase.
Maximizing the Welfare Increase (WI) vs Maximizing the Value of Counterfactual Speed Up (CSU)
The simplest method could be considered as an instant-gratification method. For this method of decision making, we just choose the biggest welfare increase available to achieve as much welfare in the present as possible. If we use this method then, irrespective of what other organisations are doing, we should select the largest welfare increase available to us. If dissolved oxygen for fish is the largest welfare improvement possible, we should select it, even if we know that it is the next ask for which the rest of the animal movement will campaign. This may only speed the process up by a year, but it will minimise fish suffering in the present.
Counterfactual speed-up is the second strategy. To select asks using this methodology, we need to know one additional piece of information: the time until other actors currently in the field would have chosen this ask. This timeline for welfare improvements allows us to calculate how much sooner our campaign caused the welfare improvement to occur.
For example, imagine we are in 1975, considering a cage-free ask in the United States that would take 10 years to be successful. Knowing what we do now, we could see that conducting a corporate campaign at that time would speed up the timeline for cage-free by ~30 years. If we ignore the possibility of recidivism, by acting in the 1975s we only increase the number of cage-free years by 30. Using this model, the impact we have for chickens is 30 multiplied by the magnitude of the welfare increase caused by cage-free farming. Using this factor in our decision making, we now have some time sensitivity.
If the largest welfare improvements are many years away, both decision-making strategies will dictate the same asks. However, we can quickly see where these two strategies will diverge. Imagine we are only examining two possible welfare asks: one that is a moderate improvement that will not otherwise happen for some time, and another that is large but will occur shortly anyway. If we choose the moderate improvement, it will occur many years sooner than it otherwise would have. If we campaign for the large improvement, it will only occur a few years sooner than it would have without our involvement. This is suboptimal from the counterfactual speed-up perspective, as a moderate ask advanced by many years will speed up welfare more, and thus we will be responsible for more impact. The benefit of this model is that it is relatively simple, as only the expected value of the welfare improvement needs to be determined.
A short examination of these two strategies would lead us to believe that counterfactual speed-up is always the superior model. The time-sensitivity of this way of making decisions should increase the counterfactual impact of your actions. However, it is still just a heuristic for counterfactual impact, not a direct measure. This is not just in terms of nebulous flow-through effects, but also the effect of speed-up on other interventions. If we achieve a welfare ask sooner, other charities can conduct a different campaign in place of this one; and where that second campaign would have been conducted, another; and so on.
Thus, the two main effects of conducting a welfare ask now are:
1) The speed-up of the welfare ask selected
2) The speed-up of all subsequent welfare asks
Imagine what would happen in the timeline where we conduct the year cage-free campaign in 1975. Cage-free improvements for chickens now occur 30 years earlier. Assuming organisations follow the same timeline as before, when they would have had to conduct cage-free campaigns they can now campaign for broiler chicken or fish welfare. In fact, through a single campaign in 1975, you speed up every welfare ask that will occur until the end of animal agriculture.
As long as your use of funding does not slow down the timeline for subsequent campaigns, the impact of your cage-free ask could be 30 years of cage-free chickens and 10 years of broiler and fish welfare improvements. Therefore, if you only measure the counterfactual speed-up for your actions on cage-free, you underestimate your actual impact. This consideration led me to create a very simple model for the charity sector to examine how the two strategies mentioned above compare when trying to maximize counterfactual impact.
To explore this, I created a model of the charity sector where there is currently only one charity working on welfare improvements. There are ten years until animal agriculture ends, and ten possible asks. Once each welfare improvement has been achieved, it is never reversed and will last the remainder of the 10 years until animals are no longer farmed. The magnitude of possible welfare asks that charities can choose from are between 1000 to -100, and vary randomly within that range. This is not a perfect model of the magnitude of difference in actual welfare asks or its distribution. A more accurate model would probably have very few asks that have very cost-effective improvements, with many more closer to 0. I included the possibility of negative asks because our understanding of animal welfare may lead us to accidentally decrease the welfare of animals.
Given this is the space charities are working in, I examined how deciding our ask based on counterfactual speed-up and welfare increase affects the counterfactual impact of our actions. In the real world, we do not have perfect information about the magnitude of welfare asks or the timelines of organizations currently in the field. It is possible that some ways of making decisions perform better in lower-information environments, but I thought examining the ideal situation could still provide some insight.
In the first case, I examined a world in which a Random Altruist charity is already works on welfare asks. This charity chooses randomly from the set of welfare asks; this could be similar to (although probably worse than) choosing asks based on salience and emotional impact. In this case, I modeled a sample of 30 random welfare ask ranges and selections. I found that entering the space with the welfare increase method leads to more optimal outcomes ~17% of the time. The counterfactual speed-up approach leads to more optimal outcomes ~60% of the time. The two models were equally good ~23% of the time. Therefore, if we did not have perfect information and so could not just pick the optimal ask, maximizing counterfactual speed-up is a superior strategy. No surprise there.
However, I then examined how varying the strategy of the existing charity would change the outcome. If the existing charity follows the simple welfare increase approach, the results change. Now following the simple welfare increase strategy is the optimal outcome 100% of the time! This would be the same if the actor in the space is trying maximize counterfactual speed-up, as the best way to do this when alone is to choose the biggest welfare asks.
This outcome seems to suggest that the optimal decision-making strategy in imperfect information environment changes based on the strategy of the other charities in the sector. It seems that the better other organizations are at differentiating between strong and weak welfare asks, the better the welfare increase strategy becomes. However, if asks are currently selected closer to random then counterfactual speed-up is a much better heuristic.
I am unsure how accurate existing organizations would have to be at maximising welfare for the welfare increase strategy to win out. A more complex model may be able to capture this by altering the accuracy of welfare selection. It could also vary the amount and accuracy of the information for the three possible actors in the system.
Author: George Bridgwater, CE's Research Analyst