What gets measured gets improved.
- Robin Sharma
When selecting which projects to implement, you should also try to maximize your impact. Your ethical beliefs will dictate what ‘good achieved’ means to you personally - that will be the metric you use to measure your charity’s success. In this we will discuss the pros and cons of some common metrics but, ultimately, your values will underpin your decision: do you value education as an intrinsic good? How about income? Is it worse to be blind for a year or deaf for a year? Read on to find out why we think wellbeing is often the best metric of all.
The Soviet nail factory: a parable on the importance of metrics
A metric is a unit of measurement. Usually people think of things like centimeters or kilograms, but it can include anything that that can be measured, such as income per capita, health, education, and happiness. Which metrics you choose to pay attention to, and specifically how you measure them, will dramatically change the effectiveness of your charity. Take for example, the following, probably apocryphal, story from the Soviet Union.
During Stalin’s reign, the government set a quantity quota for a nail factory. The manager then orders people to churn out thousands of tiny, useless nails. The government, frustrated, then set set a weight quota. The nails could not be so miniscule as to be useless. The factory hears the order and promptly starts producing big, heavy nails, weighing a tonne each.
The moral of the story? In both cases, the factory was using unhelpful metrics (first quantity and then weight) when ‘number of functional nails’, as well as perhaps some other metrics, like employee satisfaction and health, might have been better indicators of success. Unfortunately, this same mistake is very common in the nonprofit sector.
In order to understand how successful a program really is, you need to understand how to recognize whether a metric is useful or not. This chapter will cover how you can do so, as well as particular metrics that are more or less helpful.
How to spot an incomplete metric
Identify the core goal
It is hugely important to clarify exactly what your target is early on, so you can identify the key metric for your charity or project. Choose a metric that has a thoroughly proven causal link with your end goal. Once you know what your metric is, you’ll be able to monitor whether you’re hitting it - and so will your donors.
Would it be possible to cheat this metric?
Ask yourself whether it is possible to game this metric, thus making it less valuable. For example, if Kate wanted to gain a bunch of traffic to her charity’s website, it would be quite easy for her to invest in non-targeted online ads. Although this would boost her website traffic, it would be very unlikely cause any real increase in welfare, which is the metric Kate really cares about.
A causality chain is a representation of the causal steps linking an intervention to its intended impact. Good metrics measure the end result, so should occur at the end of the causality chain. Take a look at these two examples, one of which measures the endline metric and one of which measures the input and blindly trusts that the chain to impact will work in practice.
Consider an unconditional cash transfer charity that uses self-reported happiness as a metric. Their ‘causality chain’ looks like this:
↓Cash is sent to the global poor with no strings attached↓
↓The global poor spend the money on what they desperately need↓
↓The global poor don’t need to worry about being able to afford basic necessities↓
↓Not having to worry about basic necessities makes the global poor happier↓
☆When people are happier, they say that they are happier when asked.☆
Measuring future happiness is a good way to test the validity of the causal chain. If one of the links is broken, the global poor will not be happier, making this a great measure of the intervention’s effectiveness.
Now consider an organization that lobbies the local government to provide the homeless with free suits for job interviews. They measure success by the number of petitions sent:
↓Many petitions are sent lobbying for suits for the homeless↓
↓The government feels a sense that they should do what the public wants↓
↓The government states they will start a program to provide suits↓
↓The government gets the bill passed↓
↓The program is functional enough that it’s relatively easy for the homeless to get suits↓
↓The homeless go to the program and get suits↓
↓The homeless get job interviews↓
↓The homeless wear the suits to their interviews↓
↓The homeless get jobs they wouldn’t have otherwise↓
↓The jobs prevent them from being homeless↓
☆The homeless are happier working at their new jobs than they were on the street☆
This metric is not the best measure of the intervention’s effectiveness because ‘petitions sent’ occurs at the start of the causality chain, so any weak links further down will go unnoticed.
There’s actually a mathematical explanation for this:
Probability of both A & B happening = [Chance of occurrence A] x [Chance of occurrence B]
For example, to find the odds of flipping two heads in a row, you multiply 0.5 by 0.5. The free suit lobbyists’ chain had 10 connections, so even if there is a 70% chance that each one will follow through, which is fairly high, that’s 0.7 to the power of 10, which is 2.8%. This means there’s a 97.2% chance that the plan will fail. All this assuming that each step has a 70% chance of working, which is implausible.
Of course, sometimes you cannot measure the endline metric, so you will have to make do with something further up the chain. Additionally, if there are multiple studies supporting the causality between each link, we can be more confident in the validity of the chain. In example 1, the causality chain is backed up by evidence. In example 2, the supposed causality is based on problematic/ debatable assumptions. Do petitions affect legislation? Is a new suit enough to get a homeless person a job? In fact, at each step there are plenty of ways for the plan to go wrong. Generally, the more steps there are, the more confidence you need to have in each one.
Examples of incomplete metrics
Let’s look at some real-life examples of ‘incomplete’ metrics that don’t tell the whole story:
Imagine that you are starting a charity and deciding what intervention to implement. You work really hard. In fact, last week, you completed 54 hours of research. Was it effective? It’s hard to know without knowing what you actually spent the hours doing. Sure, your inputs matter some, and it’s possible that working more hours translates into finding a better intervention, but it’s much more accurate to measure your outputs. How much did you accomplish during those hours, and how much more effective is your new best option than your original one?
Imagine that you’re the CEO of a charity. The organization was founded in 2011 and had a budget of $30K. Now it’s 2016 and it has a budget of $300K. That’s tenfold growth in just five years. But is the charity effective? The amount your organization spends, like the amount of time you spend working, is only a means to an end. Again, what matters is what you accomplish with that budget. It’s possible to do more good with $30K than $300K.