This article outlines some simple tips for how to answer questions you have about your program. These tips will hopefully be useful regardless of what stage you are at with your intervention. Most of my experience comes from running surveys in India with J-PAL and Charity Science Health.
SPEAK DIRECTLY WITH RECIPIENTS
If you are more broadly trying to understand the recipients of your program or how to improve your program, focus groups and directly having more in-depth conversations with those recipients can be much more valuable than a standard survey. The first place to start when trying to understand your program's recipients should likely be these types of in-depth conversations. You should only move to more standard surveys after having done this
START WITH A PHONE SURVEY
Phone surveys are cheaper, faster, easier to monitor and should be considered the primary option unless there is a very good reason to do the survey in person. The exceptions to this are when you are asking sensitive questions where you are more likely to get a more representative answer in person (contraceptive use, mental health questions, etc.). Additionally, if you are working in the very poorest communities, it might be hard to identify and reach people by phone. I recommend initially using Google Forms, Skype and a Skype Recorder. If you start doing thousands of surveys, it might be worth the investment costs to move to SurveyCTO.
KEEP IT SHORT
It is very easy to spend more time on surveys than necessary. Surveys tend to grow and become more complicated with time. I recommend creating surveys that help you make a specific decision or are focused on a specific topic. A good rule of thumb is to aim for your surveys to take between 5 and 10 minutes. Following this rule will limit the questions you ask to what really matters and not combine too many topics in one survey. Another way to really get at what matters is to ask, “how does this question inform my decisions?” for every question. For example, you might be interested in knowing all the demographic information about participants in your program. But unless you actually plan to use that information to change your program, it is likely best to just collect a few basic questions on this.
HIGH-QUALITY DATA
The quality of the people conducting your surveys should be very high to start with. You should have confidence that the person has the ability to identify issues with the survey independently and can evaluate for themselves if the answers they are getting back from questions are reasonable. If the surveys are being conducted in another language, you will be putting a lot of trust in the person doing the surveys or the person directly monitoring the surveyors. I find the minimum skill required is found in most people coming out of high-level undergraduate programs. You may want to collect the data yourself if you only plan on collecting a small number of surveys (less than a few hundred).
The risk of making a poor decision because a survey was not designed well or implemented properly is high. The amount of time it will take to set up systems to ensure a survey is working as intended with less experienced staff is significant. If you only plan on collecting fewer than 200 surveys, I recommend that a highly competent member of your staff do the surveys directly with participants. If you're planning to collect more data than this, you may want to hire a less experienced staff member that is directly supervised and monitored fairly heavily by someone with high competence on your team.
Overall, I do not think charity entrepreneurs should be thinking about hiring teams of surveyors until you need to collect hundreds to thousands of surveys every month.
ITERATING
You should be very quickly updating on how your survey is doing. The rule of thumb I usually follow for a survey we plan to do fewer than a few hundred times is:
Directly monitoring every survey being done and making changes to the survey on the spot until we can do the survey 5 times in a row without me finding a substantial improvement.
I will want to look at the data collected after 1-2 days of surveys being completed by a highly competent staff member to see if anything sticks out in the data that indicates the survey is not working as intended.
I will then want to quickly look at the data again when we have collected at about the halfway point to check for red flags.
Then look at the data after the surveys are completely done and make a decision based on the result.
It is a waste of everyone’s time if you look at the data only at the very end and realize that the survey did not collect what you intended to.
SURVEY SOFTWARE
I recommend using Google Forms and a call recording software for most of your phone surveys. If you need to do very complicated surveys that require more complex logic or you are sending less experienced staff directly to participants’ households, I would recommend looking into SurveyCTO. SurveyCTO is a mobile data collection platform used by large research organizations such as J-PAL and IPA. SurveyCTO will allow you to have complicated survey logic, will track the location of where surveys are happening, and will allow you to audio record the surveys being taken in the field. It also has many other features that ensure your data is being collected as you intended by field surveyors. I would only look at SurveyCTO if there is something you can’t do with Google Forms.
SUMMARY
Speak directly with recipients.
Start with a phone survey.
Keep your surveys as short as possible to start with.
Have highly competent members of your team or yourself conduct the surveys to start with.
Quickly update based on early data from your survey.
Use Google Forms unless you need a more sophisticated option