This article outlines some simple tips for how to answer questions you have about your program. These tips will hopefully be useful regardless of what stage you are at with your intervention. Most of my experience comes from running surveys in India with J-PAL and Charity Science Health.
Speak directly with recipients
If you are more broadly trying to understand the recipients of your program or how to improve your program, focus groups and directly having more in-depth conversations with those recipients can be much more valuable than a standard survey.
The first place to start when trying to understand your program's recipients should likely be these types of in-depth conversations. You should only move to more standard surveys after having done this.
Start with a phone survey
Phone surveys are cheaper, faster, easier to monitor and should be considered the primary option unless there is a very good reason to do the survey in person. The exceptions to this are when you are asking sensitive questions where you are more likely to get a more representative answer in person (contraceptive use, mental health questions, etc.). Additionally, if you are working in the very poorest communities, it might be hard to identify and reach people by phone.
I recommend initially using Google Forms, Skype and a Skype Recorder. If you start doing thousands of surveys, it might be worth the investment costs to move to SurveyCTO.
Keep it short
It is very easy to spend more time on surveys than necessary. Surveys tend to grow and become more complicated with time. I recommend creating surveys that help you make a specific decision or are focused on a specific topic.
A good rule of thumb is to aim for your surveys to take between 5 and 10 minutes. Following this rule will limit the questions you ask to what really matters and not combine too many topics in one survey. Another way to really get at what matters is to ask, “how does this question inform my decisions?” for every question.
For example, you might be interested in knowing all the demographic information about participants in your program. But unless you actually plan to use that information to change your program, it is likely best to just collect a few basic questions on this.
The quality of the people conducting your surveys should be very high to start with. You should have confidence that the person has the ability to identify issues with the survey independently and can evaluate for themselves if the answers they are getting back from questions are reasonable. If the surveys are being conducted in another language, you will be putting a lot of trust in the person doing the surveys or the person directly monitoring the surveyors. I find the minimum skill required is found in most people coming out of high-level undergraduate programs. You may want to collect the data yourself if you only plan on collecting a small number of surveys (less than a few hundred).
The risk of making a poor decision because a survey was not designed well or implemented properly is high. The amount of time it will take to set up systems to ensure a survey is working as intended with less experienced staff is significant. If you only plan on collecting fewer than 200 surveys, I recommend that a highly competent member of your staff do the surveys directly with participants. If you're planning to collect more data than this, you may want to hire a less experienced staff member that is directly supervised and monitored fairly heavily by someone with high competence on your team.
Overall, I do not think charity entrepreneurs should be thinking about hiring teams of surveyors until you need to collect hundreds to thousands of surveys every month.
You should be very quickly updating on how your survey is doing. The rule of thumb I usually follow for a survey we plan to do fewer than a few hundred times is:
I recommend using Google Forms and a call recording software for most of your phone surveys. If you need to do very complicated surveys that require more complex logic or you are sending less experienced staff directly to participants’ households, I would recommend looking into SurveyCTO.
SurveyCTO is a mobile data collection platform used by large research organizations such as J-PAL and IPA. SurveyCTO will allow you to have complicated survey logic, will track the location of where surveys are happening, and will allow you to audio record the surveys being taken in the field. It also has many other features that ensure your data is being collected as you intended by field surveyors.
I would only look at SurveyCTO if there is something you can’t do with Google Forms.