A high response rate is not only crucial for collecting data as fast as possible with as little effort as possible. It potentially has a significant impact on the quality of your survey results. The phenomenon in question is called non-response bias and means that a study or poll is unrepresentative when a certain part of the target group didn't participate.
There are numerous factors that promote a better response rate:
- A smart survey design that speaks to the participant, including the right length, use of pictures and colours, topping off the structure with piping, tailoring and skip logic, using a progress bar etc.
- Personalized emails and follow-up reminders (they are most effective 1 to 2 weeks after the first email)
- Ensuring anonymity
- Presenting the prospect of incentives in the form of lotteries or rewards
Creating Incentives
The Harvard professor Michael Sandel explains in his book "What Money Can't Buy" in what ways putting a prize on an object can make it worthless or less worth. One textbook example is blood donation: When you give out a reward for donating blood with the intention of receiving more blood, the quality of the blood decreases – according to Sandel's theory. His reasoning is the following:
A monetary incentive motivates especially people from the lower end of the economic spectrum. And people on the lower end of the economic spectrum tend to have a lower health status, hence lower blood quality with a higher risk for diseases like hepatitis.
Another effect he describes is that blood donation loses its altruistic value when adding a monetary value to it. It becomes another object on the economic market and leaves the domain of morality. By making it a commodity, people lose the sense of a moral responsibilty to donate blood. This might, in the long-term, lead to less donors.
A monetary incentive motivates especially people from the lower end of the economic spectrum. And people on the lower end of the economic spectrum tend to have a lower health status, hence lower blood quality with a higher risk for diseases like hepatitis.
Another effect he describes is that blood donation loses its altruistic value when adding a monetary value to it. It becomes another object on the economic market and leaves the domain of morality. By making it a commodity, people lose the sense of a moral responsibilty to donate blood. This might, in the long-term, lead to less donors.
Application to surveys
Transferring those thoughts onto the more mundane act of paying people for perfoming surveys, you can analyze the same two affected levels: a shift in the target group and a shift of morality. What are the implications here? And what do studies suggest?
Firstly, do incentives attract more people and trigger a demographic shift? To cut a long story short, yes, incentives work in terms of improving the response rate. Stating a specific percentage is nearly impossible, since different studies show different results, ranging from 10 to 50 %, and they seem to be largely influenced by other factors. The current state of research does not indicate a demographic effect, so it seems the incentive is not attracting a certain group.
Secondly, what effects do incentives have on the morality of people when performing a survey? Does the time invested and the quality of the answers change? There are studies that even indicate that it's the contrary: The incentive might be a stimulus to complete the survey more thoroughly.
At the same time, the social desirability bias might be affected, meaning the partipants tend to give answers, they anticipate are expected of or wished from them. But this effect is ambigious and depends on factors like the country and type of question. For example, within one study the answers to questions concerning the income seemed biased, whereas the answers to questions regarding political views remained unaffected.
Firstly, do incentives attract more people and trigger a demographic shift? To cut a long story short, yes, incentives work in terms of improving the response rate. Stating a specific percentage is nearly impossible, since different studies show different results, ranging from 10 to 50 %, and they seem to be largely influenced by other factors. The current state of research does not indicate a demographic effect, so it seems the incentive is not attracting a certain group.
Secondly, what effects do incentives have on the morality of people when performing a survey? Does the time invested and the quality of the answers change? There are studies that even indicate that it's the contrary: The incentive might be a stimulus to complete the survey more thoroughly.
At the same time, the social desirability bias might be affected, meaning the partipants tend to give answers, they anticipate are expected of or wished from them. But this effect is ambigious and depends on factors like the country and type of question. For example, within one study the answers to questions concerning the income seemed biased, whereas the answers to questions regarding political views remained unaffected.
What recommendations do studies suggest?
Depending on your budget you might want to decide between a lottery style incentive with a chance to win and a reward for everyone. When deciding, take into account, that other factors play into the response rate as well. It seems that for short surveys lotteries are more cost efficient and vouchers for longer surveys.
A fun effect of the lottery choice is, that it not only increases the response rate but also the response speed by a few days. Even though it's irrational, people assumably base this on the misconception that their chance of winning increases when entering early.
To reduce the incidence of unfinished surveys, it has been observed that giving the reward at the beginning might increase the sense of responsibilty to finish the study.
What might have a bigger impact on the quality of the answers is the length of the survey. Studies showed that you receive a higher amount of "I don't know"-answers with longer questionnaires.
A fun effect of the lottery choice is, that it not only increases the response rate but also the response speed by a few days. Even though it's irrational, people assumably base this on the misconception that their chance of winning increases when entering early.
To reduce the incidence of unfinished surveys, it has been observed that giving the reward at the beginning might increase the sense of responsibilty to finish the study.
What might have a bigger impact on the quality of the answers is the length of the survey. Studies showed that you receive a higher amount of "I don't know"-answers with longer questionnaires.
Conclusion
It appears that Sandel's theory does not apply to the pricing of survey-taking. So, in fact yes, money can buy higher response rates and seemingly without causing any damage. The impact on the quality of the surveys is not yet conclusively determined. The non-response bias mentioned in the beginning might be reduced while accepting the risk of slightly raising the social desirability bias.
In any case, it always makes sense to check all of the above mentioned factors for increasing your response rate. Every one of them can be implemented conveniently into your questionnaire with the LimeSurvey software.
In any case, it always makes sense to check all of the above mentioned factors for increasing your response rate. Every one of them can be implemented conveniently into your questionnaire with the LimeSurvey software.
If you want to stay informed check out our newsletter.
And if you have further questions, simply contact our support and we are going to help asap.
