The evolution of surveys has kept up with the development of technology: going from door to door, mailing questionnaires out, calling people to complete phone interviews, using portable computers – and finally online surveys. Online surveys provide some obvious advantages such as reduced paper and administration costs. However, it makes sense to look a bit deeper into the differences between the two general options, online and offline surveys, especially with respect to the quality of the collected data. Quality of data The quality of survey results is made up of the right questions, representativeness of the examined group of interest, and the validity and depth of the answers. Error in your data can occur systematically due to biases, like researcher bias, survey bias, respondent bias and non-response bias, or randomly by misrepresentation. But it’s not just the accuracy of the data: Especially when surveying feedback on customers or an event experience or gathering new ideas and suggestions for improvement, the quality of free text fields are crucial for usable data. All those aspects are notably influenced by your choice of survey format. Response time and rates Online surveys most certainly produce quicker responses, since you can share the link to your survey immediately after creating it without having to call people, mail it to them or meet them in person. Offline surveys, on the other hand, usually produce higher response rates with in-person surveys being the most effective, followed by mail surveys. Online and email surveys still have a better response rate than telephone surveys. The height of response rates doesn’t necessarily translate to representation: If you want to represent the whole population of one country, you’ll need a much lower response rate than when you wish for feedback from a defined group. In a smaller group, you’ll get much smaller sample sizes and it’s much more likely to miss neutral responses and get more positive and negative results. Target group selection Most of the time, the lower response rates of online surveys can be compensated by being able to reach more people – even worldwide if you wish. With the right software you can run a survey in multiple languages of which the participants can select. If the survey has a specific target group or needs a certain frequency distribution in order to be representative, quota setting is a solution that can be used in survey software. A downside to online surveys is that random sampling is more difficult, since generating random email addresses like with phone numbers is not possible. That is not a problem if the group of interest consists of the members of a club, the students of a certain university, or registered customers to a shop. Otherwise panel companies can offer a solution to getting the wished sample size. The benefits of offline interviews include going to specific places after an event to reach specific people and also reaching the computer-illiterate population. Though the “online population” used to have special characteristics like being younger, better educated and more tech-savvy, the computer literacy has drastically increased in the last 15 years. Online surveys are thus reaching more and more people, whereas phone surveys, being still a popular medium of choice, are reaching less and less people, since households that have no landline are increasing, in particular with younger people. (Perceived) anonymity Especially upon addressing sensitive topics like smoking, drinking, sexual behavior, cheating on taxes or gambling, self-protective answering can be observed. This means the participant is not answering honestly, downplays or exaggerates his or her answer. It occurs especially with an interviewer present and even depends on factors like age, gender and ethnicity. But also when asking less sensitive questions, and instead about the experience with a product or event, the results might be biased by the presence of the interviewer in in-person interviews, since people may be more polite and less honest. This also applies to situations with an existing relationship of dependence, as often observed at universities, where the professor to be rated is present at the evaluation. Oftentimes people mistrust online surveys, when they are unsure of their data safety. Particularly if you want to prevent multiple participation, you either have to select setting cookies, saving the IP address or only allow access with a personalized token. All those options might prevent the participants from answering upfront or from answering at all, if you neglect assuring the participants of their anonymity and also choosing a software tool that allows anonymization as needed. This not only concerns the data accessible to the survey creators but also to the software provider. This is why the hosting and data storage should be at best on servers within countries with high data protection legislation like Iceland, Germany, the Netherlands or Spain.   Other technical advantages Although the readability of black ink on white paper goes unchallenged, online surveys still win the victory in the design category. By not being restricted in space, it is not only possible to create a more appealing appearance with more coloring and implementation of pictures and videos, but also to provide unlimited space for free text entries. To keep participants interested, online surveys allow many options for personalization, including branching to only show relevant questions, piping to reuse answers or defining different end URLs. Though readability might be better in paper surveys, online surveys grant access to people with disabilities such as poor eyesight by being compatible with screen readers. On the researcher’s side deciphering handwriting is being skipped and less errors in data entry occur with online surveys. Interaction A special benefit of personal interviews arises when you want feedback to an event taking place or to ask for customer feedback immediately after the experience: Since the experience (and the excitement at best) is still fresh in their mind, the motivation to fill in a questionnaire might be higher, particularly in regards to open text fields. The interviewer bias, that has already been mentioned, can be utilized in a good way: From my own experience, a face to face interaction can be beneficial when you are looking for answers in form of free text fields. I learned that people often do not think of their opinion or thoughts as relevant, and so they’d rather skip a question than entering something they deem wrong. In personal interviews you can encourage those to share their experiences. If required, you can give further explanations or ask further questions when fitting. Although offline surveys immediately after the examined experience make a lot of sense, online surveys are still superior when it comes to time management. People are often reluctant to answering at all or fill out their questionnaire as quickly as possible if they feel pressured. Did you know? When creating questionnaires with LimeSurvey, you can have respondents take your questionnaire offline with no need for internet availability. The free Android app OfflineSurveys enables caching any LimeSurvey survey and collecting your data offline. Once the Internet is available again, the app can automatically synchronise the collected responses with your LimeSurvey system. Feel free to check it out here. Conclusion The question whether to choose an online or offline survey cannot be answered conclusively – and it does not need to be. The choice has to be made based on the particular cause depending on the survey goal and type, employee or customer feedback, academic research or feedback to an event. In many cases a combination of both might make the most sense, because that way you can also combine the benefits of both: broadly speaking, the quantity of online surveys with the quality of paper surveys and the combined accessibility of different target groups. LimeSurvey provides a convenient way to combine both: questionnaires that have been created online can be easily converted into a word document or a PDF file. The question display is thereby adjusted in order to be used in a written or personal interview. Afterwards the collected data can be entered manually into LimeSurvey, so that in the end both data sets can be analyzed collectively. All aforementioned features like multiple languages, data security, choosing the degree of anonymity and personalization of the survey design are also provided by LimeSurvey.

The phenomenon in question is called non-response bias and means that a study or poll is unrepresentative when a certain part of the target group didn't participate. There are numerous factors that promote a better response rate: A smart survey design that speaks to the participant, including the right length, use of pictures and colours, topping off the structure with piping, tailoring and skip logic, using a progress bar etc. Personalized emails and follow-up reminders (they are most effective 1 to 2 weeks after the first email) Ensuring anonymity Presenting the prospect of incentives in the form of lotteries or rewards Creating Incentives The Harvard professor Michael Sandel explains in his book "What Money Can't Buy" in what ways putting a prize on an object can make it worthless or less worth. One textbook example is blood donation: When you give out a reward for donating blood with the intention of receiving more blood, the quality of the blood decreases – according to Sandel's theory. His reasoning is the following: A monetary incentive motivates especially people from the lower end of the economic spectrum. And people on the lower end of the economic spectrum tend to have a lower health status, hence lower blood quality with a higher risk for diseases like hepatitis. Another effect he describes is that blood donation loses its altruistic value when adding a monetary value to it. It becomes another object on the economic market and leaves the domain of morality. By making it a commodity, people lose the sense of a moral responsibilty to donate blood. This might, in the long-term, lead to less donors. Application to surveys   Transferring those thoughts onto the more mundane act of paying people for perfoming surveys, you can analyze the same two affected levels: a shift in the target group and a shift of morality. What are the implications here? And what do studies suggest? Firstly, do incentives attract more people and trigger a demographic shift? To cut a long story short, yes, incentives work in terms of improving the response rate. Stating a specific percentage is nearly impossible, since different studies show different results, ranging from 10 to 50 %, and they seem to be largely influenced by other factors. The current state of research does not indicate a demographic effect, so it seems the incentive is not attracting a certain group. Secondly, what effects do incentives have on the morality of people when performing a survey? Does the time invested and the quality of the answers change? There are studies that even indicate that it's the contrary: The incentive might be a stimulus to complete the survey more thoroughly. At the same time, the social desirability bias might be affected, meaning the partipants tend to give answers, they anticipate are expected of or wished from them. But this effect is ambigious and depends on factors like the country and type of question. For example, within one study the answers to questions concerning the income seemed biased, whereas the answers to questions regarding political views remained unaffected. What recommendations do studies suggest? Depending on your budget you might want to decide between a lottery style incentive with a chance to win and a reward for everyone. When deciding, take into account, that other factors play into the response rate as well. It seems that for short surveys lotteries are more cost efficient and vouchers for longer surveys. A fun effect of the lottery choice is, that it not only increases the response rate but also the response speed by a few days. Even though it's irrational, people assumably base this on the misconception that their chance of winning increases when entering early. To reduce the incidence of unfinished surveys, it has been observed that giving the reward at the beginning might increase the sense of responsibilty to finish the study. What might have a bigger impact on the quality of the answers is the length of the survey. Studies showed that you receive a higher amount of "I don't know"-answers with longer questionnaires. Conclusion It appears that Sandel's theory does not apply to the pricing of survey-taking. So, in fact yes, money can buy higher response rates and seemingly without causing any damage. The impact on the quality of the surveys is not yet conclusively determined. The non-response bias mentioned in the beginning might be reduced while accepting the risk of slightly raising the social desirability bias. In any case, it always makes sense to check all of the above mentioned factors for increasing your response rate. Every one of them can be implemented conveniently into your questionnaire with the LimeSurvey software.   If you want to stay informed check out our newsletter.   And if you have further questions, simply contact our support and we are going to help asap.  

As announced in your previous blog post the 2.06LTS (Long Term Support) version is now available!Please note that this version is available on ComfortUpdate and release package download only to our Premium Package subscribers.This LTS version is only meant for existing users of LimeSurvey 2.06 - if you are a new user please download and use the latest Version 2.50 instead.