Beginners Guide to LimeSurvey Part 2: What counts as a response in LimeSurvey Cloud – and how you can lower survey break-off

How are responses counted in LimeSurvey Cloud? How you can minimize survey break-off? What can you do if you are running short of responses? How many responses do you need for a certain net sample? Check out this blog post and learn how to get as many complete questionnaires out of your package as possible. Watch out for our pro tips!

Responses and how they are counted in LimeSurvey Cloud

In LimeSurvey Cloud the term "response" basically translates to completed questionnaire. In the best case, a respondent starts your questionnaire and completes it until the end page. Each finished questionnaire counts as one response and lowers your remaining responses available in your LimeSurvey Cloud package by one point.

But in web surveys – like in all self-administered survey modes – we frequently see respondents who start a questionnaire but break off participation during the survey-taking process. Dropout that happens during survey-taking counts as half a response. In LimeSurvey one response can be used for really extensive questionnaires. However the longer the questionnaire, the higher the risk of incompletes.

Many dropouts take place on the first page of a survey – for example, when participants find out that the topic is not relev.ant to them. That is when the welcome screen should be used. The welcome screen is a special page where you can describe your research and show your survey data policy. Once enabled, dropouts on the welcome screen of a survey won’t lower your response balance

Pro Tip 1: Consider using the welcome screen with your survey as break-off on that page does not lower your available response contingent.

Besides regular survey participation, manually editing the entries of the survey response table counts as response. If you create a new entry directly in the survey response table – e.g., if you do a mixed-mode study – it counts as one response. Also if you change the status of a survey from partially completed to completed, your response balance will be lowered by one response instead of half a response. 

What you can do to lower break-off

Break-off is inevitable in surveys. But you can keep it as low as possible when considering the following:

- Make sure you target a relevant audience.

- Make your survey not too long. How long? Well, actually that depends on your audience and the survey’s topic. In any case be honest about the survey’s topics and its length in the invitation, as well as on the welcome page.

- Show that you are a serious survey researcher and that you respect data protection and privacy. Use LimeSurvey's built-in feature to display your data policy on the welcome page.

- Offer an incentive – some kind of reward for your respondents – for complete participation. Incentives are either an per capita incentive (everyone gets it), the participation in a lottery drawing (e.g., of money or a voucher), or indirect motivation (like the donation to charity or that respondents get access to certain information).

- Follow the general guidelines and rules for questionnaire design and for good survey practice (e.g., question wording, response scales, task difficulty, etc.).

- Test your survey logic and question relevance.

- Make mandatory as few questions as possible.

- Take advantage of LimeSurvey’s feature to save a survey and resume it later.

- If you have the possibility, use email not only for invitations but also for reminders (we recommend up to two reminders).

- Before you field your actual survey, do a pretest – e.g., with colleagues – to check if there are any problems. Consider paradata like response times to detect difficult questions.

Pro Tip 2: Thoroughly design your survey following established guidelines and test it with a small sample in advance. This will help to get higher completion rates and higher data quality. 

Tracking your responses in LimeSurvey Cloud

The most obvious difference between the LimeSurvey Cloud packages is how many questionnaires (aka responses) can be completed. The number of responses available in LimeSurvey Cloud starts from 25 per month (Free package) and for regular packages up to 1.000.000 responses per year (Corporate package) – which can be even topped with individual offers. At the end of each subscription period unused responses expire and the response limit is reset to the start value.

There are two places where you can find your current responses balance. In your survey installation (also called survey instance) you see your available responses in the top menu. The other place is inside your LimeSurvey profile in the menu survey application. If you choose edit in that menu, you can set a limit for remaining responses. As soon as your remaining responses fall below this limit, you will receive an alert email. 

Running out of responses? Read our following tips

Sometimes more people participate in your survey than you expect or high break-off rates lower your response balance. Before you run out of responses you will receive an alert email, but as soon as your response balance hits zero, people who try to access your survey will get an error message.

So what can you do if you run out of responses?

1) You can either wait until your subscription renews or 2) You can renew your subscription early - just log in on our platform, access the survey application menu, and renew your package earlier..3) If you exceed your responses not only once in a while, upgrading to a higher package is the recommended solution.

Pro Tip 3: For more flexibility, you should opt for a yearly subscription. We also offer multi-year packages for companies - just contact our sales team.

Estimation of response volume

To avoid running short of responses during your survey, we recommend you do a rough estimation of the responses you need in advance. Let’s think it from the end: Given the case you want to analyze a net sample of 500 complete questionnaires. However, frequently not all complete questionnaires are really suited for data analysis. When checking the answers for missing values, strange response patterns (e.g., always selecting the first response), illogical answers, response duration (if a participant is too fast, this can be an indicator for low seriousness), you may have to remove 10% of the cases. So you need about 550 completes. If the break-off rate of your survey is around 30%, you need to invite another 240 persons (120 responses) to fill it out. Moreover, you should also plan for your pretest another 30 responses. Overall, you need approximately 700 responses to end with a net sample of 500 cases. Please keep in mind that the numbers in this example may greatly vary! Especially the dropout rate which can be considerably lower or higher.

Pro Tip 4: Estimate the responses you need in advance – but always monitor your response numbers while your survey is online.

Further reading

You can find additional information in the online manual:

https://manual.limesurvey.org/Responses_&_statistics 

https://manual.limesurvey.org/Survey_quotas 

Ready for Part 3?

Now that you are really into our "Beginners Guide" Series, you should take a look into our last blog post from that series.

Part 3

but if you haven´t read part 1, then you should go ahead and read that first.

Part 1