- Posts: 21
- Thank you received: 1
Welcome to the LimeSurvey Community Forum
Ask the community, share ideas, and connect with other LimeSurvey users!
200k participants over a 7 day period. How big of a server would I need?
- pokeydale
- Topic Author
- Offline
- Junior Member
Less
More
7 years 1 month ago #150189
by pokeydale
200k participants over a 7 day period. How big of a server would I need? was created by pokeydale
We are planning to rollout serveral surveys with an overall participation of 200k users. Can anybody suggest a server configuration to be able to handle this volume.
The topic has been locked.
- Ben_V
- Offline
- Platinum Member
Less
More
- Posts: 1128
- Thank you received: 329
7 years 1 month ago #150191
by Ben_V
Benoît
EM Variables => bit.ly/1TKQyNu | EM Roadmap => bit.ly/1UTrOB4
Last Releases => 2.6x.x goo.gl/ztWfIV | 2.06/2.6.x => bit.ly/1Qv44A1
Demo Surveys => goo.gl/HuR6Xe (already included in /docs/demosurveys)
Replied by Ben_V on topic 200k participants over a 7 day period. How big of a server would I need?
I suggest you this read:
www.limesurvey.org/community/forums/desi...urvey-takers?#144504
www.limesurvey.org/community/forums/desi...urvey-takers?#144504
Benoît
EM Variables => bit.ly/1TKQyNu | EM Roadmap => bit.ly/1UTrOB4
Last Releases => 2.6x.x goo.gl/ztWfIV | 2.06/2.6.x => bit.ly/1Qv44A1
Demo Surveys => goo.gl/HuR6Xe (already included in /docs/demosurveys)
The topic has been locked.
- jelo
- Offline
- Platinum Member
Less
More
- Posts: 5033
- Thank you received: 1257
7 years 1 month ago #150193
by jelo
The meaning of the word "stable" for users
www.limesurvey.org/forum/development/117...ord-stable-for-users
Replied by jelo on topic 200k participants over a 7 day period. How big of a server would I need?
How do you come to the conclusion of getting 200000 participants over 7 days?
How many surveys? How different are these surveys in terms of content and amount of questions?
How many surveys? How different are these surveys in terms of content and amount of questions?
The meaning of the word "stable" for users
www.limesurvey.org/forum/development/117...ord-stable-for-users
The topic has been locked.
- holch
- Offline
- LimeSurvey Community Team
Less
More
- Posts: 11639
- Thank you received: 2737
7 years 1 month ago #150234
by holch
I answer at the LimeSurvey forum in my spare time, I'm not a LimeSurvey GmbH employee.
No support via private message.
Replied by holch on topic 200k participants over a 7 day period. How big of a server would I need?
As you can see, this is not a simple question. As Jelo says, it depends very much on the length of the survey, how many conditions/equations, etc. So while a short survey with simple questions and no equations might run fine with 200k in 7 days on one hardware, a very complex survey with a lot of questions and equations might totally crash the same server with this amount of reponses. I think 200k in 7 days is not very common out there, so there won't be many that have experience with this.
I answer at the LimeSurvey forum in my spare time, I'm not a LimeSurvey GmbH employee.
No support via private message.
The topic has been locked.
- ddrmoscow
- Offline
- Junior Member
Less
More
- Posts: 21
- Thank you received: 10
7 years 1 month ago #150243
by ddrmoscow
Replied by ddrmoscow on topic 200k participants over a 7 day period. How big of a server would I need?
Hi all,
200k in 7 days is not a problem by itself.
The bottleneck is peak Nr of conncurrent requests (or Nr of respondents at once) * survey duration.
With even distribution of 200k participants over 7 fays it makes 28.5k+ daily or 2.4k /h (in 12h frame) or just 1.9k /h (with 24h frame).
For more accurate evaluation of required server capacity and configuration I'd suggest to:
1. estimate response rate >> plan intitation e-mail sending campain (if you can) >> estimate max respondents at once (for each suvey if several running simultaneusly) >> estimate total Nr of simultaneus requests.
2. prepare and test all your surveys on some other installation with desired server configuration (type of load balancer + http server + DB server + caching + LS version) ** imho 2.06 is a bit faster than 2.50, though 2.50 more platorm-independent (due to bootstrap) + mobile friendly **
3. test on diff platfoms if required ()
4. measure survey duration, avg memory load, processor load, db process time and memory load, template weight, page load speed for each survey and platform >> replace with cdn (if possible) +optimize templates/ resourses to obtain minimal values
5. Calculate approx. resourses required >> RAM (~max. mem load per request * Nr of concurrent respondents), CPU, network bandwidth for each process (balancer/http/db/cache) + 25/30% extra capacity (if you're more or les certain regarding Nr of concurrent respondents) or even +50/60% if not.
6. decide if spare (backup) server is required ,and if you plan to use 1 server config or cluster
Hope this will help you to estimate and plan your installation configuration.
BTW, latest high-load experience: 22k+ in first 15 mins after start, 75k+ total responses in 2 days.
Configuration used >> failover switch (nginx server) + 2 application clusters :
main cluster -- 2 load balancers (nginx) + 3 http servers (apache) + 1 cahing server (memcached) + 1 db server (maria)
reserve cluster (clone mirror instance) -- 1 load balancer + 2http + 1 cahing +1 db
This time All servers with automatic scale up/down from 2*2.4Ghz/4Gb to 2*24Ghz /32Gb
Good luck,
Regards
200k in 7 days is not a problem by itself.
The bottleneck is peak Nr of conncurrent requests (or Nr of respondents at once) * survey duration.
With even distribution of 200k participants over 7 fays it makes 28.5k+ daily or 2.4k /h (in 12h frame) or just 1.9k /h (with 24h frame).
For more accurate evaluation of required server capacity and configuration I'd suggest to:
1. estimate response rate >> plan intitation e-mail sending campain (if you can) >> estimate max respondents at once (for each suvey if several running simultaneusly) >> estimate total Nr of simultaneus requests.
2. prepare and test all your surveys on some other installation with desired server configuration (type of load balancer + http server + DB server + caching + LS version) ** imho 2.06 is a bit faster than 2.50, though 2.50 more platorm-independent (due to bootstrap) + mobile friendly **
3. test on diff platfoms if required ()
4. measure survey duration, avg memory load, processor load, db process time and memory load, template weight, page load speed for each survey and platform >> replace with cdn (if possible) +optimize templates/ resourses to obtain minimal values
5. Calculate approx. resourses required >> RAM (~max. mem load per request * Nr of concurrent respondents), CPU, network bandwidth for each process (balancer/http/db/cache) + 25/30% extra capacity (if you're more or les certain regarding Nr of concurrent respondents) or even +50/60% if not.
6. decide if spare (backup) server is required ,and if you plan to use 1 server config or cluster
Hope this will help you to estimate and plan your installation configuration.
BTW, latest high-load experience: 22k+ in first 15 mins after start, 75k+ total responses in 2 days.
Configuration used >> failover switch (nginx server) + 2 application clusters :
main cluster -- 2 load balancers (nginx) + 3 http servers (apache) + 1 cahing server (memcached) + 1 db server (maria)
reserve cluster (clone mirror instance) -- 1 load balancer + 2http + 1 cahing +1 db
This time All servers with automatic scale up/down from 2*2.4Ghz/4Gb to 2*24Ghz /32Gb
Good luck,
Regards
The following user(s) said Thank You: tpartner, driz
The topic has been locked.
- holch
- Offline
- LimeSurvey Community Team
Less
More
- Posts: 11639
- Thank you received: 2737
7 years 1 month ago #150244
by holch
I answer at the LimeSurvey forum in my spare time, I'm not a LimeSurvey GmbH employee.
No support via private message.
Replied by holch on topic 200k participants over a 7 day period. How big of a server would I need?
Hi ddrmoscow, thank you for this hands on information about high traffic Limesurvey usage.
I think there are very few users out there, that have experience with these numbers of respondents. Especially in market research samples are usually a lot smaller.
While the details are a little over my head in terms of server configuration, etc. it is good to get a glimpse into the high traffic practice and I hope it helps the OP to get their surveys running.
22k in the first minute is heavy. But also sounds like you could have distributed the email sending a little bit to keep the peaks low.
Did you send out all emails at the same time?
I think there are very few users out there, that have experience with these numbers of respondents. Especially in market research samples are usually a lot smaller.
While the details are a little over my head in terms of server configuration, etc. it is good to get a glimpse into the high traffic practice and I hope it helps the OP to get their surveys running.
22k in the first minute is heavy. But also sounds like you could have distributed the email sending a little bit to keep the peaks low.
Did you send out all emails at the same time?
I answer at the LimeSurvey forum in my spare time, I'm not a LimeSurvey GmbH employee.
No support via private message.
The topic has been locked.
- ddrmoscow
- Offline
- Junior Member
Less
More
- Posts: 21
- Thank you received: 10
7 years 1 month ago #150246
by ddrmoscow
Replied by ddrmoscow on topic 200k participants over a 7 day period. How big of a server would I need?
LOL , you're definitelly right about proper e-mail campaing planning and distribution .
But not this time - sending was totally out of our control, moreover, sent in 1 batch of 500k+ as far as I know...
..and no tokens this time.
It was the main reason to get that "ddos-like" load at start , and I had to create robust expandable instance.
But not this time - sending was totally out of our control, moreover, sent in 1 batch of 500k+ as far as I know...
..and no tokens this time.
It was the main reason to get that "ddos-like" load at start , and I had to create robust expandable instance.
The topic has been locked.
- jelo
- Offline
- Platinum Member
Less
More
- Posts: 5033
- Thank you received: 1257
7 years 1 month ago #150247
by jelo
Professional Hosting offered by Limesurvey.org is using plain dedicated servers (perhaps that has changed in 2017, but no information about the hosting infrastructure is available on this website).
The budget alone in terms of hours (even if you have the knowledge available, which is not the case for most users here) would make outsourcing a valid option. Not sure what you had to pay for the hosting alone, but the tight budget (and not the option of selfhosting) is bringing the people to the LimeSurvey table I'm too lazy to get horizontal scaling working on webstack level. I want that whole scaling on the hypervisor level. I'm impressed by your efforts. Hopefully you get paid well. I'm looking forward to LS 3.0 tests. The new template engine might change a bit.
The meaning of the word "stable" for users
www.limesurvey.org/forum/development/117...ord-stable-for-users
Replied by jelo on topic 200k participants over a 7 day period. How big of a server would I need?
You're are a unicorn as a LimeSurvey userddrmoscow wrote: 200k in 7 days is not a problem by itself.
Professional Hosting offered by Limesurvey.org is using plain dedicated servers (perhaps that has changed in 2017, but no information about the hosting infrastructure is available on this website).
The budget alone in terms of hours (even if you have the knowledge available, which is not the case for most users here) would make outsourcing a valid option. Not sure what you had to pay for the hosting alone, but the tight budget (and not the option of selfhosting) is bringing the people to the LimeSurvey table I'm too lazy to get horizontal scaling working on webstack level. I want that whole scaling on the hypervisor level. I'm impressed by your efforts. Hopefully you get paid well. I'm looking forward to LS 3.0 tests. The new template engine might change a bit.
The meaning of the word "stable" for users
www.limesurvey.org/forum/development/117...ord-stable-for-users
The topic has been locked.