Humongous sessions

More
1 year 2 months ago - 1 year 2 months ago #140917 by jschreuder
jschreuder created the topic: Humongous sessions
We're running a very big survey with 173 questions in 25 groups with many conditions. After opening it up to about 25000 potential participants it crashed after about 900 had tried filling it out. It turned out that the session table exploded, each session being about 2 megabytes in size. I've had to truncate the sessions table to get the site running again. I've switched to PHP's own session driver instead of MySQL and looked in the generated session file, it turns out it stores the complete survey (questions, conditions, intros, etc.) in there for each and every user. Is that how it is intended to work? Our table exploded to about 2 gigabytes in size because of this. Is there any way I can switch this off? Because there's no way I can run the survey like this...
Last Edit: 1 year 2 months ago by jschreuder.

Please Log in to join the conversation.

More
1 year 1 month ago #140945 by Moradzadeh
Moradzadeh replied the topic: Humongous sessions
Looking at my own database (running 7 small surveys), I see nothing of the kind of storage you report. What table is the complete survey appearing in for each session?

Please Log in to join the conversation.

More
1 year 1 month ago #140949 by jschreuder
jschreuder replied the topic: Humongous sessions
I've put it on another server now. And with file-based storage, and each individual session still takes about 1.8 megabytes. But it won't crash my DB anymore. It was stored in the lime_sessions table.

Still when I look into each session file PHP's native session handler generates it has the entire survey in there. With everything. So I'm not surprised you won't see as much with a small survey (we haven't had trouble up till now with smaller surveys). But the sessions should still be surprisingly big, even for small surveys. As they take up much more space than just the keys + answers should.

Please Log in to join the conversation.

More
1 year 1 month ago #140950 by Moradzadeh
Moradzadeh replied the topic: Humongous sessions
Glad you found a solution. Odd, when running a survey response, I still do not see that, but I am probably trying to replicate your issue the wrong way.

Please Log in to join the conversation.

More
1 year 1 month ago #140953 by jschreuder
jschreuder replied the topic: Humongous sessions
I was actually hoping for a misconfiguration on my part, but I found the following in the documentation this morning:

"When using database sessions, make sure then MySQL setting max_allowed_packet is set large enough, some surveys generate over 2 MB of session data."
- manual.limesurvey.org/Global_settings#General_description (last bit of "Session lifetime")

So this seems to be intended behaviour. I fear it makes LimeSurvey unfit for large-scale surveys with big questionnaires. Not complaining, I'm very happy with it for the small stuff. But it seems a pity as I think it's unnecessary to make sessions this huge.

Please Log in to join the conversation.

More
1 year 1 month ago #140954 by Moradzadeh
Moradzadeh replied the topic: Humongous sessions
Looks like an opportunity to optimize with a plug-in or a re-code.

Please Log in to join the conversation.

More
1 year 1 month ago #140990 by jelo
jelo replied the topic: Humongous sessions

jschreuder wrote: So this seems to be intended behaviour. I fear it makes LimeSurvey unfit for large-scale surveys with big questionnaires. Not complaining, I'm very happy with it for the small stuff. But it seems a pity as I think it's unnecessary to make sessions this huge.

Sessionconcept of Limesurvey is still a big malus point.
Even when using php session files you will have to keep an eye on the session generation with big surveys with a lot of users in a small timeframe. Each time someone hits the survey a 2MB file is generated in the php session directory. The default session directory is tmp on UNIX machines. With some hosting providers TMP is put on a secured, locked down partition with a size limit. So there might be a barrier hit, when having a lot of session files generated in a small timeframe.
The I/O burden can also be a pity. Having a SSD for the sessionfiles can help since the 2 MB file is directly written when the user hits the first surveypage.

Please Log in to join the conversation.

More
1 year 1 month ago #140992 by Moradzadeh
Moradzadeh replied the topic: Humongous sessions
I'm guessing that the original developers figured that the pre-written file would be a performance improvement over generating the survey on the fly for each user's visit to the site. For smaller surveys, they may be right.

For your case, I wonder whether breaking your humongous survey into perhaps three smaller ones might serve your purposes.

Please Log in to join the conversation.

More
1 year 1 month ago #141051 by DenisChenu
DenisChenu replied the topic: Humongous sessions

Moradzadeh wrote: ..... a re-code.

Yes for some part, this must needed.
But without breaking existing ....

And we must use Cache and not session for some other part ... session must only keep the answers.

Assistance on LimeSurvey forum and LimeSurvey core development are on my free time.
I'm not a LimeSurvey GmbH member, professional service on demand (or search sondages pro).
An error happen ? Before make a new topic : remind the Debug mode .

Please Log in to join the conversation.

More
10 months 1 week ago #145788 by bruce78
bruce78 replied the topic: Humongous sessions

DenisChenu wrote:

Moradzadeh wrote: ..... a re-code.

And we must use Cache and not session for some other part ... session must only keep the answers.


Hi Denis, are there any plans on adding caches/session caches to v. 3.0 at all?

Please Log in to join the conversation.

More
10 months 1 week ago #145843 by DenisChenu
DenisChenu replied the topic: Humongous sessions

bruce78 wrote: ....
Hi Denis, are there any plans on adding caches/session caches to v. 3.0 at all?

Not currently. And remind : we still use $_SESSIOn : it's one of the reason the first pacge can take a long long long time to call.

Assistance on LimeSurvey forum and LimeSurvey core development are on my free time.
I'm not a LimeSurvey GmbH member, professional service on demand (or search sondages pro).
An error happen ? Before make a new topic : remind the Debug mode .

Please Log in to join the conversation.

More
10 months 1 week ago #145851 by bruce78
bruce78 replied the topic: Humongous sessions

DenisChenu wrote: Not currently. And remind : we still use $_SESSIOn : it's one of the reason the first pacge can take a long long long time to call.


Interesting... btw, do you think disk sessions are faster/more reliable than db sessions, all things being equal?

Please Log in to join the conversation.

More
10 months 1 week ago #145913 by DenisChenu
DenisChenu replied the topic: Humongous sessions

bruce78 wrote:

DenisChenu wrote: Not currently. And remind : we still use $_SESSIOn : it's one of the reason the first pacge can take a long long long time to call.


Interesting... btw, do you think disk sessions are faster/more reliable than db sessions, all things being equal?

Depend ;)

If you can set session save path on a SSD or better in RAM ! file session is quickest sure.
If both are in HHD : file or SQL : i think file still quick ? because we use SQL for other than session
=> Maybe alternative soluton is to use another DB for session, for example a SQLite db (saved in RAM/SSD)

Assistance on LimeSurvey forum and LimeSurvey core development are on my free time.
I'm not a LimeSurvey GmbH member, professional service on demand (or search sondages pro).
An error happen ? Before make a new topic : remind the Debug mode .

Please Log in to join the conversation.

More
10 months 1 week ago #145916 by bruce78
bruce78 replied the topic: Humongous sessions

DenisChenu wrote: => Maybe alternative soluton is to use another DB for session, for example a SQLite db (saved in RAM/SSD)


That's a neat idea... do you think something like that might reduce my first page load time from 10 seconds to something less?

Please Log in to join the conversation.

More
10 months 1 week ago #145920 by DenisChenu
DenisChenu replied the topic: Humongous sessions

bruce78 wrote:

DenisChenu wrote: => Maybe alternative soluton is to use another DB for session, for example a SQLite db (saved in RAM/SSD)


That's a neat idea... do you think something like that might reduce my first page load time from 10 seconds to something less?

Not tested, the no idea ....
But think the best solution is to save session in RAM only or SSD. Maybe in a SQLite DB on RAM is the better way.

Denis

Assistance on LimeSurvey forum and LimeSurvey core development are on my free time.
I'm not a LimeSurvey GmbH member, professional service on demand (or search sondages pro).
An error happen ? Before make a new topic : remind the Debug mode .

Please Log in to join the conversation.

Start now!

Just create your account and start using Limesurvey today.

Register now