Export data to SPSS: missing participants in our database ?

More
4 years 6 days ago #94160 by EPRIS
Hello everyone,

I have tried exporting data from LS to SPSS. We have a large amount of participants (5043: 2503 completed questionnaires). When I look at my data on SPSS I don't get the correct amount of participants and everytime I try to export data I get a different number of participants in my database (once I had 1902, the other time I had 1867, and so on...). I have tried exporting the data separately (only completed questionnaires) but the problem still occured. We would like to know how we could fix this data export problem.

Thank you !

- Limesurvey Version 2.00+ Build 130317
- Our survey has 400 variables

(if you need more info to help you figure out our problem, please tell me)


Carl

Please Log in to join the conversation.

More
4 years 5 days ago #94161 by mdekker
You can not export participants to spss, only survey respsones. When a participant does not start the survey you can not see him in spss export. Only when he starts the survey it will be incomplete.

---
Menno Dekker

Please Log in to join the conversation.

More
4 years 5 days ago #94167 by EPRIS
Ok sorry I just didn't use the right words:

Hello everyone,

I have tried exporting data from LS to SPSS. We have a large amount of survey responses (5043: 2503 completed questionnaires). When I look at my data on SPSS I don't get the correct amount of survey responses and everytime I try to export data I get a different number of survey responses in my database (once I had 1902, the other time I had 1867, and so on...). I have tried exporting the data separately (only completed questionnaires) but the problem still occured. We would like to know how we could fix this data export problem.

Thank you !

- Limesurvey Version 2.00+ Build 130317
- Our survey has 400 variables

(if you need more info to help you figure out our problem, please tell me)


Carl

Please Log in to join the conversation.

More
4 years 3 days ago - 4 years 3 days ago #94201 by mdekker
Ok so we forget about the participants and tokens etc.

You go into the responses & statistics screen, in that screen you see an overview with three numbers: full, incomplete and total responses.

Then in the same screen you go to spss export and do the export. The numbers should match, and the only thing that could be different is what happended in the time between you looking at the response overview and you doing the export. So when someone is filling out the survey that will ofcourse change the numbers.

If the numbers in the response overview do not match that of the spss export, could you try exporting to excel and see what the numbers do there?

---
Menno Dekker
Last Edit: 4 years 3 days ago by mdekker.

Please Log in to join the conversation.

More
4 years 3 days ago #94205 by EPRIS
We were able to export all of our data when we had less survey responses.
The problem occured when we suddently had more than 2000 responses... Our
export was inconsistent with the survey responses we had.

For example, I tried to export all my completed survey responses (2200) but
when I opened the data file with SPSS I had only 1660. I tried another time
and it gave me 1590. It seems like I can't export all of these reponses
at once...

When I tried to export into Excel LimeSurvey crashed. I have tried exporting data using only one variable and it worked... so I guess this is linked to our massive database (with lots of survey responses).

Where do I go from here ?

Please Log in to join the conversation.

More
4 years 3 days ago #94206 by mdekker
You probably hit the php max execution time. Try if you can get that number up (use google to find out how)

---
Menno Dekker

Please Log in to join the conversation.

More
4 years 3 days ago #94211 by holch
As a quick solution you could try to export in steps of 1000 responses.

But the best solutions (for a long term) is probably what mdekker suggests.

I'm not a LimeSurvey GmbH member. I answer at the LimeSurvey forum in my spare time. No support via private message.
Some helpful links: Manual (EN) | Question Types | Workarounds

Please Log in to join the conversation.

More
4 years 2 days ago #94223 by DenisChenu
Think SPSS/R export don't have "restrict from/to id" and maybe some "restrict to this column".

Maybe future plugin ;).

Assistance on LimeSurvey forum and LimeSurvey core development are on my free time.
I'm not a LimeSurvey GmbH member, professional service on demand (or search sondages pro).

Please Log in to join the conversation.

More
4 years 2 days ago #94237 by holch
Ahhh, I didn't know that Denis!

I thought those functions would be available for all export. Thanks for letting me know!

I'm not a LimeSurvey GmbH member. I answer at the LimeSurvey forum in my spare time. No support via private message.
Some helpful links: Manual (EN) | Question Types | Workarounds

Please Log in to join the conversation.

More
3 years 11 months ago #94483 by EPRIS
Unfortunately the problem is still ongoing.

Complete cases: 3592
Incomplete: 3335
Total: 6927

The .sql file size is 15MO.


I don't think tht mem_limit or max_exec are in cause here.
extract from the php.ini file:
default_socket_timeout = 1200
max_execution_time = 2400
max_input_time = 1200
memory_limit = 1024M
post_max_size = 128M

The problem is related to exporting in SPSS or R. If I export in CSV, I can get all complete cases (3600), but not the total incomplete and complete (over 6K). However, I need an importation syntax (R or SPSS).

The .csv file for complete cases only is 6MO.
The maximum file size I can get with R or SPSS export is 2MO.

One solution would be to make available extraction by ID range (it would be awesome if this function, already available in the general export tab, was also made available for R or SPSS!).
Who that be possible?

Another solution would be to use the SPSS syntax to import the csv file. Unfortunetaly, the number of variables differ between what is extract in the .dat (SPSS) file and the .csv file. I get 400 variables in SPSS and 435 in the .csv file, so this is not easily implemented.

Is there any log of file I can join that could help identify the problem(s)?

Very many thanks for your input and suggestions!

Please Log in to join the conversation.

More
3 years 11 months ago #94485 by mdekker
Try to edit your /application/config.php
at the end there should be a 'debug'=>0
change the 0 to 1 or 2 and see if that gives a more clear error message

Only possible difference in number of fields I can imagine is timing fields. I think they are not present in the spss export.

The team is aware of problems with large exports and will address that in the future but I can not give an ETA for that feature at this moment.

---
Menno Dekker
The following user(s) said Thank You: EPRIS

Please Log in to join the conversation.

More
3 years 11 months ago #94501 by EPRIS
Thanks for your quick reply. I changed the parameters, I'll come back to you as soon as I have something...

As I really need the complete database soon... I was wondering if there is a way a can either
1) export cases by batch (e.g. by extracting first 1500ish cases, deleting them, then extracting the folling 1500ish cases, and so on? would that be possible?)
2) accelerate the process of solving the issue (like by making a donation)?

Many thanks!

Please Log in to join the conversation.