Custom Report Generation

More
3 years 6 months ago #108461 by KitchM
KitchM created the topic: Custom Report Generation
Because the reports I can find in the program are fairly useless to me, I need a custom report that shows the results from my election poll in the format I can use. Is there an add-on that creates custom reports? Is there any way to do this in any case?

Thanks.

Please Log in to join the conversation.

More
3 years 6 months ago #108517 by jelo
jelo replied the topic: Custom Report Generation
Export Data and use external software is the way to go. Most of the time the data needs to be checked and cleaned too. So don't expect AddOns in the reporting section. I find the report tool quite powerful. What is missing?

Please Log in to join the conversation.

More
3 years 6 months ago #108524 by tammo
tammo replied the topic: Custom Report Generation
We also have been searching a good report generator, since our needs were complicated we decided to write one ourselves.

One of the results, based on a Limesurvey questionnaire (in Dutch, sorry) is here:
boomenbos.toolsforresearch.com/tfr/ishc.php?savedid=61

The software we have written for this is not open source.


Tammo ter Hark at Tools for Research
Proud developers of the TFR Responsive Limesurvey template
toolsforresearch.com/limesurvey-responsive-template
The following user(s) said Thank You: Ben_V

Please Log in to join the conversation.

More
3 years 6 months ago #108526 by jelo
jelo replied the topic: Custom Report Generation
Very nice. Looks easy, but hard work. What is the technical approach.

Direct query of Limesurvey database?
Or export of Limesurvey results and import in your webbased reporting tool?

I wonder if LS 2.05 would allow you to offer a reporting addon for the commercial space?

BTW: Added "Kleine Stappen" to my very small collection of dutch words ;-)

Please Log in to join the conversation.

More
3 years 6 months ago #108527 by tammo
tammo replied the topic: Custom Report Generation
We did this using javascript, based on querying the limesurvey database, so real time reports are also possible. We also used this for a survey with now about 28.000 complete responses.

Also see: Demo.gewoongoedwerkmeter.nl, which generates a report at the end (after many mandatory questions about how employees rate their employer). The Demo prefix makes it easy to participate without registering and without storing the Demo-answers in the actual database.

It was done on basis of version 1.92, we are now planning the migration to 2.05 this summer.

Limesurvey is great, but left us amazed at certain moments. We are now developing a fully responsive theme design.


Tammo ter Hark at Tools for Research
Proud developers of the TFR Responsive Limesurvey template
toolsforresearch.com/limesurvey-responsive-template

Please Log in to join the conversation.

More
7 months 5 days ago - 7 months 5 days ago #152495 by Kevin
Kevin replied the topic: Custom Report Generation
This took me a week to complete because I'm not a database person. It should help people produce reports without exporting the data to excel and creating a pivot table.

CREATE DEFINER=`A_VALID_USER_NAME_HERE`@`localhost` PROCEDURE `usp_produce_survey_report`(IN survey_id VARCHAR(10), IN lang VARCHAR(2))
    SQL SECURITY INVOKER
BEGIN
 
/*---------------------------------------------------------------------------------
I do not guarantee that this will work for you or that it cannot be hacked with
with SQL injections or other malicious intents. 
 
This stored procedure will produce output that you may use to create a report.
It accepts two arguments; The survey id (745) and the language (en).
It parses the column name in the survey table to get the qid.
 
It will copy the answers from the survey table to the survey_report
table if the answer is type S or K. It will get the answers from
the answers table for other types. NOTE: Other types might need to
be added to the if statement.
 
Additionally, the qid and id from the survey table are also copied to
the survey_report table.
 
Then the questions from the questions table, and answers from the answers
and survey_report tables are combined and displayed.  
 
The data in the survey_report table is deleted after the data is displayed.
 
The id from the survey table is displayed as the respondent_id which may
be used to combine the questions and answers from a specific respondent.  
 
You may have to change the prefix on the table names.
Example: survey_answers to my_prefix_answers.
 
Use this to call the procedure.
Syntax:  call survey.usp_produce_survey_report('<SURVERY_ID>', '<LANGUAGE>');
Example: call survey.usp_produce_survey_report('457345', 'en'); 
 
use this to create the table that stores the data
 CREATE TABLE `survey_report` (
    `id` int(11) NOT NULL AUTO_INCREMENT,
    `qid` int(11) NOT NULL DEFAULT '0',
    `survey_row_id` int(11) NOT NULL DEFAULT '0' COMMENT 'id that is in the survey_<id> table',
    `answer` mediumtext COLLATE utf8mb4_unicode_ci DEFAULT NULL,
   PRIMARY KEY (`id`)
);
*/
 
DECLARE v_col_name VARCHAR (25);
DECLARE v_qid INT;
DECLARE v_col_count INT DEFAULT 0;
DECLARE done INT DEFAULT false;
DECLARE tname VARCHAR(24) DEFAULT CONCAT('survey_survey_',survey_id);
DECLARE counter INT DEFAULT 0;
DECLARE x INT DEFAULT 0;
 
-- select locate ('X','123457X212X1125', 8);  -- use locate to determine location of second X - returns 11
-- select substring('123457X212X1125', 11+1, 7); -- use substring to get the qid - returns 1125
 
DECLARE cur1 cursor for
    SELECT column_name, substring(column_name, 11+1, 7) as qid -- get the qid from the column name.  the 7 might need to be higher depending on the id.  
       FROM information_schema.columns -- this has the column names
       WHERE table_name = tname -- table name created from the id that was passed to the stored procedure
       AND column_name REGEXP 'X'; -- get the columns that have an X
 
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
SET done = FALSE;
 
OPEN cur1;
SELECT FOUND_ROWS() INTO v_col_count; -- Get number of column names
   -- SELECT v_col_count; -- print the number of column names
   read_loop: LOOP
      FETCH cur1 INTO v_col_name, v_qid; -- v_col_name is the original column name and v_qid is the qid that is taken from the column name
 
      IF done THEN
         LEAVE read_loop;
      END IF;
      -- SELECT v_col_name, v_qid;   
      SET counter = 1; -- use to compare id's
      SET x = 1; -- used for the while loop
 
      WHILE x < v_col_count DO
         SET @sql := NULL;
         -- SELECT v_col_name, v_qid, counter, x; 
         -- SELECT counter as id, v_col_name, v_qid as qid, x; 	  
         -- SET @sql = CONCAT ('SELECT id ', ',',v_qid, ' as qid ,', v_col_name,' FROM ', tname, ' WHERE id = ', counter );
         -- I would have to join the survey table below if I did not add the answer (v_col_name). I assume this is faster than another join.
         SET @sql = CONCAT ('INSERT INTO survey_report(qid,survey_row_id,answer) SELECT ',v_qid, ',id,' , v_col_name, '  FROM ', tname, ' WHERE id = ', counter );
 
         -- SELECT @sql;       
         PREPARE stmt FROM @sql;
         EXECUTE stmt;
         DEALLOCATE PREPARE stmt;
         -- SELECT counter, x; 
         SET x = x + 1; -- increment counter for while loop
         SET counter = counter + 1; -- increment counter for id's
      END WHILE;
   END LOOP; -- read_loop
   CLOSE cur1;
 
   -- SELECT * FROM survey_report
   -- ORDER BY id, qid;
 
   SET @counter = 0;
 
   SELECT 
      -- @counter:=@counter + 1 AS newindex, -- increment the counter that is in the header
      -- survey_report.id AS id,
      survey_report.survey_row_id as respondent_id, -- the id that copied from the survey table
      -- survey_report.qid,
      question,
      IF(type IN ('S' , 'K'),
         (SELECT answer
            FROM survey_report
            WHERE qid NOT IN (SELECT qid
                    FROM survey_answers)
                    AND id = @counter),
        (SELECT answer
            FROM survey_answers
            WHERE survey_questions.qid = survey_answers.qid
                    AND survey_report.qid = survey_questions.qid
                    AND survey_report.answer = survey_answers.code
                    AND survey_answers.language = lang)) AS answer
            FROM survey_questions
                    JOIN survey_report ON survey_report.qid = survey_questions.qid
                    WHERE survey_questions.sid = survey_id
					   AND survey_questions.language = lang
                    ORDER BY survey_report.survey_row_id;
 
   TRUNCATE TABLE survey_report;    
END
Last Edit: 7 months 5 days ago by Kevin.

Please Log in to join the conversation.

More
7 months 5 days ago - 7 months 5 days ago #152500 by Kevin
Kevin replied the topic: Custom Report Generation
The solution returned 2,838 rows (43 respondents) but the original survey table has 28,994 rows and respondents. I did not get an error but I wonder if there is an issue with memory? The tested survey table has 44 columns for answers for approximately 33 questions.

It worked ok on a previous test with only a few questions and answers.
Last Edit: 7 months 5 days ago by Kevin.

Please Log in to join the conversation.

More
7 months 4 days ago - 7 months 4 days ago #152518 by Kevin
Kevin replied the topic: Custom Report Generation
CREATE DEFINER=`some_user`@`localhost` PROCEDURE `usp_produce_survey_report`(IN survey_id VARCHAR(10), IN lang VARCHAR(2))
    SQL SECURITY INVOKER
BEGIN
 
/*---------------------------------------------------------------------------------
I do not guarantee that this will work for you or that it cannot be hacked with
with SQL injections or other malicious intents. 
 
This stored procedure will produce output that you may use to create a report.
It accepts two arguments; The survey id (745) and the language (en).
It parses the column name in the survey table to get the qid.
 
It will copy the answers from the survey table to the survey_report
table if the answer is type S or K. It will get the answers from
the answers table for other types. NOTE: Other types might need to
be added to the if statement.
 
Additionally, the qid and id from the survey table are also copied to
the survey_report table.
 
Then the questions from the questions table, and answers from the answers
and survey_report tables are combined and displayed.  
 
The data in the survey_report table is deleted after the data is displayed.
 
The id from the survey table is displayed as the respondent_id which may
be used to combine the questions and answers from a specific respondent.  
 
You may have to change the prefix on the table names.
Example: survey_answers to my_prefix_answers.
 
Use this to call the procedure.
Syntax:  call survey.usp_produce_survey_report('<SURVERY_ID>', '<LANGUAGE>');
Example: call survey.usp_produce_survey_report('457345', 'en'); 
 
use this to create the table that stores the data
 CREATE TABLE `survey_report` (
    `id` int(11) NOT NULL AUTO_INCREMENT,
    `qid` int(11) NOT NULL DEFAULT '0',
    `survey_row_id` int(11) NOT NULL DEFAULT '0' COMMENT 'id that is in the survey_<id> table',
    `answer` mediumtext COLLATE utf8mb4_unicode_ci DEFAULT NULL,
   PRIMARY KEY (`id`)
);
*/
 
DECLARE v_col_name VARCHAR (25);
DECLARE v_qid INT;
DECLARE v_col_count INT DEFAULT 0;
DECLARE done INT DEFAULT false;
DECLARE tname VARCHAR(24) DEFAULT CONCAT('survey_survey_',survey_id);
DECLARE counter INT DEFAULT 0;
DECLARE x INT DEFAULT 0;
DECLARE r_count INT DEFAULT 0;
 
-- select locate ('X','123457X212X1125', 8);  -- use locate to determine location of second X - returns 11
-- select substring('123457X212X1125', 11+1, 7); -- use substring to get the qid - returns 1125
 
DECLARE cur1 cursor for
    SELECT column_name, substring(column_name, 11+1, 7) as qid -- get the qid from the column name.  the 7 might need to be higher depending on the id.  
       FROM information_schema.columns -- this has the column names
       WHERE table_name = tname -- table name created form the id that was passed to the stored procedure
       AND column_name REGEXP 'X'; -- get the columns that have an X
 
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
SET done = FALSE;
 
   OPEN cur1;
   -- SELECT FOUND_ROWS() INTO v_col_count; -- Get number of column names
   -- SELECT v_col_count; -- print the number of column names
   SET r_count = (SELECT COUNT(*) FROM survey_survey_144477);  
   -- SELECT r_count;
 
   read_loop: LOOP
      FETCH cur1 INTO v_col_name, v_qid; -- v_col_name is the original column name and v_qid is the qid that is taken from the column name
 
      IF done THEN
         LEAVE read_loop;
      END IF;
      -- SELECT v_col_name, v_qid;   
      SET counter = 1; -- use to compare id's
      SET x = 1; -- used for the while loop
 
      WHILE x <= r_count DO
         SET @sql := NULL;
         -- SELECT v_col_name, v_qid, counter, x; 
         -- SELECT counter as id, v_col_name, v_qid as qid, x;    
         -- SET @sql = CONCAT ('SELECT id ', ',',v_qid, ' as qid ,', v_col_name,' FROM ', tname, ' WHERE id = ', counter );
         -- I would have to join the survey table below if I did not add the answer (v_col_name). I assume this is faster than another join.
         SET @sql = CONCAT ('INSERT INTO survey_report(qid,survey_row_id,answer) SELECT ',v_qid, ',id,' , v_col_name, '  FROM ', tname, ' WHERE id = ', counter );
 
         -- SELECT @sql;       
         PREPARE stmt FROM @sql;
         EXECUTE stmt;
         DEALLOCATE PREPARE stmt;
         -- SELECT counter, x; 
         SET x = x + 1; -- increment counter for while loop
         SET counter = counter + 1; -- increment counter for id's
      END WHILE;
   END LOOP; -- read_loop
   CLOSE cur1;
 
   -- SELECT * FROM survey_report
   -- ORDER BY id, qid;
 
   SET @counter = 0;
 
   SELECT 
      @counter:=@counter + 1 AS newindex, -- increment the counter that is in the header
      survey_report.id,
      survey_report.survey_row_id as respondent_id, -- the id that copied from the survey table
      survey_report.qid,
      question,
      IF(type IN ('S' , 'K'),
         (SELECT answer
            FROM survey_report
            WHERE qid NOT IN (SELECT qid FROM survey_answers)
            AND survey_questions.language = lang
                  AND survey_report.id = @counter),
         (SELECT answer
            FROM survey_answers
            WHERE survey_questions.qid = survey_answers.qid
                  AND survey_report.qid = survey_questions.qid
                  AND survey_report.answer = survey_answers.code
                  AND survey_answers.language = lang
             )
          ) AS answer
       FROM survey_questions
          JOIN survey_report ON survey_report.qid = survey_questions.qid
          WHERE survey_questions.sid = survey_id
          ORDER BY survey_report.survey_row_id, survey_report.id;
 
   TRUNCATE TABLE survey_report;    
END
Last Edit: 7 months 4 days ago by Kevin. Reason: Changed the while loop variable and the location of the language condition.

Please Log in to join the conversation.

More
7 months 4 days ago #152523 by Kevin
Kevin replied the topic: Custom Report Generation
I'm posting this again because I cannot edit the previous post. I got the number of rows from the schema and changed the loop variables.
CREATE DEFINER=`some_user`@`localhost` PROCEDURE `usp_produce_survey_report`(IN survey_id VARCHAR(10), IN lang VARCHAR(2))
    SQL SECURITY INVOKER
BEGIN
 
/*---------------------------------------------------------------------------------
I do not guarantee that this will work for you or that it cannot be hacked with
with SQL injections or other malicious intents. 
 
This stored procedure will produce output that you may use to create a report.
It accepts two arguments; The survey id (745) and the language (en).
It parses the column name in the survey table to get the qid.
 
It will copy the answers from the survey table to the survey_report
table if the answer is type S or K. It will get the answers from
the answers table for other types. NOTE: Other types might need to
be added to the if statement.
 
Additionally, the qid and id from the survey table are also copied to
the survey_report table.
 
Then the questions from the questions table, and answers from the answers
and survey_report tables are combined and displayed.  
 
The data in the survey_report table is deleted after the data is displayed.
 
The id from the survey table is displayed as the respondent_id which may
be used to combine the questions and answers from a specific respondent.  
 
You may have to change the prefix on the table names.
Example: survey_answers to my_prefix_answers.
 
Use this to call the procedure.
Syntax:  call survey.usp_produce_survey_report('<SURVERY_ID>', '<LANGUAGE>');
Example: call survey.usp_produce_survey_report('457345', 'en'); 
 
use this to create the table that stores the data
 CREATE TABLE `survey_report` (
    `id` int(11) NOT NULL AUTO_INCREMENT,
    `qid` int(11) NOT NULL DEFAULT '0',
    `survey_row_id` int(11) NOT NULL DEFAULT '0' COMMENT 'id that is in the survey_<id> table',
    `answer` mediumtext COLLATE utf8mb4_unicode_ci DEFAULT NULL,
   PRIMARY KEY (`id`)
);
*/
 
DECLARE v_col_name VARCHAR (25);
DECLARE v_qid INT;
DECLARE v_col_count INT DEFAULT 0;
DECLARE done INT DEFAULT false;
DECLARE tname VARCHAR(24) DEFAULT CONCAT('survey_survey_',survey_id);
DECLARE counter INT DEFAULT 0;
DECLARE current_row INT DEFAULT 0;
DECLARE total_rows INT DEFAULT 0;
 
-- select locate ('X','123457X212X1125', 8);  -- use locate to determine location of second X - returns 11
-- select substring('123457X212X1125', 11+1, 7); -- use substring to get the qid - returns 1125
 
DECLARE cur1 cursor for
    SELECT column_name, substring(column_name, 11+1, 7) as qid -- get the qid from the column name.  the 7 might need to be higher depending on the id.  
       FROM information_schema.columns -- this has the column names
       WHERE table_name = tname -- table name created form the id that was passed to the stored procedure
       AND column_name REGEXP 'X'; -- get the columns that have an X
 
DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;
SET done = FALSE;
 
OPEN cur1;
 
   SET total_rows = (SELECT table_rows -- get the number of rows
      FROM INFORMATION_SCHEMA.TABLES
      WHERE table_name = tname);
   -- SELECT total_rows;
 
   read_loop: LOOP
      FETCH cur1 INTO v_col_name, v_qid; -- v_col_name is the original column name and v_qid is the qid that is taken from the column name
 
      IF done THEN
         LEAVE read_loop;
      END IF;
      -- SELECT v_col_name, v_qid;   
      SET counter = 1; -- use to compare id's
      SET current_row = 1; -- used for the while loop
 
      WHILE current_row <= total_rows DO
         SET @sql := NULL;
         -- SELECT v_col_name, v_qid, counter, x; 
         -- SELECT counter as id, v_col_name, v_qid as qid, x;    
         -- SET @sql = CONCAT ('SELECT id ', ',',v_qid, ' as qid ,', v_col_name,' FROM ', tname, ' WHERE id = ', counter );
         -- I would have to join the survey table below if I did not add the answer (v_col_name). I assume this is faster than another join.
         SET @sql = CONCAT ('INSERT INTO survey_report(qid,survey_row_id,answer) SELECT ',v_qid, ',id,' , v_col_name, '  FROM ', tname, ' WHERE id = ', counter );
 
         -- SELECT @sql;       
         PREPARE stmt FROM @sql;
         EXECUTE stmt;
         DEALLOCATE PREPARE stmt;
         -- SELECT counter, x; 
         SET current_row = current_row + 1; -- increment counter for while loop
         SET counter = counter + 1; -- increment counter for id's
      END WHILE;
   END LOOP; -- read_loop
   CLOSE cur1;
 
   -- SELECT * FROM survey_report
   -- ORDER BY id, qid;
 
   SET @counter = 0;
 
   SELECT 
      @counter:=@counter + 1 AS newindex, -- increment the counter that is in the header
      survey_report.id,
      survey_report.survey_row_id as respondent_id, -- the id that copied from the survey table
      survey_report.qid,
      question,
      IF(type IN ('S' , 'K'),
         (SELECT answer
            FROM survey_report
            WHERE qid NOT IN (SELECT qid FROM survey_answers)
            AND survey_questions.language = lang
                  AND survey_report.id = @counter),
         (SELECT answer
            FROM survey_answers
            WHERE survey_questions.qid = survey_answers.qid
                  AND survey_report.qid = survey_questions.qid
                  AND survey_report.answer = survey_answers.code
                  AND survey_answers.language = lang
             )
          ) AS answer
       FROM survey_questions
          JOIN survey_report ON survey_report.qid = survey_questions.qid
          WHERE survey_questions.sid = survey_id
          ORDER BY survey_report.survey_row_id, survey_report.id;
 
   TRUNCATE TABLE survey_report;    
END

Please Log in to join the conversation.

Start now!

Just create your account and start using Limesurvey today.

Register now
Join our Newsletter!