Check out the LimeSurvey source code on GitHub!
Welcome, Guest
Username: Password:

TOPIC: GSOC 2010: File Upload Question Type

GSOC 2010: File Upload Question Type 6 years 4 months ago #41860

  • texens
  • texens's Avatar
  • Offline
  • Junior Lime
  • Posts: 24
  • Thank you received: 1
  • Karma: 0
Hi,

I'm Amit Shanker (texens), a Senior Undergraduate student pursuing Bachelor of Technology in Computer Science and Engineering at Indian Institute of Technology Kharagpur (IIT Kgp).

Basically from central India, I've spent a good part of my teenage years in New Delhi and will be moving to the Indian silicon valley - Bangalore very soon.

This summers, I'll be working on the much demanded File upload Question type for Limesurey 1, and the proposal document for the same can be found here. I'll be working under the mentorship of Carsten Schmitz. Marcel Minke will be the co-mentor for the project.
I'm very excited about this project and am looking forward to your comments and feedback on the proposal. Please feel free to post in any comments on the aforementioned link.

Looking forward to an awesomely awesome summer with Limesurvey \m/ \m/
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 4 months ago #41864

  • jelo
  • jelo's Avatar
  • Offline
  • Platinum Lime
  • Posts: 1352
  • Thank you received: 172
  • Karma: 52
Hello Amit,

thanks for your post and for taking the time to contribute to LimeSurvey.
Keep my fingers crossed that you will still have some fun while coding ;-)

Since you asked for feedback and thoughts in this early stage:

Uploading a file is a nice feature for the good and bad guys.

Security is a process as you stated on your project page. And in the process more and more omnipotent php functions are disabled on public running php stacks. Via disable_functions in the php.ini you often find these routines disabled: exec,passthru,shell_exec,system,proc_open,popen,curl_exec,curl_multi_exec,parse_ini_file,show_source, phpinfo

Another security layer more and more used is Suhosin ( www.hardened-php.net/suhosin/index.html ).

When planning the upload and save part of the files it would be nice to try the least powerful functions terms of abuse potential.

I would' look at the upload routines of bugtrackers but more on the upload routines of gallerys and discussion boards, which have a bigger installed base and are more likely to be attacked.

A simple way to secure the uploaded content from webacess it to move it out of the webroot (a simple additional path in the configfile would to the trick). A viewing routine would be more complicated because no direct url access to the file would be possible. But since the focus is on the upload and not on exposing the file to many concurrent viewer (e.g. gallery) I don't see the additional payload of a viewing routine as a performance killer.

Just a few raw thoughts. Hope they make sense. If not, feel free to punish me with words ;-)

P.S.
What is in your opinion the best english speaking website for news coverage about/from India?
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 4 months ago #41914

  • texens
  • texens's Avatar
  • Offline
  • Junior Lime
  • Posts: 24
  • Thank you received: 1
  • Karma: 0
Thanks jelo, for the quick feedback :)

Indeed, File system security is of utmost importance and one of the major challenges of this project.

Yet another factor is that we have to consider the fact that a lot of people use shared hosting services and hence they might not have access to a lot of things that usually dedicated servers have. Hence, moving the uploaded files outside the webroot might not be a possibility. We can make the directory inaccessible for any outsiders by setting the right properties in the .htaccess file though. I haven't digged into suhosin yet, but a quick looks gives the impression that its a patch for PHP and hence may not be feasible, yet again due to the aforementioned reason.

While everyone (given he is eligible to participate in the survey) can upload, only admin/privileged users can browse/download the files. So, the focus is going to be on uploading the files, storing them in a temp folder unless the files are submitted, and then keeping them safe from any evil users after they are submitted. The real trouble comes when we execute third party tools on the server for editing the uploaded files. During this period, a malicious file could exploit security loopholes in the third party tool. Hence, this is one area where we're going to need a lot of brainstorming and feedback from everyone.

I'll be regularly updating the proposal with all the ideas that we come up with for the project.
Thanks again for the feedback :)
What is in your opinion the best english speaking website for news coverage about/from India?
I love to read the Times of India, but Hindustan Times is equally popular in India.
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 4 months ago #42011

  • Mazi
  • Mazi's Avatar
  • Offline
  • LimeSurvey Team
  • Posts: 6009
  • Thank you received: 370
  • Karma: 260
At the proposal which can be found in our wiki at docs.limesurvey.org/tiki-index.php?page=...Upload+Question+Type the idea to add 3rd party hooks e.g. to convert uploaded images into a certain format.

We are interested in user feedback on what kind of 3rd party hooks you are interested in!
Opinions?
Suggestions?

Best regards/Beste Grüße,
Dr. Marcel Minke
(Limesurvey Head of Support)
Need Help? We offer professional Limesurvey support
Contact: marcel.minke(at)limesurvey.org'"
The administrator has disabled public write access.

Project Status and Weekly Progress Report 6 years 4 months ago #43452

  • texens
  • texens's Avatar
  • Offline
  • Junior Lime
  • Posts: 24
  • Thank you received: 1
  • Karma: 0
This week saw quite some coding and I am glad to inform you that the Simple Browser Uploader has been implemented and committed to the GSoC branch. (limesurvey_dev)

With this implementation, we can do the following:

A survey administrator can create surveys that have file upload question types, and activate them.
* A surveyee can upload one or many files in response to the File upload question type using the browser.
* The files uploaded by the surveyee are saved into the file system.
* The file meta-data such as title, comments, file name etc are saved as a JSON string into the database.
* create a map of the fields(fieldmap), that this question type has. This fieldmap will be used for implementing various functionalities such as data entry and response browsing.

This implementation still lacks the following:

* Form validation to allow surveyee to upload only those file types that have been permitted by the survey administrator.
* Form validation to restrict the number of files that can be uploaded by the surveyee.

We can say that more or less, the base for the file upload question type is all set. Now, we'll be using this base to implement all other functionalities such as statistics, data entry, response browsing, import export etc.

The target for the next iteration (May 24 - May 31) is the Advanced File uploader. The motivation behind this advanced file uploader is to give a fancy frontend to the file upload procedure, and also give feedback to the surveyee regarding the progress of the upload with features like progress bar, etc. It also adds a couple more features and the mockups and details for the same can be found on the FUQT proposal page: docs.limesurvey.org/tiki-index.php?page=...estion+Type#Uploader

Please feel free to drop back any comments on the mailing list or on the support forums. I am looking forward to feedback from the community on the aforementioned work :)
The administrator has disabled public write access.

Design/CSS for Advanced File Uploader 6 years 3 months ago #43645

  • texens
  • texens's Avatar
  • Offline
  • Junior Lime
  • Posts: 24
  • Thank you received: 1
  • Karma: 0
Hello,

The front-end for the Advanced File Uploader has started taking some shape. As mentioned in my last weekly report, the back-end for the upload has already been completed. Also, a simple form has been created to upload the files.

This Advanced File uploader will serve the following purposes:

* Facility to preview the uploaded files
* Edit the file's meta-data after uploading the files
* If the surveyee feels that he doesn't want to submit a file after uploading it, he can delete that file from the gallery tab.

This Advanced File uploader has 3 tabs:

1. Upload from Computer
* Contains the actual form that is used to upload the files
* Since, the files that will be uploaded will in most cases be documents and/or images, and won't be very big, the upload process won't be very long. And hence, I've thought of replacing the progress bar (as in the proposal) with a simple loading animation. We can easily switch to the "progress bar" thingy instead of the "loading" animation, but the upload will be so fast that one won't be able to see the progress bar actually 'progressing'. I'd really appreciate a feedback on this topic from everyone
2. Upload from URL
* The surveyee can give the URL of a document on the web, instead of uploading it off his own hard drive
3. Uploaded files Gallery
* Once the files have been uploaded, the surveyee can preview all the uploaded files for that particular question, edit the file's meta-data, or even delete them. It will also have a submit button from where he can submit all the uploaded files for that particular question.

While "1. Upload from Computer" has already been implemented, I'm working on the "2. Upload from URL" and "3. Uploaded files Gallery".

The File Upload question in the survey will have a simple upload button. On clicking this button, the Advanced File Uploader will come up in a shadowbox.

I'm maintaining a copy of this uploader at the following address: texens.5gigs.net/uploader/survey.php
Please click on the Upload link on the page, and you'll be able to see the shadowbox and its contents, along with the tabs on the top. (No, they don't look like tabs because we don't have the tab button images yet.)

We need to beautify this shadowbox. and I'd like to invite suggestions for designs from the Community for the same. I'm looking for the following things in particular:

1. tab button images for the menu
2. design of the "upload from computer" page and
3. desing of the "gallery" page
4. "upload" button image on the "upload from computer" page
5. "submit" button on the "gallery"page.

We can discuss the designs here on the forum or you can reply to my email on the LimeSurvey mailing list on the aforementioned topic. You can also catch me on IRC (freenode #limesurvey) where I go by the nick texens.

Looking forward to your feedback,
Amit
Last Edit: 6 years 3 months ago by texens.
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 3 months ago #43689

  • CarbonaCat
  • CarbonaCat's Avatar
  • Offline
  • Fresh Lemon
  • Posts: 3
  • Karma: 0
My two cents:
-> As you said, pictures will be uploaded pretty often. Will there be a way to limit the picture width/height, and/or a tool to resize/crop the picture?
-> KISS: Keep It Simple, Stupid. I think it's best if the surveyee can only upload a single file, review it and confirm it as well. If the uploader's UI is too complicated for him, the surveyee will be scared and will either ignore the question or stop the survey. I think it's a critical point...
- Also... When a surveyee is uploading a file, does he have to wait until it's done to be able to answer other questions?
- For the design: How will the file uploader integrate with the current template? Will it be customizable?

Also, some technical remarks, but you don't have to take them in account since this is only a prototype:
- On Chrome, it's empty, and on IE7 a javascript error. I assume you're making your devs with Firefox?
- There's some "document.getElementById" when I open the generated source code... Would'nt it be best to use jQuery for this?
- Will it be gracefully degradable?

I'm looking forward for this dev :)

Kind regards,
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 3 months ago #43700

  • texens
  • texens's Avatar
  • Offline
  • Junior Lime
  • Posts: 24
  • Thank you received: 1
  • Karma: 0
CarbonaCat wrote:
My two cents:
-> As you said, pictures will be uploaded pretty often. Will there be a way to limit the picture width/height, and/or a tool to resize/crop the picture?
Yes, we will have hooks for third party tools to process the files post uploading. Resize/Crop is indeed a very basic requirement and will be inbuilt in the Advanced File Uploader. We'd like to invite suggestions on more such 3rd party tools that might be helpful for the surveyee as well as the survey administrator.
-> KISS: Keep It Simple, Stupid. I think it's best if the surveyee can only upload a single file, review it and confirm it as well. If the uploader's UI is too complicated for him, the surveyee will be scared and will either ignore the question or stop the survey. I think it's a critical point...
Once the files have been uploaded, the surveyee will be redirected to the gallery page where he can review the file's metadata such as title, comments etc. In case of images etc, he can also preview the image, resize, or crop it. Once, he's done with all this, he can press the submit button on this page; the shadowbox will close and the surveyee will be taken back to the survey page from where he entered the shadowbox.
I should have the complete Advanced Uploader in a couple days. We can give it a test run, and depending on the feedback from the community, we can modify the user interface if it seems to be too complicated to use.
- Also... When a surveyee is uploading a file, does he have to wait until it's done to be able to answer other questions?
I opine that the surveyee should wait until the file(s) are uploaded.
Indeed, we can save surveyee's time if we allow her/him to take up other questions while the files are being uploaded. But, there might be a lot of scenarios where we'd like to wait for the surveyee to finish uploading his files.
Take, for example, while uploading, there can be issues such as file size restrictions, allowed file type restrictions, disk space issues and other errors. And the next question might be dependent on this file upload type question, or this file upload type question might be the last question of the survey. Hence, it's a tradeoff between the two. But, sure thing its open for discussion and I'd like to know what others feel about this issue.
- For the design: How will the file uploader integrate with the current template? Will it be customizable?
Yes, the file upload question type, like all other question types will indeed be customizable.
Also, some technical remarks, but you don't have to take them in account since this is only a prototype:
- On Chrome, it's empty, and on IE7 a javascript error. I assume you're making your devs with Firefox?
Its not working on chrome due to CSS issues; we are yet to do the styling for the upload page. Once we have the CSS, the problems should get fixed without much effort. I haven't tried it on IE yet and will test it only after I'm done with the CSS. Yes, I'm using Firefox for all the development and testing for this project.
- There's some "document.getElementById" when I open the generated source code... Would'nt it be best to use jQuery for this?
I'm basically a server-side guy and this is my first client side code. I learned javascript and jQuery recently; 3 days back to be more precise ;-)
At first look it seems to be a nice idea to use jQuery library, but I'm not sure about the performance and compatibility issues of both of them. I'll digg into it and also discuss it at length with my mentor and colleagues at the dev meeting and accordingly finalize on this issue.
I'm looking forward for this dev :)
Thanks a lot for your feedback and suggestions :) We should have this advanced file uploader up and running in a couple days. Once its complete, I'll post it right here so that everyone can try it out and give me some feedback, especially on the usability and user interface. The aim is to make the entire upload procedure very smooth and flexible for the surveyee, but definitely it has to be easy as well, 'cause the last thing we want is to scare away our surveyee ;-)
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 3 months ago #43747

  • Mazi
  • Mazi's Avatar
  • Offline
  • LimeSurvey Team
  • Posts: 6009
  • Thank you received: 370
  • Karma: 260
Also... When a surveyee is uploading a file, does he have to wait until it's done to be able to answer other questions?
We should wait for the upload process to be finished. In general, when writing the documentation we should add a note like "When allowing users to upload large files we recommend to add these file upload questions at the end."

Best regards/Beste Grüße,
Dr. Marcel Minke
(Limesurvey Head of Support)
Need Help? We offer professional Limesurvey support
Contact: marcel.minke(at)limesurvey.org'"
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 3 months ago #43757

  • texens
  • texens's Avatar
  • Offline
  • Junior Lime
  • Posts: 24
  • Thank you received: 1
  • Karma: 0
- There's some "document.getElementById" when I open the generated source code... Would'nt it be best to use jQuery for this?
LimeSurvey1 already ships with jQueryUI library and hence, we'll be using jQueryUI library for all the client side scripts including the tabs and hiding/showing animation during upload process. I'm replacing all raw javascripts with jQureyUI functions in the uploader.
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 3 months ago #44135

  • texens
  • texens's Avatar
  • Offline
  • Junior Lime
  • Posts: 24
  • Thank you received: 1
  • Karma: 0
Hi,

The Advanced File Uploader has been implemented and is fully functional, well almost fully functional ;-)

So, Now one can do the following using the advanced file uploader:
    * Upload one or more files by just the click of a single button
    * Define Title and add comments to the file
    * Once uploaded, the system lets you know that the file has been uploaded by giving a nice notice.
    * Once the files have been uploaded, the surveyee has 2 options -

    1. go to the "Gallery" Tab
    2. upload more files on the "From Computer" Tab

    * In the Gallery Tab, one can

    1. Preview image files. Support for other file types may be added later
    2. Edit the metaadata (title/comment) of the uploaded files
    3. delete uploaded files

    * Once, the surveyee is satisfied that he doesn't want to make any further modifications to the uploaded files, he can close the dialog, and return to the survey page.

The following still needs to be done with regard to the Advanced File Uploader.

* Post-uploading, when the surveyee returns to the survey page from the Advanced File uploader, he should see the list of uploaded files and small delete icons next to them for ease of use.

The uploader works, but looks a little ugly, as we are yet to work on the template. Also, there are a couple of form fields for debugging purposes and are going to stay there for a while during the development period. These will be set to hidden once the uploader becomes stable.

So, go ahead and give it a shot !
You can grab a copy of this dev version with File Upload Question Type at
limesurvey.svn.sourceforge.net/svnroot/l...ource/limesurvey_dev

Select 'File Upload Type' in the question type while creating a question. You will have to create the directory "limesurvey_dev/upload/tmp" and give write permissions. If you come across any bugs, please report them at bugs.limesurvey.org and assign them to 'texens'.

The targets for the upcoming iteration (June 8 to June 15) are:

Implementation in

* Data Entry
* Response Browsing
* Response Editing
* Zipper/Compressor (to compress the uploaded files)
* Downloader (download the uploaded files - after compression)


Looking forward to your feedback and comments,
Amit
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 3 months ago #44167

  • CarbonaCat
  • CarbonaCat's Avatar
  • Offline
  • Fresh Lemon
  • Posts: 3
  • Karma: 0
Live testing :) The underlined parts are what I was not expecting or suggestions.

1. I have updated my work copy of the /dev repository.
2. I created a new question of type "Upload file". I selected the following advanced parameters: one file of type "*.png" have to be submitted.
3. Tried to test the question alone... However, it ends up with a 404 not found error on the following URL: "/limesurvey_dev/admin/uploader.php". Could you check it why?
4. Testing the whole survey seems to work. I'm under Google Chrome 5.0.
5. I'm uploading a file (the active icon, active.png) with a title and a comment, I click on "Upload" (Not touching to the other texts fields), there is a progress bar...
6. Then it says it worked! However, although I did upload the requested file, it didn't automatically switch to the gallery view so I had to click.
7. YAY I CAN SEE IT
8. Trying to update title & comment... I click on Save change, but there is no confirmation if it was really saved.
9. I click on delete... it disappeared, nice!
10. Going back to the "From computer" tab.
11. Browsing file in order to upload "junk.txt"... I think it would be nice if it rejected the file as soon as I chosen a wrong one (Remember that I have chosen *.png filter).
12. I haven't filled the title and comment fields.
13. Uploading... It says it worked... filter not working yet?
14. Checking in the Gallery... There is a broken image frame as it tries to display the junk.txt as an image.
15. In this case, there is no any information about what the file is. I think it would be nice to put somewhere the filename...
16. Closing the window and submitting the page.
17. The *.png filter didn't work so I guess the validation is not yet fully working.
18. I checked the /upload/tmp folder... There is a really strange thing there: The two uploaded file are still there but they're empty (0 bytes)...

This is taking a good form, almost fully functional as you said...
Hopefully my feedback will help you! I think this post may be used as a test case...
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 3 months ago #44169

  • texens
  • texens's Avatar
  • Offline
  • Junior Lime
  • Posts: 24
  • Thank you received: 1
  • Karma: 0
Hi CarbonaCat,

Thanks a lot for testing it out :)

CarbonaCat wrote:
Live testing :) The underlined parts are what I was not expecting or suggestions.

1. I have updated my work copy of the /dev repository.
2. I created a new question of type "Upload file". I selected the following advanced parameters: one file of type "*.png" have to be submitted.
3. Tried to test the question alone... However, it ends up with a 404 not found error on the following URL: "/limesurvey_dev/admin/uploader.php". Could you check it why?
You're getting a 404 error on the aforementioned URL because the uploader sits in the limesurvey_dev directory and not the limesurvey_dev/admin/ directory. It won't work on its own, as it is dependent on the survey question for all its input. Its like a template for the upload question and not a complete entitiy on its own.
4. Testing the whole survey seems to work. I'm under Google Chrome 5.0.
5. I'm uploading a file (the active icon, active.png) with a title and a comment, I click on "Upload" (Not touching to the other texts fields), there is a progress bar...
6. Then it says it worked! However, although I did upload the requested file, it didn't automatically switch to the gallery view so I had to click.
The automatically switch to gallery has not been implemented yet. But you can expect it in the coming iteration :)
7. YAY I CAN SEE IT
8. Trying to update title & comment... I click on Save change, but there is no confirmation if it was really saved.
Thanks for pointing it out. We will have a flash notice for all the actions in the next iteration.
9. I click on delete... it disappeared, nice!
10. Going back to the "From computer" tab.
11. Browsing file in order to upload "junk.txt"... I think it would be nice if it rejected the file as soon as I chosen a wrong one (Remember that I have chosen *.png filter).
12. I haven't filled the title and comment fields.
13. Uploading... It says it worked... filter not working yet?
Yes, the filter has not been implemented yet, I don't want to use just the file's extension for testing a file's type. There are a couple handy PHP scripts that I'm looking into for this function. But, this is something that might be delayed for a while. Also, the number of minimum/maximum files in not yet enforced.
14. Checking in the Gallery... There is a broken image frame as it tries to display the junk.txt as an image.
15. In this case, there is no any information about what the file is. I think it would be nice to put somewhere the filename...
We'll get placeholders for the various filetypes and depending on the filetype, we'll display these placeholder images in their preview. Also, it would be a good idea to display the filename, thanks for the idea :)
16. Closing the window and submitting the page.
We'll also get a "Save and Exit" button, and on pressing this button the modal dialog will close. So, the users won't have to explicitly click on the "cross" on the dialog's top right.
17. The *.png filter didn't work so I guess the validation is not yet fully working.
18. I checked the /upload/tmp folder... There is a really strange thing there: The two uploaded file are still there but they're empty (0 bytes)...
This doesn't sound good. Could you please give me some more information on this bug? Which OS are you using?
This is taking a good form, almost fully functional as you said...
Hopefully my feedback will help you! I think this post may be used as a test case...
Definitely CarbonaCat, the feedback is very useful and helps me identify possible issues and fix them. Thanks a lot for your time :)
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 3 months ago #44171

  • CarbonaCat
  • CarbonaCat's Avatar
  • Offline
  • Fresh Lemon
  • Posts: 3
  • Karma: 0
Oh, right, here is my config:
-Windows XP SP2,
-PHP5.3, running on WampServer 2.0
-FYI: My HDD partition is FAT32 (Well, I don't have any control over it, it's a corporate policy). Just in case it may have an impact.

It's likely as the files were overwritten with empty content instead of simply deleted.

For testing the question alone, I think you'll have to do something sooner or later. Well, it's not critical, but it'll help the surveyor to configure its upload question.
In fact, I think the testing itself should lead you to a dummy uploader that allow you to test your filters and parameters (number of files, ...).

It may become complicated to use your filter if you want to test more than the extension. May you should think of file type groups, like images, texts, xml,... instead of simply a filter type...
The administrator has disabled public write access.

GSOC 2010: File Upload Question Type 6 years 3 months ago #44173

  • texens
  • texens's Avatar
  • Offline
  • Junior Lime
  • Posts: 24
  • Thank you received: 1
  • Karma: 0
I haven't tested it on a windows machine, so it might be a windows specific thingy. I'll take a look into this issue.
For testing the question alone, I think you'll have to do something sooner or later. Well, it's not critical, but it'll help the surveyor to configure its upload question.
In fact, I think the testing itself should lead you to a dummy uploader that allow you to test your filters and parameters (number of files, ...).
I misunderstood the problem in the previous post. Definitely, the test uploader should work and is a valid issue. Glad that you caught it. You see, this is why I love feedbacks :)
It may become complicated to use your filter if you want to test more than the extension. May you should think of file type groups, like images, texts, xml,... instead of simply a filter type...
This file type filter thingy sounds like a good idea, so I'll implement this one.

once again, thanks for your awesome feedbacks :)
The administrator has disabled public write access.
Time to create page: 0.528 seconds
Imprint                   Privacy policy         General Terms & Conditions         Revocation information and revocation form