Short description
It would be fine to have the possibility to upload multiple files at once. At the moment only one file could be selected for uploading. Probably this could be changed. File renaming for multiple files does not matter in this case. This would be very useful for the Gallery2 parser. -- ReimarBauer 2005-03-26 09:05:58
This seems to be difficult. But what is by uploading a tar file which could be extracted on demand by selecting a radio button on the normal attachment form. -- ReimarBauer 2005-03-27 23:14:43
Its not difficult, simply provide multiple file input elements, there are sites that use up to 10 files uploads.
- Yes, but ten times selcting ... On the other hand I had read opera is able in it's file dialog to select multiple files by the CTRL key. But this seems to be no standard. And there is no switch.
Windows or mac user does not know how to create tarballs, but zip files can be created in Mac OS X with no additional software (same in XP?), available on Linux too (GPL?). Maybe support both.
- zip is available on linux too. It licensed under a BSD-like license.
This is a necessary feature when you're working on long or complicated documents via the wiki, that require lots of screenshots or other attachments, e.g. documentation. This was the first thing we added into our TWiki installation. File uploads via web browsers is so finicky that one larger upload was safer than multiple small ones on the same form, and when you're attaching fifty or more screenshots, it's definitely necessary. However, even this is suboptimal, and we wished for alternatives, like uploading of attachments via WebDAV, or using a client-side utility.
/RulesForUnzip and patch and tests and example
And see PackageInstaller for an enhanced patch (fixed XSS issues, added view (i.e. list) ZIP file capability and finally added package handling)
I think it's easy to do it in this way that's in AttachFile in _build_filelist a new label is added label_unzip = _("unzip") and it must be added to paramdict. Then the files are checked by zipfile.is_zipfile(os.path.join(attach_dir,file)). The real zip files get an unzip label.
viewlink='<a href="%(baseurl)s/%(urlpagename)s?action=%(action)s&do=unzip&target= %(urlfile)s">%(label_unzip)s</a>' % parmdict
If one clicks on this then unzip is used on this file to extract it in the attach_dir.def execute could get these changes
elif request.form['do'][0] == 'unzip': if request.user.may.delete(pagename) and request.user.may.read(pagename): attachment_path = getAttachDir(request,pagename) target=request.form['target'][0] file=attachment_path+'/'+target if zipfile.is_zipfile(file): zf = zipfile.ZipFile(file) for name in zf.namelist(): outfile = open(os.path.join(attachment_path, name), 'wb') outfile.write(zf.read(name)) outfile.flush() outfile.close() msg = _('file '+target+' unziped.') else: msg = _('file '+target+' is no zip file!') else: msg = _('You are not allowed to unzip attachments of this page.')
We need to import zipfile.
-- ReimarBauer 2005-03-28 18:11:45
Exploding a zipfile can be dangerous. So I would prefer another solution. -- ThomasWaldmann 2005-03-28 22:10:49
I am too. What do you think on uploading of an iso file and a loop mount. This will not duplicate the data on the server. -- ReimarBauer 2005-03-29 07:45:55
May be we could use the apvfs. If it is not necessary to unpack a file a whole bunch of problems won't appear.Using such file systems just delegate this problem. You e.g. will not have a problem while unpacking a file but while listing the file if it contains too many directories entries. Accepting tgz and zip files is just right IMHO and very usable at other points as well (upload packages of pages, upload/install plugins etc.). -- AlexanderSchremmer 2005-04-03 18:56:56
Yes, we had to insert of a bunch of security checks in our TWiki version. We checked the size of each file to make sure people couldn't bomb the wiki with a 500K zip that unzips to a 500MB file. Since attachments can't have subdirectories, we also ignore those completely so you can't unzip to any random path that the web site as write access to, we ignore extra unix permissions, things like that.
Please could you attach your code? -- ReimarBauer 2005-03-29 07:45:55
Wouldn't do any good; TWiki is a Perl wiki. The security issues we dealt with specifically were:
- subdirectories (we dropped them since you can't attach directories to pages)
- also, if you have a file named test.txt in your .zip archive, plus a file named test/test.txt, in our code, whichever one came later in the archive would overwrite the earlier one. It would be better to not attach either one and present a message to the user, perhaps not attach any files at all, or perhaps rename the directoried one
If we add this line into the for loop all subdirectories and their files got dropped.
if ((string.find(name,'/') == -1) and (string.find(name,'\\') == -1)):
- also, if you have a file named test.txt in your .zip archive, plus a file named test/test.txt, in our code, whichever one came later in the archive would overwrite the earlier one. It would be better to not attach either one and present a message to the user, perhaps not attach any files at all, or perhaps rename the directoried one
- the size of the uncompressed files in the .zip (allowed the wiki admin to set an upper limit on the total size of uncompressed contents, and also applied the limits on the size of individual attachments to each file in the .zip)
The uncompressed individual file size could be checked by adding
zi = zf.getinfo(name) if (zi.file_size < (10.0*1024*1024)):
Here the restriction is set to 10MB but this should be configurable within the wikiconfig.py. I don't see a difference between attachment file size general and "the total size of uncompressed contents". Perhaps attachment upload size should be limited.
This is not enough if the info is just read from the zip TOC because there are .zip bombs with faked TOCs.
- removing spaces from and normalizing the case in filenames (Windows and Macintosh file systems are case insensitive, and filenames with spaces can cause issues in some cases, so we allowed the wiki admin to specify (for all attachments) whether spaces would be stripped and filenames would be normalized to all lower or upper case.
The same rules as for uploading an unpacked file should be used.
open(os.path.join(attachment_path, name).encode(config.charset), 'wb')
- ignoring stored permissions on the files
This seems to be already included to me -- ReimarBauer 2005-04-04 22:10:00
The code gots changes again. I have tried two kinds of zip bombs.
- a zip file including zip files
prevention is done by dropping zip files - no other wiki syntax could be used recursive so I think this is a rule which does not make problems for users. The file is dropped after unpacking
- a zip file including a very large file or many files
prevention is done in a defined limitation of size for the result file and a limitation of size for the sum of all attached files. This two parameters for attachments of a page should be defined best in wikiconfig. The size checking is done before unpacking.
Unpacking a zip file could not overwrite existing files. Unpacking should have the same rules as attaching a file. -- ReimarBauer 2005-04-05 17:56:56
- a zip file including zip files
elif request.form['do'][0] == 'unzip': if request.user.may.delete(pagename) and request.user.may.read(pagename): attachment_path = getAttachDir(request, pagename) files = _get_files(request, pagename) fsize =0.0 if files: for file in files: fsize += float(os.stat(os.path.join(attachment_path,file).encode(config.charset))[6]) # in byte single_file_size=10.0*1024*1024 attachments_file_space=100.0*1024*1024 avalaible_attachments_file_space=attachments_file_space-fsize target=request.form['target'][0] file=attachment_path+'/'+target if zipfile.is_zipfile(file): zf = zipfile.ZipFile(file) sum_size_over_all_valid_files=0.0 for name in zf.namelist(): if ((string.find(name,'/') == -1) and (string.find(name,'\\') == -1)): zi = zf.getinfo(name) sum_size_over_all_valid_files+=zi.file_size if (sum_size_over_all_valid_files < avalaible_attachments_file_space): valid_name=[] for name in zf.namelist(): if ((string.find(name,'/') == -1) and (string.find(name,'\\') == -1)): zi = zf.getinfo(name) if (zi.file_size < (single_file_size)): new_file=os.path.join(attachment_path, name).encode(config.charset) if not os.path.exists(new_file): outfile = open(new_file, 'wb') outfile.write(zf.read(name)) outfile.flush() outfile.close() # it's not allowed to zip a zip file so this is dropped if zipfile.is_zipfile(new_file): os.unlink(new_file) else: valid_name.append(name) os.chmod(new_file, 0666 & config.umask) _addLogEntry(request, 'ATTNEW', pagename, new_file) if (len(valid_name) > 0): upload_form(pagename, request, msg=_("Attachment '%(filename)s' unziped.") % {'filename': target}) else: upload_form(pagename, request, msg=_("Attachment '%(filename)s' not unziped because result to big or included zip file or file exists.") % {'filename': target}) else: upload_form(pagename, request, msg=_("Attachment '%(filename)s' could not be unziped because result to big (%(space)d kB missing).") % {'filename': target, 'space':(sum_size_over_all_valid_files-avalaible_attachments_file_space)/1024}) else: msg = _('file '+target+' is no zip file!') else: msg = _('You are not allowed to unzip attachments of this page.')
When it is ready, attach a unified diff in order to make it easier for others to merge it.
Please add also tests for all important cases that show that this code really works.
- I think he cannot use our unit test framework because we do not have request assemblers, right? But specially prepared .zip files should be enough IMHO.
see /RulesForUnzip and patch and tests and example
We should note that the famous 42.zip will not affect this code because it is based on stacked .zip files.
The zip mechanism is nice but not very end-user friendly. Plus, sites on the web have set the bar higher by being able to upload multiple files relatively easily. It would be very very nice to have some way to do this directly with MoinMoin. Any clever ideas on this? How about an optional Java helper applet like some sites use? It would be nice to select multiple files or say all files in a directory.
A better place as this would be a new FeatureRequests. Please move it to a separate page -- ReimarBauer 2007-04-26 21:40:32