Description

Our users are using their Moinmoin-wiki also to share files. I just uploaded the new Service Pack for WinXP (317 MB) as an attachment. The memory consumption of the Moinmoin-Standalone-Instance which serves the file increases in one second by the size of the file, which is being uploaded. When I upload a small (2 KB) attachment afterwards, the memory consumption does not decrease. However, when I delete any attachment, the memory consumption shrinks back to 20 MB.

Steps to reproduce

1. Start uploading a big (~ 317 MByte) attachment.
2. Notice a very big increase in memory consumption on the server:

   • Before:

     USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
     www-data 24886  0.0  0.1  91420  5804 ?        Sl   10:22   0:00 python ./moin start
     www-data 24938  0.0  0.0   4264   988 ?        S    10:22   0:00 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf

   • After uploading is finished:

     USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
     www-data 24886 21.3  8.2 423000 338088 ?       Sl   10:22   0:15 python ./moin start
     www-data 24938  4.2  0.0   5140  2076 ?        S    10:22   0:02 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf

   • After deleting any attachment:

     USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
     www-data 24886  4.0  0.3  99300 14396 ?        Sl   10:22   0:17 python ./moin start
     www-data 24938  0.6  0.0   5140  2140 ?        S    10:22   0:02 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf

Example

Unfortunately this is an internal wiki, so I can't give out a public URL :-/

Component selection

• general (attachment handling)

Details

Workaround

Nothing known yet.

Discussion

I don't see how deleting an attachment is related to the drop of memory usage.

Can you try the upload again and then report about memory usage when doing:

• a normal page view as next request
  • Nothing changes (memory usage increases by just a few KB)
• a page edit/save as next request
  • Nothing changes (memory usage increases by just a few KB)

It's completely reproducible: I browse the wiki just normally, upload other attachments, rename an attachment, download another one, edit and save a normal page - the memory usage will not shrink. This only happens when I delete any attachment (it doesn't have to be the big one uploaded earlier!)

Moin was processing file uploads in a bad way (reading whole file content into memory). Code was changed to use the open temporary file that cgi.FieldStorage (util.Field for mod_python) provides.

Related code (in general, not only for standalone server):

• request.setup_args

• request._setup_args_from_cgi_form

• cgi.FieldStorage (or mod_python.util.Field)

See: http://docs.python.org/lib/node559.html

If a field represents an uploaded file, accessing the value via the value attribute or the getvalue() method reads the entire file in memory as a string. This may not be what you want. You can test for an uploaded file by testing either the filename attribute or the file attribute. You can then read the data at leisure from the file attribute: ...

Using the file attribute does not work as it is also used with cStringIO for non-file-upload stuff. Thus we check for the filename.

Feedback

I can confirm that on my latest 1.6. (changeset 2646 801e1c808bc3) wiki the RES (memory usage) stays around the same level. Tested with a 20MB File on Apache mod_wsgi and standalone (behind an apache proxy). Note: On mod_wsgi I got a incrase from 11mb to 14mb, on standalone I got an increase from 9.7mb to 10mb. That's maybe also due normal page accessing, well the increase is definitely not 1:1 the filesize any more! Maybe somebody should test it with a 600MB or bigger file. bye and thanks for the fast fix. -- MarcelHäfner 2008-05-08 16:47:19

Plan

• Priority:
• Assigned to:
• Status:

  • 1.6 fix: http://hg.moinmo.in/moin/1.6/rev/fca832a19d32

  • 1.7 fix: http://hg.moinmo.in/moin/1.7/rev/6fe1ea4d9d1a

    • standalone, wsgi works (and likely also the other request methods as they share the same code)

    • Twisted is untested

    • mod_python is untested

CategoryMoinMoinBugFixed