Description

Our users are using their Moinmoin-wiki also to share files. When they want to download big files (200 - 400 MByte) from the wiki, usually the download doesn't even start or at least runs very slowly (~ 10 kb/s ...) in a 100 MBit LAN. Memory consumption of the Moinmoin-Instance which serves the file grows very quickly to the size of the file, which is being served. Downloading the same file over the same network to the same computer via FTP works perfectly.

I can remember that this worked better some time ago, but as I had trouble with Apache and reconfigured it every few days, I can't remember with which method it worked best (standalone server, mod_wsgi, mod_python or FastCGI ... I tried them all). Note that now I'm not using Apache, but lighttpd instead.

Steps to reproduce

1. Start downloading a big (~ 420 MByte) attachment from a Wiki page
2. After some seconds the download starts, running very slowly (10 kb/sec) most of the time
3. Notice a very big increase in memory consumption on the server:

   • Before:

     USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
     www-data  9811  0.0  0.0   4664  1672 ?        SN   07:37   0:00 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
     [...]
     www-data  9855  0.0  0.1   8904  6436 ?        SN   07:37   0:00  \_ python /var/www/moin/moin.fcgi
     www-data  9856  0.0  0.1   8900  6432 ?        SN   07:37   0:00  \_ python /var/www/moin/moin.fcgi
     www-data  9857  0.0  0.2  19312  9016 ?        SN   07:37   0:01  \_ python /var/www/moin/moin.fcgi

   • While downloading:

     USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
     www-data  9811  0.0  0.0   4664  1692 ?        SN   07:37   0:00 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
     [...]
     www-data  9855  0.0  0.1   8904  6436 ?        SN   07:37   0:00  \_ python /var/www/moin/moin.fcgi
     www-data  9856  0.0  0.1   8900  6432 ?        SN   07:37   0:00  \_ python /var/www/moin/moin.fcgi
     www-data  9857  0.0 27.4 1494732 1131516 ?     SNl  07:37   0:16  \_ python /var/www/moin/moin.fcgi

4. Notice that after aborting the request, Moinmoin doesn't free all the memory:

   USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
   www-data  9811  0.0  0.0   4664  1692 ?        SN   07:37   0:00 /usr/sbin/lighttpd -f /etc/lighttpd/lighttpd.conf
   [...]
   www-data  9855  0.0  0.1   8904  6436 ?        SN   07:37   0:00  \_ python /var/www/moin/moin.fcgi
   www-data  9856  0.0  0.1   8900  6432 ?        SN   07:37   0:00  \_ python /var/www/moin/moin.fcgi
   www-data  9857  0.1 11.4 584556 469992 ?       SN   07:37   0:27  \_ python /var/www/moin/moin.fcgi

Example

Unfortunately this is an internal wiki, but if it's really helpful I could create another Wiki instance on the server and upload a big attachment...

Component selection

• general (attachment handling)

Details

Workaround

• Plan A: Provide attachment downloads using FTP

• Plan B: Use direct attachment serving (which seems to get removed sometime in the future, see RemovingAttachmentsDirectServing)

Discussion

• On my hosted Wiki (MoinMoin 1.6.3 with Apache/2.0.52 (Red Hat) mod_wsgi/2.0 Python/2.5 configured) I can confirm that a 100MB tar.gz file pushed my memory usage from ~80MB to 220MB shortly after the download started (maybe you like to see the printscreen memory_usage_download.png. Also uploading a 100MB Files generate a plus of 100MB memory usage, but first after the upload was completed. The Download are working with normal speed, so no cancel or failure... well mod's like fastcgi, python and fastcgi likes to cache data - so this is maybe a feature/behavior and not really a MoinMoin bug -- MarcelHäfner 2008-05-04 12:38:05

  We don't discuss upload in this bug report (see title and consider it is a POST being used for upload and some servers might keep all POST data in memory). If you like to analyze upload, please in a different bug report and split discussion there for the different web servers / server methods. -- ThomasWaldmann 2008-05-04 13:00:54

Plan

• Priority:
• Assigned to:
• Status: confirmed that big memory consumption happens, now searching for root cause

CategoryMoinMoinBugConfirmed