MoinMoin: MoinMoinChat/Logs/moin-dev/2008-05-04

2008-05-04T01:16:48  * ThomasWaldmann upgraded to mod_wsgi 2.0
2008-05-04T12:23:19  <ThomasWaldmann> moin
2008-05-04T13:28:50  * ThomasWaldmann improved http://en.wikipedia.org/wiki/Mod_wsgi
2008-05-04T13:29:35  <johill> can it run processes as another user?
2008-05-04T13:31:00  <ThomasWaldmann> sure
2008-05-04T13:31:11  <ThomasWaldmann> (if apache was started as root)
2008-05-04T13:34:31  <johill> guess that's not useful since I host all wikis as a farm
2008-05-04T13:35:47  <ThomasWaldmann> its most useful for python hosting
2008-05-04T13:36:12  <ThomasWaldmann> (if you have different users running untrusted code)
2008-05-04T13:36:38  <johill> right
2008-05-04T13:37:00  <johill> I just have a bunch of users who'd like to access their plugins dir and their data dir directly, but if it's all running as www-data I can't really do that
2008-05-04T13:37:05  <johill> not that it matters much, right now
2008-05-04T14:01:01  * ThomasWaldmann improved http://en.wikipedia.org/wiki/Mod_python (SCNR)
2008-05-04T14:31:04  <johill> the mod_wsgi article is weird with full sentences in headings
2008-05-04T14:31:42  <johill> but I refrain from editing wikipedia ever since they reverted me a number of times because I didn't cite sources for a book release (which you can f'ing well just look up on any number of book websites)
2008-05-04T14:32:30  <xorAxAx> wikis are evil
2008-05-04T14:35:46  <johill> at least so far my spelling fixes weren't reverted due to not citing a dictionary entry
2008-05-04T14:36:49  <johill> I mean, where do you draw the line?
2008-05-04T15:01:44  <gizmach> moin
2008-05-04T15:07:57  <ThomasWaldmann> hi gizmach
2008-05-04T15:08:12  <gizmach> hi ThomasWaldmann
2008-05-04T15:08:33  * ThomasWaldmann looks at the attachment download memory consumption
2008-05-04T15:12:40  <zenhase> moin
2008-05-04T15:12:49  <gizmach> zenhase: :) hi
2008-05-04T15:13:16  * zenhase is sitting in the sun, reading request-code and taking notes
2008-05-04T15:13:50  <zenhase> i love analogue notetaking but when it comes to making them available online you always have this information transformation barrier :/
2008-05-04T15:14:23  <zenhase> i think my next mobile computer has to be something tablet-like
2008-05-04T15:19:10  <zenhase> which ocr can read this handwriting? http://eigensex.org/requestbase-init.jpg :P
2008-05-04T15:26:24  * ThomasWaldmann will do an OLPC in the sun test soon :)
2008-05-04T15:26:40  <zenhase> ah, hi-contrast mode :)
2008-05-04T15:27:52  <TheSheep> zenhase: there are specialized handwriting recongition programs
2008-05-04T15:28:06  <TheSheep> zenhase: they are better than ocr, because they also take advantage of timing data
2008-05-04T15:28:18  <TheSheep> zenhase: unfortunatley there is none for Linux :(
2008-05-04T15:28:24  <zenhase> TheSheep: i know
2008-05-04T15:28:37  <zenhase> some friends have bought lenovo x61 tablets
2008-05-04T15:28:38  <TheSheep> zenhase: the closest you get is xstroke
2008-05-04T15:29:15  <TheSheep> it's amazing that there is such a gap
2008-05-04T15:29:20  <zenhase> and in linux they have just a nice pen interface for clicking with their cursor ... handwriting and cool stuff -> no chance
2008-05-04T15:29:27  <zenhase> it's not amazing imo
2008-05-04T15:30:03  <TheSheep> it's not that hard, there are much more complicated issues handled easily y linux programs
2008-05-04T15:30:13  <TheSheep> by
2008-05-04T15:30:39  <zenhase> it isn't hard from a theoretical point of view
2008-05-04T15:31:14  <zenhase> but i found *nixers to be generally uneasy towards 'usability' in terms of graphical uis and interaction design
2008-05-04T15:31:43  <TheSheep> nonsesne, compare windows and gnome
2008-05-04T15:31:45  <zenhase> it's strange that most *nixers i know totally grok shells and stuff ... most of them hate UIs
2008-05-04T15:31:54  <xorAxAx> oh well
2008-05-04T15:32:02  <xorAxAx> its patents/companies
2008-05-04T15:32:13  <TheSheep> xorAxAx: no software patents in Europe... yet
2008-05-04T15:32:14  <xorAxAx> zenhase: you can start today implementing the companies' papers if there are any
2008-05-04T15:32:43  <zenhase> yeah
2008-05-04T15:37:16  * ThomasWaldmann -> works :)
2008-05-04T15:39:19  <ThomasWaldmann> almost no visible difference with and without backlight
2008-05-04T15:39:57  * TheSheep curses glade
2008-05-04T15:40:47  <TheSheep> have you seen 'from __future__ import braces' already?
2008-05-04T15:40:56  <xorAxAx> yes
2008-05-04T15:41:06  <zenhase> :)
2008-05-04T15:43:25  * johill curses the linux networking code
2008-05-04T16:13:55  <ThomasWaldmann> hmm, could it be that shutil.copyfileobj creates large amounts of string trash
2008-05-04T16:14:09  <johill> yes
2008-05-04T16:14:20  <ThomasWaldmann> and thus makes it completely pointless using a small buffer
2008-05-04T16:14:53  <johill> man 2 splice :P
2008-05-04T16:15:05  <ThomasWaldmann> because it will create new trash faster than the garbage collection can get rid of it
2008-05-04T16:15:10  <johill> heh fun
2008-05-04T16:18:23  <ThomasWaldmann> that would make moin depend on a recent linux kernel
2008-05-04T16:19:29  <ThomasWaldmann> any other suggestions?
2008-05-04T17:03:20  <ThomasWaldmann> hmm, can't reproduce the problem with a minimal copyfileobj test program
2008-05-04T17:08:39  <ThomasWaldmann> ok, for the current moin WSGI it is clear: the whole file gets pumped into a StringIO object and then given back from the moin app to the wsgi server
2008-05-04T17:08:55  <ThomasWaldmann> zenhase: ^^ don't do it that way :P
2008-05-04T17:09:22  <TheSheep> I think there was a special case in wsgi for serving files directly
2008-05-04T17:09:48  <ThomasWaldmann> yeah
2008-05-04T17:10:51  <TheSheep> wsgi.file_wrapper
2008-05-04T17:17:35  <ThomasWaldmann> zenhase: http://moinmo.in/MoinMoinBugs/AttachmentDownloadsSlowAndConsumeTooMuchMemory
2008-05-04T17:23:01  <zenhase> hrhr
2008-05-04T17:23:17  <zenhase> i'll keep that in mind
2008-05-04T17:23:39  <ThomasWaldmann> zenhase: X) I guess people would be very happy if someone carefully fixed that for 1.7 :)
2008-05-04T17:23:57  <zenhase> ok, i'll have a look
2008-05-04T17:26:08  <zenhase> seems to be not WSGI-specific
2008-05-04T17:26:26  <ThomasWaldmann> I am looking at the other servers right now
2008-05-04T17:33:11  <ThomasWaldmann>         self.out = StringIO()
2008-05-04T17:33:19  <ThomasWaldmann> (from thfcgi.py)
2008-05-04T17:33:24  <zenhase> :)
2008-05-04T17:34:00  <zenhase> well, it sends fcgi records to the server
2008-05-04T17:34:45  <zenhase> so perhaps it has to accumlate some writes
2008-05-04T17:37:36  <ThomasWaldmann> could be fixed by using flush I guess
2008-05-04T18:10:47  <ThomasWaldmann> zenhase: how about wrapping the wsgi.file_wrapper sample code to moin's wsgi request object?
2008-05-04T18:14:42  <ThomasWaldmann> s/wrapping/adding/ :)
2008-05-04T18:22:41  <ThomasWaldmann> zenhase: still there?
2008-05-04T18:33:35  * ThomasWaldmann adds send_file(fileobj) to Request
2008-05-04T18:47:09  <CIA-48> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3553:1052c105b16f 1.7/MoinMoin/ (action/AttachFile.py request/__init__.py): new request.send_file() call, making it possible to use server-specific optimizations
2008-05-04T18:48:37  <ThomasWaldmann> is someone running 1.7 with fastcgi?
2008-05-04T18:49:13  * johill is
2008-05-04T18:49:33  <johill> but why are you fixing this if doing wsgi will fix it anyway? :)
2008-05-04T18:50:04  <ThomasWaldmann> well, maybe some users don't want to have crawling servers until 1.8 :P
2008-05-04T18:50:12  <johill> :)
2008-05-04T18:50:27  <ThomasWaldmann> and wsgi has same problem currently
2008-05-04T18:51:24  <ThomasWaldmann> johill: can you please hg pull -u for 1.7 and reproduce that the problem happens for you
2008-05-04T18:51:44  <ThomasWaldmann> (just download a file of 500MB and watch moin eat your memory)
2008-05-04T18:56:18  <ThomasWaldmann> can you reproduce?
2008-05-04T18:59:17  <ThomasWaldmann> after reproducing, try that: http://paste.pocoo.org/show/47153/
2008-05-04T19:22:07  <ThomasWaldmann> johill: still there?
2008-05-04T19:33:36  * gizmach can't wait python 2.6 cause she has to implement base 2 stuff , float conversation :(
2008-05-04T19:34:13  <ThomasWaldmann> hopefully the conversation is interesting :)
2008-05-04T19:35:44  <gizmach> ThomasWaldmann: heh... its for genetic algorithm assignment
2008-05-04T19:51:20  <ThomasWaldmann> zenhase: i'll test an easy hack for the wsgi stuff soon
2008-05-04T20:58:19  <CIA-48> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3554:c0becca3d0d9 1.7/MoinMoin/ (request/request_wsgi.py server/server_wsgi.py): request_wsgi.send_file: use wsgi.file_wrapper (or a builtin simple wrapper). Fixes memory consumption for sending of large file attachments.
2008-05-04T20:58:40  <johill> ThomasWaldmann: sorry
2008-05-04T20:58:44  <johill> was digging into net code
2008-05-04T20:58:56  <johill> I'm not sure I want to reproduce that problem on my server that only has 256M memory ;)
2008-05-04T20:59:48  <johill> ThomasWaldmann: but that patch you pasted seems to do exactly nothing since fcgi request inherits from RequestBase
2008-05-04T21:00:17  <ThomasWaldmann> it flushes
2008-05-04T21:00:32  <johill> oh it changes the default
2008-05-04T21:00:42  <johill> ?
2008-05-04T21:00:50  <ThomasWaldmann> yes
2008-05-04T21:22:03  <ThomasWaldmann> the mod_wsgi wsgi.file_wrapper is really fast :) sucks much less now.
2008-05-04T22:04:13  <CIA-48> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 2639:ac6e3ce989ad 1.6/MoinMoin/ (action/AttachFile.py request/__init__.py): new request.send_file() call, making it possible to use server-specific optimizations (backport from 1.7)
2008-05-04T22:04:13  <CIA-48> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 2640:17ff68ef3be7 1.6/MoinMoin/ (request/request_wsgi.py server/server_wsgi.py): request_wsgi.send_file: use wsgi.file_wrapper (or a builtin simple wrapper). Fixes memory consumption for sending of large file attachments.
2008-05-04T22:10:52  <zenhase> ThomasWaldmann: sorry, my battery ran low
2008-05-04T22:13:48  <ThomasWaldmann> use duracell ;P
2008-05-04T22:14:02  <johill> keeps going and going and going
2008-05-04T22:14:09  <johill> what happens if you put the battery in backwards? :)
2008-05-04T22:15:07  <xorAxAx> as we all know from tv ads, rabbits run on duracells
2008-05-04T22:15:27  <johill> oh yeah I forgot to mention it's a bunny
2008-05-04T22:15:34  * johill wonders when he'll be kicked
2008-05-04T22:16:51  <TheSheep> ThomasWaldmann: it's only faster on apache
2008-05-04T22:17:15  <TheSheep> ThomasWaldmann: most wsgi servers don't provide that method, which is sad
2008-05-04T22:17:32  <johill> but it puts the bug into them instead ;)
2008-05-04T22:17:44  <ThomasWaldmann> TheSheep: it doesn't consume a lot memory any more, so it is faster everywhere
2008-05-04T22:17:58  <TheSheep> ah
2008-05-04T22:18:13  <zenhase> xorAxAx: only the pink ones
2008-05-04T22:18:15  <ThomasWaldmann> TheSheep: mod_wsgi 1.3 does not provide wsgi.file_wrapper, btw. 2.0 does iirc.
2008-05-04T22:18:19  <zenhase> xorAxAx: i am not one of those
2008-05-04T22:18:38  * TheSheep colors zenhase pink
2008-05-04T22:18:47  <zenhase> eek
2008-05-04T22:18:51  <xorAxAx> zenhase: you are purple instead? :)
2008-05-04T22:19:06  <TheSheep> salmon :)
2008-05-04T22:19:27  <zenhase> xorAxAx: http://eigensex.org/hasi.jpg
2008-05-04T22:20:28  <TheSheep> aww, cute! *splat*
2008-05-04T22:21:25  <xorAxAx> zenhase: hehe
2008-05-04T22:21:53  <zenhase> TheSheep: splat?
2008-05-04T22:22:05  * TheSheep cleans the sole of his shoe
2008-05-04T22:22:15  <TheSheep> sorry
2008-05-04T22:22:22  <zenhase> :'(
2008-05-04T22:26:43  * ThomasWaldmann lols at "Flushed Away"
2008-05-04T22:31:47  <zenhase> is this a socket-joke?
2008-05-04T22:32:17  <ThomasWaldmann> no, an animation film :)
2008-05-04T22:32:25  <zenhase> oh
2008-05-04T22:37:30  <zenhase> TheSheep: we talked about wsgi middlewares some while ago and what to put into this layer and what not to put there
2008-05-04T22:38:45  <zenhase> TheSheep: you condemned the approaches which require a certain middleware to be present for the application to work, right?
2008-05-04T22:39:08  <zenhase> like session-middleware and then depending on some session-object in the environ
2008-05-04T22:40:48  <ThomasWaldmann> zenhase: these are thoughts for after your project is completed, right?
2008-05-04T22:43:12  <ThomasWaldmann> dennda: how far are your preparations?
2008-05-04T22:43:13  <TheSheep> zenhase: yes, why?
2008-05-04T22:44:44  <zenhase> ThomasWaldmann: since i want to move some functionality out of the current request object (like for example .run, which should be an WSGI-app) i am thinking about, what stuff belongs to the request and what makes a middleware/application
2008-05-04T22:47:39  <ThomasWaldmann> do small steps :)
2008-05-04T22:56:24  <zenhase> well, i will leave this open for the time being
2008-05-04T22:56:35  <ThomasWaldmann> zenhase: btw, after having a short look at wsgi server/request stuff in moin some hours ago, I am wondering if the sending of data is good the way it is.
2008-05-04T22:56:38  <zenhase> just saved the current rev. of my project page
2008-05-04T22:56:54  <ThomasWaldmann> (i am not thinking about fundamental changes there right now)
2008-05-04T22:56:59  <zenhase> hmm
2008-05-04T22:57:10  <zenhase> could you explain what you mean
2008-05-04T22:57:53  <ThomasWaldmann> but it currently does request.write -> writes to a StringIO -> later gets the content of the StringIO "file" and returns it wrapped in a list to make it an iterator
2008-05-04T22:58:13  <zenhase> uh
2008-05-04T22:58:52  <ThomasWaldmann> it could just use captureinglist.append(fragment) to do the request.write instead of that
2008-05-04T22:59:05  <ThomasWaldmann> (and later return that list)
2008-05-04T22:59:19  <zenhase> uhm
2008-05-04T22:59:27  <ThomasWaldmann> not sure if that makes a real difference
2008-05-04T22:59:31  <zenhase> isn't that how stringio is implemented anyway?
2008-05-04T22:59:36  <zenhase> i mean the python version
2008-05-04T22:59:47  <zenhase> cStringIO might be different
2008-05-04T23:00:07  <zenhase> but stringio is nothing more than a list in this regard :o
2008-05-04T23:00:08  <ThomasWaldmann> i don't know how StringIO is implemented internally
2008-05-04T23:00:37  <ThomasWaldmann> i guessed it uses some buffer rather than a list internally
2008-05-04T23:02:00  <zenhase> it's a list of strings and some instance vars which track position
2008-05-04T23:03:43  <xorAxAx> yes, but with thomas' idea, you save the string joining
2008-05-04T23:03:49  <zenhase> on seeking around you have some overhead because he consolidates the list into a single string
2008-05-04T23:03:56  <TheSheep> of course it would be best to have most parts of moin that output something to be iterator generators
2008-05-04T23:04:00  <ThomasWaldmann> yeah, read the src, you're right. and xorAxAx is, too. :)
2008-05-04T23:04:42  <ThomasWaldmann> TheSheep: yeah, but we don't want to change all request.write during soc.
2008-05-04T23:04:44  <zenhase> xorAxAx: ah yes
2008-05-04T23:06:12  <zenhase> this joining is what makes for the memory hog on big data chunks i guess :o
2008-05-04T23:06:50  <ThomasWaldmann> so it would have a slight benefit (if not the wsgi server code then does the join :)
2008-05-04T23:07:27  <zenhase> hmm
2008-05-04T23:07:29  <ThomasWaldmann> (or some middleware)
2008-05-04T23:07:40  <zenhase> yes
2008-05-04T23:08:29  <zenhase> TheSheep: yes, that would be terrific
2008-05-04T23:09:07  <zenhase> TheSheep: but try changing all those request.write calls ... it's over 300 iirc and that is not counting 3rd party plugins
2008-05-04T23:09:22  <zenhase> and it would be a major change
2008-05-04T23:09:40  <zenhase> not really related to my project ;)
2008-05-04T23:09:56  <TheSheep> shame you cannot just puut yield in request.write
2008-05-04T23:10:05  <zenhase> hehe yes ;)
2008-05-04T23:10:13  <TheSheep> greenlets let you do something like that
2008-05-04T23:10:20  <zenhase> hmm
2008-05-04T23:10:29  <TheSheep> but that's an extra library
2008-05-04T23:11:15  <zenhase> also it's C isn't it?
2008-05-04T23:11:25  <zenhase> a C extension
2008-05-04T23:11:36  <TheSheep> yes
2008-05-04T23:11:41  <xorAxAx> as i probably once said, converting moin to yield while retaining request.write support is doable in a short timespan
2008-05-04T23:12:03  <xorAxAx> esp. because the translation of the write calls themselves is easy
2008-05-04T23:12:11  <xorAxAx> while the thinking about compatiblity is not :)
2008-05-04T23:12:20  <zenhase> that's the problem ;)
2008-05-04T23:12:21  <xorAxAx> or the paradigm changes in send_page and others :)
2008-05-04T23:12:29  <zenhase> i really would love this change
2008-05-04T23:12:57  <zenhase> but for this summer i wouldn't intermingle that with my project
2008-05-04T23:13:06  <TheSheep> I guess the most important woul be parser, send page and themes
2008-05-04T23:13:10  <zenhase> yes
2008-05-04T23:13:17  <zenhase> parser -> see waldi
2008-05-04T23:13:26  <zenhase> or parser/formatter that is
2008-05-04T23:14:12  <zenhase> his project will be a big gain for this change to happen
2008-05-04T23:14:20  <TheSheep> yup
2008-05-04T23:14:58  <zenhase> i think if parser/formatter is cleared, theme won't be much of a problem too ... and then send_page can dissolve
2008-05-04T23:22:26  <livibetter> I just installed mod_wsgi, and I can get rid of "moin.wsgi" from links of generated pages by adding properties = {'script_name' :''} to moin.wsgi
2008-05-04T23:22:56  <livibetter> so I check the code, it seems that wsgi version is different than fastcgi or cgi
2008-05-04T23:24:02  <livibetter> I need to modify request_wsgi.py to remove 'moin.wsgi' from links, did I miss some settings?
2008-05-04T23:26:11  <livibetter> sorry, "I can't get rid of ..."
2008-05-04T23:28:06  <ThomasWaldmann> livibetter: you don't need that, just use / for the wsgi alias
2008-05-04T23:28:29  <livibetter> you mean WSGIScriptAlias?
2008-05-04T23:28:45  <ThomasWaldmann> yes
2008-05-04T23:29:35  <livibetter> when I installed it under /, I can't get something like /robots.txt
2008-05-04T23:30:26  <ThomasWaldmann> use an alias line before the wsgi alias :)
2008-05-04T23:30:51  <livibetter> so I have to have more alias?
2008-05-04T23:31:09  <zenhase> hmm
2008-05-04T23:33:18  <ThomasWaldmann> livibetter: http://master17.moinmo.in/HelpOnInstalling/ApacheWithModWSGI
2008-05-04T23:39:46  <ThomasWaldmann> livibetter: i added the stuff there
2008-05-04T23:41:42  <livibetter> I see