MoinMoin: MoinMoinChat/Logs/moin-dev/2008-06-10

2008-06-10T00:26:44  <TheSheep> PawelPacana: lets just continue here
2008-06-10T00:28:01  <PawelPacana> ok, we stopped on choosing method to safely store item names on various filesystems
2008-06-10T00:28:28  <PawelPacana> (as long as wroking copy of repo lives on fs)
2008-06-10T00:29:43  <PawelPacana> you mentioned urlib.quote
2008-06-10T00:31:25  * PawelPacana is searching for posssible alternatives
2008-06-10T00:36:00  <TheSheep> I think that cgi module has some quoting mechanism too
2008-06-10T00:36:09  <TheSheep> but it might use the same as urllib.quote
2008-06-10T00:36:23  <TheSheep> we could use moin's quoting from wikiutil, I think
2008-06-10T00:37:05  <PawelPacana> just found that, http://moinmo.in/QuotingWikiNames
2008-06-10T00:39:12  <PawelPacana> ok, lets stick to internal quoting
2008-06-10T00:40:03  <TheSheep> ouch
2008-06-10T00:40:27  <PawelPacana> ?
2008-06-10T00:40:53  <TheSheep> maybe we should check the resulting fiel name length and trim it, appending a suffix for disambiguation?
2008-06-10T00:42:31  <PawelPacana> not following you
2008-06-10T00:43:15  <TheSheep> s/fiel/file/
2008-06-10T00:43:40  <TheSheep> windows has a limit on the length of filenames
2008-06-10T00:45:44  <PawelPacana> how does it work now in fs backend?
2008-06-10T00:47:58  <TheSheep> I think there is just a limit on the length of the page title
2008-06-10T00:48:03  <TheSheep> let me check
2008-06-10T00:48:08  <PawelPacana> to totally obfuscate working copy and provide some slowdown we could use sha1 hashes from names ;)
2008-06-10T00:48:38  <TheSheep> better DSA keys
2008-06-10T00:48:39  <TheSheep> ;)
2008-06-10T00:48:46  <ThomasWaldmann> :P
2008-06-10T00:53:37  <TheSheep> PawelPacana: looks like it doesn't do anything with file length
2008-06-10T00:54:00  <PawelPacana> got "Nie mogę zmienić nazwy strony ze względu na błąd systemu plików: [Errno 36] File name too long." when entering veery long name
2008-06-10T00:54:01  <TheSheep> PawelPacana: it's MoinMoin.wikiutil.QuoteWikinameFS()
2008-06-10T00:54:52  <TheSheep> sorry, quoteWikinameFS
2008-06-10T00:56:22  <TheSheep> we could do something like this: if the quoted filename is longer than, say, 240 characters, truncate it to 220 characters and append 20 characters of a checksum of the complete name
2008-06-10T00:56:46  <TheSheep> note that we need to store the original page name somewhere
2008-06-10T00:57:05  <TheSheep> and that it could have changed, so we need to store it in the revision
2008-06-10T00:59:47  <PawelPacana> antoher line in message, huh, this doesnt seem clean
2008-06-10T01:00:47  * PawelPacana wonders how does mercurial handle such cases
2008-06-10T01:02:40  <TheSheep> we could make the commit message look like e-mail/http headers
2008-06-10T01:02:55  <TheSheep> it can be easily parsed with ConfigParser
2008-06-10T01:03:18  <TheSheep> the first line is what appears in shortlog
2008-06-10T01:03:36  <TheSheep> so it should probably be the comment
2008-06-10T01:07:35  <PawelPacana> much cleaner, not relying on order
2008-06-10T01:07:54  <PawelPacana> bbim
2008-06-10T01:19:58  <TheSheep> I wonder if whole revision metainfo could be stored in this way
2008-06-10T01:31:22  <PawelPacana> back
2008-06-10T01:32:04  <PawelPacana> minute has stretched
2008-06-10T01:33:25  <PawelPacana> TheSheep: gtg and finish tomorrows presentation
2008-06-10T01:34:19  <PawelPacana> i`ll drop our previous an current conversation extracts  on wiki
2008-06-10T01:37:36  <PawelPacana> just to finish for today, i thought earlier about flat repo structure. if all meta go into hg and only Item content is left on working copy, there is no need for nesting
2008-06-10T01:38:36  <PawelPacana> when some pages are nested we could map this nesting in Item's name on working copy, just like pages are presented now
2008-06-10T01:39:14  <PawelPacana> so any Item moves will be renames
2008-06-10T01:39:26  <PawelPacana> what do you think of it?
2008-06-10T01:40:12  <TheSheep> yeah
2008-06-10T01:40:32  <TheSheep> but keeping too much info in the commit messages may make it hard to edit pages with external tools
2008-06-10T01:47:09  <PawelPacana> that is a drawback of lower magnitude, however 'no longer easy ctags iface' will be sad enough :|
2008-06-10T01:47:25  <PawelPacana> ok, enough for today
2008-06-10T01:47:31  <PawelPacana> good night
2008-06-10T07:10:00  <gayane> хи
2008-06-10T07:10:02  <gayane> hi
2008-06-10T07:10:15  <gayane> can anyone help me with mail support in moinmoin
2008-06-10T08:24:04  <dreimark> moin
2008-06-10T08:24:24  <dreimark> ThomasWaldmann: currently all attachfile test do have request as param
2008-06-10T08:44:47  <CIA-51> Reimar Bauer <rb.proj AT googlemail DOT com> default * 3687:207ae45bab18 1.7/MoinMoin/action/_tests/test_attachfile.py:
2008-06-10T08:44:47  <CIA-51> test_attachfile: added a test for add_attachment using a file object / removed
2008-06-10T08:44:47  <CIA-51> the duplicated str using test
2008-06-10T08:45:09  <dreimark> ThomasWaldmann: I do refactor it to a class definition
2008-06-10T09:10:23  <CIA-51> Reimar Bauer <rb.proj AT googlemail DOT com> default * 3688:57d85b82bc3e 1.7/MoinMoin/action/_tests/test_attachfile.py: test_attachfile: refactored to test class
2008-06-10T11:01:07  <xorAxAx> hmm, my student is not here
2008-06-10T11:08:59  <ThomasWaldmann> his irc client was here until 10:03
2008-06-10T11:09:13  <xorAxAx> yes
2008-06-10T11:09:33  <xorAxAx> its 18:xx local time for him
2008-06-10T11:35:18  <zenhase> moin
2008-06-10T11:35:26  <ThomasWaldmann> hi zenhase
2008-06-10T11:40:26  <ThomasWaldmann> xorAxAx: the last hunk in his last changeset looks like he is using tabs or wrong indentation
2008-06-10T11:54:42  <ThomasWaldmann> 2008-06-10 11:54:37,138 WARNING MoinMoin.search.builtin:1072 can't index: can't acquire lock
2008-06-10T11:54:58  <ThomasWaldmann> looks like we need a different method to rebuild the index for busy sites
2008-06-10T12:30:48  * dreimark tries to fix AttachmentUnzipCanCauseUnicodeDecodeError
2008-06-10T12:34:39  <ThomasWaldmann> but please fix it at the right place :)
2008-06-10T12:35:20  <ThomasWaldmann> (that is not where the crash happens that's shown in the traceback)
2008-06-10T12:35:30  <dreimark> I know
2008-06-10T12:36:19  <dreimark> ThomasWaldmann: it is not only the zip file you can get with add_atachment too
2008-06-10T12:40:11  <ThomasWaldmann> the current code does not use add_attachment
2008-06-10T12:40:31  <ThomasWaldmann> (but it better should :)
2008-06-10T12:40:39  <ThomasWaldmann>                                 outfile.write(zf.read(origname))
2008-06-10T12:40:46  <ThomasWaldmann> that sucks for big stuff
2008-06-10T12:42:31  <ThomasWaldmann>                                 # it's not allowed to zip a zip file so it is dropped
2008-06-10T12:42:34  <ThomasWaldmann>                                 if zipfile.is_zipfile(new_file):
2008-06-10T12:42:35  <ThomasWaldmann> why is that?
2008-06-10T12:44:19  <waldi> ThomasWaldmann: shutil.*?
2008-06-10T12:44:27  <dreimark> I wonder on that line too, it should be above. it should not first unpack and then remove it
2008-06-10T12:44:43  <dreimark> we do not want to have zip bombs
2008-06-10T12:44:59  * waldi .o0( 23.zip ... )
2008-06-10T12:45:16  <ThomasWaldmann> waldi: add_attachment already handles files in a sane way
2008-06-10T12:45:23  <waldi> ThomasWaldmann: okay
2008-06-10T12:45:51  <ThomasWaldmann> dreimark: a zip bomb is handled by other code
2008-06-10T12:47:28  <dreimark> I know but there is no reason to have zip files in one zip archive
2008-06-10T12:47:54  <ThomasWaldmann> you can't know if someone has a reason for it
2008-06-10T12:48:59  <ThomasWaldmann> offerering some download files as zip seems even rather reasonable
2008-06-10T12:49:21  <dreimark> no
2008-06-10T12:49:32  <ThomasWaldmann> sure :)
2008-06-10T12:49:46  <dreimark> if you do this you can get 42.zip unpacked manual
2008-06-10T12:50:17  <ThomasWaldmann> imagine someone wants to offer 50 zip files for download, but doesn't want to upload them all one-by-one
2008-06-10T12:51:01  <ThomasWaldmann> it won't unpack a zip bomb because it has too many files inside, or too big files or both
2008-06-10T12:51:20  <ThomasWaldmann> the uncompressed zip bomb itself is no problem as it is usually rather small
2008-06-10T12:51:34  <dreimark> it does unpack if you do it slow enough until you reach the limit
2008-06-10T12:51:58  <ThomasWaldmann> ?
2008-06-10T12:52:29  <dreimark> the first layer unpacks 50 zip files
2008-06-10T12:52:43  <ThomasWaldmann> that's allowed
2008-06-10T12:53:20  <dreimark> if they have stacked zip files you get with the next request the file limit exceeded
2008-06-10T12:53:33  <dreimark> and you have no real file
2008-06-10T12:53:57  <dreimark> the next request on each zip file will unpack 50 zip files again
2008-06-10T12:54:59  <ThomasWaldmann> if you can do it that (cumbersome) way, it just means that our limits are not completely doing what we want
2008-06-10T12:55:14  <ThomasWaldmann> it doesn't mean that zip files within a zip are a bad thing
2008-06-10T12:56:01  <ThomasWaldmann> but if I look at _do_unzip, the limits are per page, not per zip file
2008-06-10T12:56:31  <ThomasWaldmann> so there is no way beyond it
2008-06-10T12:56:31  <dreimark> yes, you can get for each page e.g. 200MB
2008-06-10T12:57:05  <dreimark> and we have this kind of quota only for zip files
2008-06-10T12:57:58  <ThomasWaldmann> so again,what's the problem with unpacking a zip member of a zip file?
2008-06-10T12:58:45  <dreimark> I do see only the problem that one can do exceed the 200MB easily.
2008-06-10T12:58:47  <ThomasWaldmann> (it seems to be rarely done, otherwise we would have a open bug about missing zip files maybe :)
2008-06-10T12:59:04  <ThomasWaldmann> dreimark: how?
2008-06-10T13:00:09  <dreimark> one can upload a small file and can umnapck it
2008-06-10T13:00:16  <dreimark> unpack
2008-06-10T13:00:31  <ThomasWaldmann> that won't get you beyond the limit
2008-06-10T13:01:18  <dreimark> if you don't get the size limit you will catch the no of files limit
2008-06-10T13:01:45  <ThomasWaldmann> maybe we can have another approach for that:
2008-06-10T13:02:07  <dreimark> yeah
2008-06-10T13:02:08  <ThomasWaldmann> remove the code denying extraction of a zip member file
2008-06-10T13:02:17  <ThomasWaldmann> and then try to exploit it
2008-06-10T13:02:33  <ThomasWaldmann> exploiting means going beyond the limits in your config
2008-06-10T13:02:43  <ThomasWaldmann> and then tell me how you did it
2008-06-10T13:02:54  <dreimark> hmm, I think I have done this sometime ago
2008-06-10T13:03:17  <ThomasWaldmann> I didn't try, but looking at the code you can't go beyond the limit.
2008-06-10T13:03:54  <ThomasWaldmann> (and this is completely unrelated to whether the member file is a zip or not)
2008-06-10T13:08:48  * ThomasWaldmann hopes dennda does a good job and we can get rid of the whole AttachFile stuff soon :)
2008-06-10T13:13:09  <dreimark> yeah you can't exceed the limit you can reach it only with stupid files
2008-06-10T13:13:43  <CIA-51> Bastian Blank <bblank@thinkmo.de> default * 3649:49499eedcde9 1.8-dom-bblank/MoinMoin/converter2/ (_tests/test_html_out.py html_out.py):
2008-06-10T13:13:43  <CIA-51> Support heading outline-level.
2008-06-10T13:13:43  <CIA-51> * MoinMoin/converter2/html_out.py
2008-06-10T13:13:43  <CIA-51>  Check the outline-level attribute and crop to [1-6].
2008-06-10T13:13:43  <CIA-51> * MoinMoin/converter2/_tests/test_html_out.py:
2008-06-10T13:13:44  <dreimark> if I think those zip files could be packages I think I do agree one qants them in one file
2008-06-10T13:13:45  <CIA-51>  Add tests for outline-level.
2008-06-10T13:13:51  <dreimark> ThomasWaldmann:
2008-06-10T13:15:15  <dreimark> if removed the docs needs to be changed  for "zip files in zip file will be ignored"
2008-06-10T13:15:41  <dreimark> and one i18n string could be killed too
2008-06-10T13:16:03  <dreimark> hmm, others have to be changed then too
2008-06-10T13:16:08  <ThomasWaldmann> killing i18n strings is no problem, just changing and adding them is
2008-06-10T13:16:22  <dreimark> HelpOnActions/AttachFile
2008-06-10T13:16:55  <dreimark> unfortunately zip files only is often used
2008-06-10T13:17:03  <dreimark> -> 1.8 ?
2008-06-10T13:17:07  <ThomasWaldmann> and I guess noone will complain that it didn't nuke his zip member ;)
2008-06-10T13:18:06  <dreimark> because they didn't complain or they did have accepted it
2008-06-10T13:18:56  <dreimark> bbl ~2h
2008-06-10T15:01:55  <johill> PawelPacana: was that you updating the hg backend page?
2008-06-10T15:17:37  <johill> ThomasWaldmann: it does look like the new account notification happens twice if the browser has a non-default language (i.e. not en)
2008-06-10T15:25:25  <ThomasWaldmann> that's strange :) it shouldn't care for the browser language at all. is that tip?
2008-06-10T15:27:15  <johill> pretty much, sec
2008-06-10T15:28:41  <johill> 1.7.0rc2
2008-06-10T15:30:32  <ThomasWaldmann> so the 2 mails you get are in which language?
2008-06-10T15:31:19  <ThomasWaldmann> http://hg.moinmo.in/moin/1.7/log/57d85b82bc3e/MoinMoin/events/emailnotify.py see the 2 changes 2 weeks ago
2008-06-10T15:31:45  <johill> both in english
2008-06-10T15:33:04  <ThomasWaldmann> i don't see how the code should send 2 emails for the same userid
2008-06-10T15:33:27  <ThomasWaldmann> but you maybe have 2 userids with same username?
2008-06-10T15:33:33  <johill> no, I checked all that
2008-06-10T15:33:39  <johill> only a single account has new user notification enabled
2008-06-10T15:34:36  <johill> and those changes should be in 1.7.0rc2 anyway, no
2008-06-10T15:34:39  <johill> ?
2008-06-10T15:34:52  <ThomasWaldmann> and aside from that: are there 2 different uids for the same email?
2008-06-10T15:35:11  <ThomasWaldmann> (no matter whether they have subscribed to that event)
2008-06-10T15:35:14  <johill> nope
2008-06-10T15:37:09  <ThomasWaldmann> i can try to reproduce it later. i don't see what could go wrong.
2008-06-10T15:41:17  <ThomasWaldmann> (yes, those changes are in rc2)
2008-06-10T15:44:47  <johill> just upgraded to the latest
2008-06-10T15:44:59  <johill> it doesn't _always_ happen which leads me to believe that it happens when the browser language is not en
2008-06-10T15:45:09  <johill> (it happened so far with people whose email address is @gmx etc)
2008-06-10T15:48:07  <ThomasWaldmann> i did get 3 "new user" mails yet, no dupes
2008-06-10T15:49:12  <johill> dunno
2008-06-10T15:56:10  <ThomasWaldmann> 4 :)
2008-06-10T15:56:47  <ThomasWaldmann> johill: do the dupes have same Message-ID?
2008-06-10T15:58:31  <ThomasWaldmann> or is it maybe a problem with the way the mailings are done?
2008-06-10T15:59:14  <ThomasWaldmann> moin's mails have from: wiki_from to: wiki_from and all real destinations are done by bcc
2008-06-10T15:59:21  <johill> let me see
2008-06-10T15:59:32  <johill> well the thing is I don't _always_ get dupes
2008-06-10T15:59:38  <johill> only on new user, and then not on all
2008-06-10T16:00:06  <johill> no, they do not have the same message ID, but the message ID is added by exim apparently
2008-06-10T16:00:37  <ThomasWaldmann> what happens if someone double clicks "create user"? :P
2008-06-10T16:00:55  <johill> the second ought to be refused since it already exists, no?
2008-06-10T16:01:33  <ThomasWaldmann> maybe some race condition
2008-06-10T16:02:31  <johill> but I have a hard time believing that so many people double-click
2008-06-10T16:02:41  <johill> I see that in maybe 1 out of 2 or 3 new users
2008-06-10T16:10:52  <johill> I added a small patch to add a traceback to the email... let's see
2008-06-10T16:16:52  <johill> can't reproduce...
2008-06-10T16:17:00  <johill> and double submit doesn't seem to lead to errors either
2008-06-10T16:18:25  <johill> I'll wait for it to happen again
2008-06-10T18:16:39  <ThomasWaldmann> the docs about the jabber bot are a bit thin...
2008-06-10T18:46:59  <PawelPacana> johill: that was me, i forgot to log in after upgrade :o
2008-06-10T18:56:19  <johill> PawelPacana: I added a few notes
2008-06-10T19:08:31  <ThomasWaldmann> http://www.southpolestrip.com/comic/51/South-Pole-50-Being-different/ < LOL
2008-06-10T20:31:11  <ThomasWaldmann> >           AssertionError: (inconsistently failed then succeeded)
2008-06-10T20:32:30  <johill> ?
2008-06-10T20:33:26  <ThomasWaldmann> search query parser tests
2008-06-10T20:33:36  <ThomasWaldmann> (after fixing some of the stuff)
2008-06-10T21:11:17  <dreimark> we need a shorter way to get to the advanced search if you've done one title search already
2008-06-10T21:15:21  <dreimark> in the form the "Go get it!" should be more visible
2008-06-10T21:15:32  <dreimark> e.g. bold
2008-06-10T21:23:01  <CIA-51> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3689:23a5fefa9553 1.7/MoinMoin/search/_tests/test_search.py: search query parser tests: adapt to current query parser behaviour
2008-06-10T21:28:11  <dreimark> just only an idea http://moinmo.in/ReimarBauer?action=AttachFile&do=view&target=codec_problem.diff (not completed for the msg)
2008-06-10T21:29:16  <dreimark> for this  http://moinmo.in/MoinMoinBugs/AttachmentUnzipCanCauseUnicodeDecodeError and http://moinmo.in/MoinMoinBugs/UnzipFilesWithNonAsciiName
2008-06-10T21:29:51  <CIA-51> Florian Krupicka <florian.krupicka@googlemail.com> default * 3661:30ac757e72de 1.8-wsgi-fkrupicka/MoinMoin/ (web/apps.py wsgiapp.py): Added HTTPException catching middleware
2008-06-10T21:29:53  <CIA-51> Florian Krupicka <florian.krupicka@googlemail.com> default * 3662:ddcd459c3255 1.8-wsgi-fkrupicka/MoinMoin/ (web/contexts.py wsgiapp.py): Restructured contexts classes for wrapping contexts in each other
2008-06-10T21:29:54  <CIA-51> Florian Krupicka <florian.krupicka@googlemail.com> default * 3663:75b0b7f32423 1.8-wsgi-fkrupicka/MoinMoin/web/ (contexts.py exceptions.py): Replace makeForbidden with HTTPException variant for WSGI
2008-06-10T21:29:55  <CIA-51> Florian Krupicka <florian.krupicka@googlemail.com> default * 3664:fbbc33a3d40a 1.8-wsgi-fkrupicka/MoinMoin/ (web/contexts.py wsgiapp.py): New isSpiderAgent property on RequestContext(and fixed some copy'n'paste messup)
2008-06-10T21:29:57  <CIA-51> Florian Krupicka <florian.krupicka@googlemail.com> default * 3665:7e59a7989b23 1.8-wsgi-fkrupicka/MoinMoin/ (web/contexts.py wsgiapp.py): Make getText into a context property
2008-06-10T21:30:00  <CIA-51> Florian Krupicka <florian.krupicka@googlemail.com> default * 3666:40501b3e2176 1.8-wsgi-fkrupicka/MoinMoin/ (web/contexts.py wsgiapp.py): Moved some attributes on Request into RenderContext
2008-06-10T21:30:03  <CIA-51> Florian Krupicka <florian.krupicka@googlemail.com> default * 3667:264e5687ef1c 1.8-wsgi-fkrupicka/MoinMoin/web/utils.py: Factoring out more parts of request.run
2008-06-10T21:30:34  <zenhase> sorry if i don't always push right away
2008-06-10T21:31:10  <zenhase> sometimes i restructure commits after i've done them to my local repo
2008-06-10T21:41:51  <dreimark> entered the patch to UnzipFilesWithNonAsciiName too
2008-06-10T21:41:59  * dreimark htg
2008-06-10T21:42:03  <dreimark> gn
2008-06-10T21:42:49  <ThomasWaldmann> using decodeUnknown would be too easy
2008-06-10T22:00:24  <dreimark> decodeUnknownInput does not work because of isinstance(text, unicode)
2008-06-10T22:00:38  <dreimark> everything you enter to the form is by default unicode
2008-06-10T22:01:11  <dreimark>  return text will not remove any of the wrong signs
2008-06-10T22:01:18  <dreimark> ThomasWaldmann:
2008-06-10T22:03:50  <dreimark> I would like a solution which replaces those signs
2008-06-10T22:07:04  <dreimark> it is an idea about replacing them similiar to taintfilename
2008-06-10T22:08:28  * dreimark has not slept much the last nights, sorry 
2008-06-10T22:10:17  <ThomasWaldmann> then it is already decoded wrong
2008-06-10T22:10:30  <ThomasWaldmann> s/signs/chars/
2008-06-10T22:10:44  <ThomasWaldmann> I'll have a look.
2008-06-10T22:12:50  <ThomasWaldmann> the error msgs of unzip are wrong
2008-06-10T22:33:48  * ThomasWaldmann rewrite unzipping code