2008-08-09T00:00:17 * dennda tries to hack the former error
2008-08-09T00:00:52 <dennda> oh
2008-08-09T00:01:14 <dennda> looks like an issue in pycdb
2008-08-09T00:01:16 <dennda> whatever that is
2008-08-09T00:05:00 <xorAxAx> ThomasWaldmann: can you write a unittest triggering the fd bug?
2008-08-09T00:05:16 <xorAxAx> hmm
2008-08-09T00:05:30 <xorAxAx> you probably only need to create enough items
2008-08-09T00:07:03 <xorAxAx> or should i try?
2008-08-09T00:07:39 <dennda> why not? :)
2008-08-09T00:08:50 <ThomasWaldmann> it falls over every practical test, so why should I?
2008-08-09T00:09:19 <ThomasWaldmann> the flatfile backend also does not work
2008-08-09T00:09:42 <xorAxAx> ThomasWaldmann: i am already writing one
2008-08-09T00:09:44 <ThomasWaldmann> so basically conversion does not work because of no working target backend (except hg maybe)
2008-08-09T00:09:59 <xorAxAx> well if the issue is showing up only when using it but not in the unittests, a test is reaaally really needed
2008-08-09T00:10:49 <ThomasWaldmann> just create a list of some thousand items
2008-08-09T00:11:04 <xorAxAx> hmm, creating 1000 items works
2008-08-09T00:11:34 <ThomasWaldmann> and don't use memory backend
2008-08-09T00:11:45 <PawelPacana> (hg has some problem with unicode and revision meta now)
2008-08-09T00:12:14 <xorAxAx> ThomasWaldmann: hehe
2008-08-09T00:13:49 <xorAxAx> ThomasWaldmann: can i see the traceback?
2008-08-09T00:14:40 <xorAxAx> hmm, it looks like it might be non exhausted generator
2008-08-09T00:14:40 <xorAxAx> s
2008-08-09T00:17:06 <xorAxAx> i cant reproducei t
2008-08-09T00:17:14 <xorAxAx> on teh command line :)
2008-08-09T00:17:18 <ThomasWaldmann> File "/home/moin/moin-1.8-storage/MoinMoin/script/migration/backend.py", line 58, in clone
2008-08-09T00:17:21 <ThomasWaldmann> File "/home/moin/moin-1.8-storage/MoinMoin/storage/backends/fs.py", line 162, in create_item
2008-08-09T00:17:25 <ThomasWaldmann> File "/home/moin/moin-1.8-storage/MoinMoin/storage/backends/fs.py", line 156, in has_item
2008-08-09T00:17:27 <ThomasWaldmann> File "/home/moin/moin-1.8-storage/MoinMoin/storage/backends/fs.py", line 141, in _get_item_id
2008-08-09T00:17:31 <ThomasWaldmann> File "/home/moin/moin-1.8-storage/MoinMoin/support/pycdb.py", line 50, in __init__
2008-08-09T00:17:34 <ThomasWaldmann> IOError: [Errno 24] Too many open files: '/home/moin/moin-1.8-storage/output/pages/name-mapping'
2008-08-09T00:17:47 <xorAxAx> ok
2008-08-09T00:23:59 <xorAxAx> ahhhh
2008-08-09T00:24:08 * xorAxAx has an idea
2008-08-09T00:25:33 <xorAxAx> but i cant reproduce it
2008-08-09T00:25:34 <xorAxAx> hmm
2008-08-09T00:26:14 <xorAxAx> ah
2008-08-09T00:27:26 <xorAxAx> no success
2008-08-09T00:27:42 <xorAxAx> looks like we need a tracing backend :)
2008-08-09T00:27:54 <xorAxAx> ThomasWaldmann: how do i need to call the mig script?
2008-08-09T00:28:19 <dennda> from MoinMoin.script.migration.backend import clone
2008-08-09T00:28:30 <dennda> clone(data_backend_source, data_backend)
2008-08-09T00:28:58 <xorAxAx> dennda: ?
2008-08-09T00:29:19 <dennda> Didn't you just ask for that information?
2008-08-09T00:30:18 <xorAxAx> no, i asked about the script
2008-08-09T00:31:06 <xorAxAx> wuah, tabs in wikiconfig_lcaol
2008-08-09T00:32:05 <xorAxAx> ok, set it up now
2008-08-09T00:32:23 <xorAxAx> btw, some more userfriendly name scheme is needed
2008-08-09T00:32:36 <xorAxAx> and plugin integration for storage backends :)
2008-08-09T00:33:05 <xorAxAx> and a migration approach (how will we call modules so that moin ships always the current version and the older ones?)
2008-08-09T00:35:21 <xorAxAx> ok, finally saw the exception
2008-08-09T00:36:21 <PawelPacana> xorAxAx: wiki/server/moin migration backend -t data
2008-08-09T00:36:50 <xorAxAx> PawelPacana: not sure if depending on a _source config setting is nice ...
2008-08-09T00:37:00 <xorAxAx> maybe some evalable arg is better, no iddea
2008-08-09T00:37:19 <PawelPacana> that was in -s option before
2008-08-09T00:37:35 <PawelPacana> Thomas didnt like it much
2008-08-09T00:38:20 <xorAxAx> sigh
2008-08-09T00:41:00 <PawelPacana> resurecting this option is no big deal, but before lets better discuss and decide how it should look like. im not insisting on how it looks now
2008-08-09T00:43:08 <xorAxAx> yeah :)
2008-08-09T00:43:32 <xorAxAx> well, the idea is that just patching your config with a value that will be invalid after running the script doesnt make a lot of sense
2008-08-09T00:45:07 <xorAxAx> uhh, _get_revision is ugly
2008-08-09T00:45:14 <xorAxAx> why does it have an underscore?
2008-08-09T00:45:31 <dennda> in class Backend?
2008-08-09T00:45:36 <xorAxAx> yes
2008-08-09T00:45:47 <xorAxAx> hmm
2008-08-09T00:45:50 <dennda> well, you don't get the revision of a backend, but of an item
2008-08-09T00:45:55 <xorAxAx> its supposed to be overriden but not called
2008-08-09T00:46:01 <xorAxAx> dennda: ?
2008-08-09T00:46:02 <dennda> that method is called internally when item.get_revision() is called
2008-08-09T00:46:26 <xorAxAx> yes, i just havent seen methods that should be overriden even though they are marked private
2008-08-09T00:46:35 <xorAxAx> in python
2008-08-09T00:46:56 <dennda> isn't "private" two underscores?
2008-08-09T00:47:27 <dennda> afaict we used the leading underscore because you are not supposed to call that method when you use the backend
2008-08-09T00:47:59 <xorAxAx> _ means internal use, __ means enforce name mangling
2008-08-09T00:48:13 <dennda> well so it's correct
2008-08-09T00:48:22 <xorAxAx> depends on what you call internal :)
2008-08-09T00:48:39 <dennda> I do call that internal
2008-08-09T00:48:40 <xorAxAx> but i think its ok as long as its documented (!!!)
2008-08-09T00:55:31 <ThomasWaldmann> PawelPacana: revs.extend([item.get_revision(revno) for revno in item.list_revisions()])
2008-08-09T00:55:42 <ThomasWaldmann> i am quite sure this is a killer
2008-08-09T00:55:58 <ThomasWaldmann> because revisions tend to have open files
2008-08-09T00:56:04 <xorAxAx> hmm?
2008-08-09T00:56:12 <xorAxAx> underlay only has one rev each
2008-08-09T00:56:26 <ThomasWaldmann> and you create ALL revision items there X)
2008-08-09T00:57:07 <ThomasWaldmann> ALL means all revs of all items
2008-08-09T00:57:52 <xorAxAx> in different loops, no?
2008-08-09T01:00:44 <ThomasWaldmann> hmm, i add some lazyness to fs17
2008-08-09T01:01:30 <ThomasWaldmann> nevertheless, it is pretty evil when used with a simple backend
2008-08-09T01:02:36 <ThomasWaldmann> for the purpose it has, (timestamp, itemname, revno) would be enough
2008-08-09T01:06:52 <xorAxAx> hmm?
2008-08-09T01:07:10 <xorAxAx> the code you are looking at isnt using news() yet
2008-08-09T01:08:03 <ThomasWaldmann> i am not talking about news
2008-08-09T01:09:06 <xorAxAx> about the first lines of clone, right?
2008-08-09T01:11:10 <ThomasWaldmann> yes
2008-08-09T01:13:23 <xorAxAx> thats legacy code
2008-08-09T01:13:31 <xorAxAx> it will vanish soon hopefully :)
2008-08-09T01:13:38 <xorAxAx> when the code is using news()
2008-08-09T01:14:29 * xorAxAx is currently writing a tracing backend :)
2008-08-09T01:14:41 <CIA-53> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 4693:841b2833d361 1.8-storage-cdenter/MoinMoin/storage/backends/ (fs.py fs17.py): fs/fs17 storage backend: add comments about file handle usage/lifetime, consistently use open()
2008-08-09T01:15:22 <xorAxAx> it will generate a piece of code
2008-08-09T01:15:28 <xorAxAx> instead of a data directory :)
2008-08-09T01:17:43 <xorAxAx> probably useful to generate a failing unit test
2008-08-09T01:18:42 <xorAxAx> dennda: there is a set of methods that return item or revision instances. are all methods of that set defined in the backend?
2008-08-09T01:18:51 <xorAxAx> dennda: or also in the item class?
2008-08-09T01:21:34 <xorAxAx> hmm, no, there is Item.create_rev*
2008-08-09T01:25:42 <xorAxAx> for a sneak preview: http://paste.pocoo.org/show/81663
2008-08-09T01:25:46 <xorAxAx> i love python :)
2008-08-09T01:31:09 <PawelPacana> ThomasWaldmann: pull new cset
2008-08-09T01:35:04 <PawelPacana> xorAxAx: thingie, klass :)
2008-08-09T01:35:45 <xorAxAx> PawelPacana: well, klass is actually some class_ alias
2008-08-09T01:40:05 <ThomasWaldmann> PawelPacana: trying it. didnt crash yet.
2008-08-09T01:40:53 <ThomasWaldmann> wishlist item: --verbose showing progress somehow (item names revs whatever)
2008-08-09T01:42:14 <ThomasWaldmann> PawelPacana: File "/home/moin/moin-1.8-storage/MoinMoin/storage/backends/fs.py", line 147, in get_item
2008-08-09T01:42:17 <ThomasWaldmann> raise NoSuchItemError("No such item, %r" % (itemname))
2008-08-09T01:42:22 <ThomasWaldmann> MoinMoin.storage.error.NoSuchItemError: No such item, u'tezd'
2008-08-09T01:42:32 <xorAxAx> we will be able to use the tracing backend also to benchmark :)
2008-08-09T01:42:37 <ThomasWaldmann> called from clone:67
2008-08-09T01:42:59 <xorAxAx> or, profile
2008-08-09T01:43:13 <ThomasWaldmann> yeah, for unusual usage patterns :P
2008-08-09T01:43:44 <xorAxAx> why unusual?
2008-08-09T01:43:53 <xorAxAx> the idea is to plug it in and do any kind of things
2008-08-09T01:44:04 <xorAxAx> like clicking wildy in the wiki :)
2008-08-09T01:44:14 * xorAxAx runs it with the importer now
2008-08-09T01:44:19 <xorAxAx> umm, migrator
2008-08-09T01:44:54 <ThomasWaldmann> PawelPacana: creating a new item should not depend on revno 0
2008-08-09T01:45:48 <xorAxAx> IOError: [Errno 24] Too many open files: '/tmp/codebuffer.py'
2008-08-09T01:45:49 <xorAxAx> gah :)
2008-08-09T01:46:47 <xorAxAx> ahhhh
2008-08-09T01:46:54 <xorAxAx> the file handles are exhausted by ThomasWaldmann's code
2008-08-09T01:47:11 <xorAxAx> looks like he wanted to tell us that
2008-08-09T01:47:25 <xorAxAx> by the above code quoting of PawelPacana's code
2008-08-09T01:47:31 <ThomasWaldmann> you run outdated code :)
2008-08-09T01:47:39 <xorAxAx> yeah, obviously :)
2008-08-09T01:55:24 <ThomasWaldmann> and the problem very likely also happens with fs backend as soon as you create lots of revision items and access any data or metadata
2008-08-09T01:57:37 <xorAxAx> ThomasWaldmann: hmm, why?
2008-08-09T01:57:46 <xorAxAx> why does your backend keep files open?
2008-08-09T01:58:03 <ThomasWaldmann> mine doesn't any more
2008-08-09T01:58:18 <ThomasWaldmann> but fs (intentionally) does that
2008-08-09T01:58:37 <xorAxAx> it does?
2008-08-09T01:58:47 <ThomasWaldmann> see all those XXX i added
2008-08-09T01:59:57 <ThomasWaldmann> the evil thing mine currently does is that it doesn't lazyload page metadata/data
2008-08-09T02:00:38 <ThomasWaldmann> but that's no big problem as long as clone is not evil :)
2008-08-09T02:02:31 <CIA-54> Pawel Pacana <pawel.pacana@gmail.com> default * 4692:099d64720712 1.8-mercurialbackend-ppacana/MoinMoin/script/migration/backend.py: (migration) Not relying on revno == 0, additional simplification.
2008-08-09T02:04:12 <PawelPacana> try now
2008-08-09T02:05:16 <PawelPacana> i neither have failing dataset, nor unittests for that (yet)
2008-08-09T02:05:25 <ThomasWaldmann> 1204529212000000 99999999 SAVENEW tezd CopyPage action test
2008-08-09T02:05:28 <ThomasWaldmann> 1204536480028847 00000002 SAVE tezd 87.180.65.28 p57B4411C.dip0.t-ipconnect.de 1012653240.98
2008-08-09T02:05:28 <xorAxAx> ok, pushed the tracing backend
2008-08-09T02:05:39 <ThomasWaldmann> that's the complete edit-log of a page...
2008-08-09T02:06:11 <xorAxAx> ThomasWaldmann: so?
2008-08-09T02:06:44 <ThomasWaldmann> there is only revisions/00000001
2008-08-09T02:07:47 <xorAxAx> so? :)
2008-08-09T02:07:59 <xorAxAx> i mean, you pasted the reason :-)
2008-08-09T02:08:40 <ThomasWaldmann> i just wanted to say that input data is sometimes corrupt
2008-08-09T02:10:10 <xorAxAx> yeah, but thats only a problem of your backend :)
2008-08-09T02:10:52 <xorAxAx> the above corruption isnt a hardware/nfs etc. issue but most likely a coding issue
2008-08-09T02:11:16 <xorAxAx> luckily its disabled nowaday
2008-08-09T02:11:16 <xorAxAx> s
2008-08-09T02:11:29 <xorAxAx> s/luckily/fortunately
2008-08-09T02:11:31 <dennda> As for that wishlist-progress thing
2008-08-09T02:11:47 <dennda> for the --verbose parameter, just look if it is in sys.argv?
2008-08-09T02:11:55 <dennda> or is there a moin-approach to this?
2008-08-09T02:12:07 <ThomasWaldmann> PawelPacana: looks better :)
2008-08-09T02:12:45 <PawelPacana> dennda: what are you planning to do with this --verbose?
2008-08-09T02:12:59 <dennda> PawelPacana: < ThomasWaldmann> wishlist item: --verbose showing progress somehow (item names revs whatever
2008-08-09T02:13:22 <dennda> I could hack that in
2008-08-09T02:13:58 <dennda> boy is it hot
2008-08-09T02:14:10 <dennda> my heartrate is at least at 3 times usual speed and I cannot open the window
2008-08-09T02:14:31 <xorAxAx> 14° here :)
2008-08-09T02:14:43 <xorAxAx> good night :)
2008-08-09T02:14:55 <dennda> one big tower, two big tfts, a laptop, a 5.1 system and me
2008-08-09T02:15:04 <dennda> heat up a few m^2 rather fast
2008-08-09T02:15:45 <ThomasWaldmann> dennda: please merge pawel's repo
2008-08-09T02:15:58 <dennda> ok, just a sec
2008-08-09T02:16:15 <PawelPacana> dennda: if you really want to add it look at MoinMoin/script/migration/backend.py
2008-08-09T02:16:36 <dennda> PawelPacana: Where else? :)
2008-08-09T02:16:39 <PawelPacana> options in __init__
2008-08-09T02:17:23 <dennda> ah ok
2008-08-09T02:20:14 <dennda> merged
2008-08-09T02:22:13 <PawelPacana> wishlist: py.test-like output, collect names of failed revisions and write in sum-up, hehe :)
2008-08-09T02:22:39 <dennda> would clearly help to know optparse here %-)
2008-08-09T02:26:21 <ThomasWaldmann> MoinMoin.storage.error.RevisionNumberMismatchError: The latest revision is -1, thus you cannot create revision number 2. The revision number must be latest_revision + 1.
2008-08-09T02:26:36 <ThomasWaldmann> that's completely useless without telling the item name
2008-08-09T02:26:51 <dennda> changing
2008-08-09T02:27:10 <dennda> which backend are you using?
2008-08-09T02:27:27 <CIA-54> Alexander Schremmer <alex AT alexanderweb DOT de> default * 4695:af391e20dc1f 1.8-storage-cdenter/MoinMoin/storage/ (_tests/test_backends_memory.py backends/memory.py): A TracingBackend in the MemoryBackend module. Can be used to generate snippets for unittests or profile upper layers in terms of executed operations.
2008-08-09T02:27:27 <CIA-54> Alexander Schremmer <alex AT alexanderweb DOT de> default * 4696:a63d1f4dd86e 1.8-storage-cdenter/MoinMoin/storage/backends/ (fs.py fs17.py): Merged main repo of the storage branch
2008-08-09T02:27:42 <dennda> memory i suppose
2008-08-09T02:29:41 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4701:07b77d778271 1.8-storage-cdenter/MoinMoin/storage/backends/memory.py: storage: MemoryBackend: Make Exception error more useful by providing further information
2008-08-09T02:29:42 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4702:5b78a542b6c2 1.8-storage-cdenter/MoinMoin/ (script/migration/backend.py storage/backends/memory.py): Merge my own repo, 1.8-storage-cdenter
2008-08-09T02:29:47 <dennda> ThomasWaldmann: there you go
2008-08-09T02:30:42 <ThomasWaldmann> no, fs
2008-08-09T02:32:48 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4703:1c1eaee98c5d 1.8-storage-cdenter/MoinMoin/storage/backends/fs.py: storage: FSBackend: Make Exception error more useful by providing further information
2008-08-09T02:38:53 <dennda> ThomasWaldmann: Do you want the migration script to report progress to stdout when --verbose is set? or rather some debug level?
2008-08-09T02:41:22 <ThomasWaldmann> try using logging.info
2008-08-09T02:41:40 <dennda> hm
2008-08-09T02:41:43 <dennda> stupid question:
2008-08-09T02:42:26 <dennda> print "%s" % "foo" <-- How to suppress the newline? I can't just do "%s", % "foo" since that, of course, raises a SyntaxError
2008-08-09T02:43:56 <ThomasWaldmann> try using logging.info
2008-08-09T02:44:17 <dennda> ah sure, the newline problem doesn't exist there ,9
2008-08-09T02:47:29 <dennda> how verbose do you want it to be? :)
2008-08-09T02:48:52 <dennda> PawelPacana: still there?
2008-08-09T02:51:50 <PawelPacana> for a few minutes
2008-08-09T02:51:58 <dennda> ok, two questions:
2008-08-09T02:52:18 <dennda> in the migration script, why do you sort the revisions by timestamp?
2008-08-09T02:53:37 <PawelPacana> to preserve news()
2008-08-09T02:53:51 <dennda> news sorts the revisions by timestamp itself
2008-08-09T02:53:56 <ThomasWaldmann> ok, converter runs now, after removing all sorts of crap pages
2008-08-09T02:54:12 <dennda> imho that is not needed there
2008-08-09T02:54:36 <dennda> I would test whether it makes a difference if only I knew how to call that script properly %)
2008-08-09T02:54:52 <dennda> wiki/server/moin migration backend -t data didn't help that much
2008-08-09T02:55:03 <ThomasWaldmann> now you see how important docstrings are :P
2008-08-09T02:55:15 <dennda> I have always been a big fan of docstrings :)
2008-08-09T02:56:11 <PawelPacana> dennda: that depends on implementaion, i dont think fs is doing it
2008-08-09T02:56:12 <ThomasWaldmann> and it is MoinMoin/script/moin.py migration backend --type data
2008-08-09T02:56:18 <dennda> it should be
2008-08-09T02:56:30 <PawelPacana> should?
2008-08-09T02:57:03 <PawelPacana> isnt that suboptimal compared to just appending to file?
2008-08-09T02:57:24 <dennda> should. at least that's what the docstring in the ABC says the method does
2008-08-09T02:58:00 <dennda> well, recentchanges is expecting to get the revisions in that order
2008-08-09T02:58:03 <PawelPacana> ah
2008-08-09T02:58:10 <dennda> you can't do it on the migration-script-layer
2008-08-09T02:58:10 <PawelPacana> i know what you mean
2008-08-09T02:58:24 <PawelPacana> previously mig script was using news to iterate
2008-08-09T02:58:57 <PawelPacana> but compat backends dont have news
2008-08-09T02:59:13 <dennda> compat?
2008-08-09T02:59:18 <PawelPacana> fs17
2008-08-09T02:59:29 <dennda> yes, there's no need for that
2008-08-09T02:59:33 <dennda> but what does compat mean?
2008-08-09T03:00:01 <PawelPacana> compatibility
2008-08-09T03:00:04 <dennda> ahh
2008-08-09T03:00:09 <dennda> yes, they don't need it
2008-08-09T03:00:20 <dennda> without verifying I claim that the line in question can be removed
2008-08-09T03:01:41 <dennda> I'll examine it
2008-08-09T03:02:30 <PawelPacana> examine what?
2008-08-09T03:03:17 <dennda> err
2008-08-09T03:03:26 <dennda> I am looking at old code
2008-08-09T03:04:04 <dennda> sorry ;)
2008-08-09T03:06:36 <PawelPacana> i'll rather examine bed
2008-08-09T03:06:38 <PawelPacana> gn
2008-08-09T03:06:43 <dennda> sleep well friend
2008-08-09T03:27:49 <dennda> hm
2008-08-09T03:28:11 <dennda> :w
2008-08-09T03:28:14 <dennda> err ;)
2008-08-09T03:32:14 <dennda> just rewrote clone()
2008-08-09T03:47:56 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4704:e112ab7cca4e 1.8-storage-cdenter/MoinMoin/script/migration/backend.py: storage: backend migration script: Rewriting clone() function. Making it more straight-forward and clear. Adding comments
2008-08-09T03:55:12 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4705:2fa0639d0d8c 1.8-storage-cdenter/MoinMoin/script/migration/backend.py: storage: backend migration script: add --verbose option to provide information about progress
2008-08-09T03:55:33 <dennda> ThomasWaldmann: please check if you like it that way. It produces quite a bit of output :)
2008-08-09T03:56:02 <dennda> RC is a bit expensive at the moment
2008-08-09T04:01:00 <dennda> How do we want to handle aborted migrations?
2008-08-09T04:01:09 <dennda> Just continue them next time?
2008-08-09T04:01:26 <dennda> (that should be rather easy to implement the way clone works now)
2008-08-09T04:17:19 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4706:32042452d216 1.8-storage-cdenter/MoinMoin/script/migration/backend.py: storage: backend migration script now theoretically allows continueing of aborted migrations. fset --verbose to False by default
2008-08-09T04:19:36 <dennda> hehe, that gives us a head start on http://cia.vc/ :) rank 4 of most active projects and I am in the most active authors top ten :)
2008-08-09T04:19:42 <dennda> bot ok, there's nearly 20 hours left
2008-08-09T04:19:47 <dennda> but*
2008-08-09T09:20:33 <ThomasWaldmann> moin
2008-08-09T09:20:57 <ThomasWaldmann> dennda: as I already said yesterday, metadata are not processed correctly
2008-08-09T09:21:24 <ThomasWaldmann> for the case when there are multiple text lines with same metadata key
2008-08-09T09:21:40 <ThomasWaldmann> usually this happens for comments, sometimes also for ACLs
2008-08-09T09:25:52 * ThomasWaldmann sees "too many open files" for a moinmo.in on fs backend
2008-08-09T09:29:50 <ThomasWaldmann> http://moinmo.in:8088/MoinMoinBugs < funny, that error msg is under heading "New bugs" :P
2008-08-09T09:44:54 <xorAxAx> PawelPacana: well, as i said, it should use news()
2008-08-09T09:45:01 <xorAxAx> PawelPacana: and the base class should probably grow a generic news
2008-08-09T09:45:15 <xorAxAx> PawelPacana: and we need an api to store the item creation time etc.
2008-08-09T09:46:00 <ThomasWaldmann> + logging.info("Copying '%s'" % old_item.name)
2008-08-09T09:46:22 <ThomasWaldmann> dennda: how do you think this works for chinese item names?
2008-08-09T09:47:34 <xorAxAx> ThomasWaldmann: and which files are opened?
2008-08-09T09:47:42 <xorAxAx> ThomasWaldmann: name-mapping a few times or others?
2008-08-09T09:48:24 <ThomasWaldmann> likely many page revision metadata objects alive
2008-08-09T09:49:11 <xorAxAx> and they have open files?
2008-08-09T09:50:36 <ThomasWaldmann> did you look at all those XXX in fs.py?
2008-08-09T09:51:29 <xorAxAx> not yet :)
2008-08-09T09:51:53 <xorAxAx> ok, saw it :)
2008-08-09T09:53:21 <ThomasWaldmann> the way the api works now kind of enforces that problem
2008-08-09T09:53:41 <xorAxAx> why?
2008-08-09T09:54:02 <xorAxAx> not the external api
2008-08-09T09:54:09 <xorAxAx> but probably the "internal" one
2008-08-09T09:54:46 <xorAxAx> which is based on instantiating Item() or the rev classes and attaching things to it
2008-08-09T09:57:57 <ThomasWaldmann> yes, not the public storage api. i think it is the callers of iteritems/search_item
2008-08-09T10:00:26 <ThomasWaldmann> dennda: iteritems method is confusing. fix docstring please (and compare it with dict.iteritems).
2008-08-09T10:10:55 <ThomasWaldmann> getPageDict should die in favor of a real backend.iteritems()
2008-08-09T11:53:35 <zenhase> ,oin
2008-08-09T11:53:41 <zenhase> s/,/m
2008-08-09T11:59:08 <ThomasWaldmann> hi zenhase
2008-08-09T12:53:03 <ThomasWaldmann> zenhase: frontend.CGIFrontEnd is used where?
2008-08-09T12:55:30 <ThomasWaldmann> zenhase: nice NOTES.moin
2008-08-09T12:56:34 <xorAxAx> mitsuhiko: ping
2008-08-09T12:58:12 <ThomasWaldmann> zenhase: try starting moin.cgi on cmdline
2008-08-09T12:58:29 <PawelPacana> xorAxAx: what was that about 09:44 < xorAxAx> PawelPacana: well, as i said, it should use news() ?
2008-08-09T12:59:30 <ThomasWaldmann> zenhase: MoinMoin.request has unused code, e.g. find_remote_addrs
2008-08-09T13:01:05 <xorAxAx> PawelPacana: well, clone() should sort revs on its own
2008-08-09T13:01:09 <xorAxAx> PawelPacana: but call news()
2008-08-09T13:01:50 <PawelPacana> we agreed on that
2008-08-09T13:01:57 <PawelPacana> and clone was using news...
2008-08-09T13:02:07 <xorAxAx> hmm?
2008-08-09T13:03:09 <xorAxAx> you mean you removed it again?
2008-08-09T13:03:19 <PawelPacana> http://hg.moinmo.in/moin/1.8-mercurialbackend-ppacana/rev/b8f590eb0b4a
2008-08-09T13:03:26 <PawelPacana> this changeset was using news
2008-08-09T13:03:42 <PawelPacana> then thomas complained (no news() in fs17)
2008-08-09T13:03:55 <xorAxAx> gah
2008-08-09T13:04:15 <xorAxAx> as i said, in that case the fix is to implement news() i nthe base class of the backends
2008-08-09T13:04:18 <xorAxAx> but not in clone
2008-08-09T13:04:20 <xorAxAx> layering violation!
2008-08-09T13:04:32 <PawelPacana> so i did this thingie with sorting
2008-08-09T13:04:45 <PawelPacana> then dennda was supposed to add verbose
2008-08-09T13:04:56 <xorAxAx> so can this be fixed please?
2008-08-09T13:05:00 <PawelPacana> and first thing what he did
2008-08-09T13:05:16 <PawelPacana> was rewritng clone to the first version
2008-08-09T13:05:22 <PawelPacana> just compare
2008-08-09T13:05:39 <xorAxAx> "first version"?
2008-08-09T13:05:44 <PawelPacana> http://hg.moinmo.in/moin/1.8-storage-cdenter/rev/e112ab7cca4e
2008-08-09T13:06:19 <PawelPacana> "first" - using unsorted iteritems
2008-08-09T13:06:29 <xorAxAx> well, both is wrong
2008-08-09T13:06:34 <PawelPacana> iterating through item, and all revisions
2008-08-09T13:07:32 <xorAxAx> do you understand the idea and why both is wrong?
2008-08-09T13:07:33 <ThomasWaldmann> maybe we could just document whether clone needs to create items in historical order or not
2008-08-09T13:07:35 <xorAxAx> dennda: you too?
2008-08-09T13:07:47 <PawelPacana> xorAxAx: i understand
2008-08-09T13:07:47 <xorAxAx> ThomasWaldmann: any operation may only add things in chronological order
2008-08-09T13:07:50 <ThomasWaldmann> (and revisions)
2008-08-09T13:07:52 <xorAxAx> thats not a property of clone
2008-08-09T13:08:01 <PawelPacana> i just tell you the short history of clone
2008-08-09T13:08:03 <xorAxAx> and we are missing indeed unittests
2008-08-09T13:08:07 <xorAxAx> PawelPacana: thank you :)
2008-08-09T13:08:24 <ThomasWaldmann> that's undocumented
2008-08-09T13:08:55 <xorAxAx> and if not too many useless advises were followed yet, the code all lived happily ever after
2008-08-09T13:09:07 <xorAxAx> ThomasWaldmann: yeah, a lot of the spec isnt documented yet :)
2008-08-09T13:09:35 <xorAxAx> and its not even tested
2008-08-09T13:09:42 <ThomasWaldmann> then lets fix the fscking docs instead of having us repeat same discussions over and over
2008-08-09T13:09:51 <xorAxAx> yep, my words :)
2008-08-09T13:09:58 <ThomasWaldmann> dennda: wake up
2008-08-09T13:10:24 <PawelPacana> xorAxAx: if news() will be fixed in correct place, i'll bring the old changeset with news alive
2008-08-09T13:11:01 <xorAxAx> PawelPacana: hmm, you could add an XXX already
2008-08-09T13:11:04 <ThomasWaldmann> PawelPacana: i guess it is just about your pseudo news being moved to storage base
2008-08-09T13:12:10 <PawelPacana> ok
2008-08-09T13:12:19 <xorAxAx> ThomasWaldmann: well, tests for that property are missing
2008-08-09T13:12:24 <PawelPacana> i'll finally take care of it
2008-08-09T13:12:30 <xorAxAx> and clone() doesnt retain creation time of items and rename history
2008-08-09T13:12:34 <PawelPacana> but this could be done much earlier
2008-08-09T13:12:44 <xorAxAx> PawelPacana: yes, its not really your job :)
2008-08-09T13:13:00 <ThomasWaldmann> and btw, for what's already documented in news(): i think it should not be required that it returns ALL revs ever made (the docstring can be read like that).
2008-08-09T13:13:26 <xorAxAx> ThomasWaldmann: it needs to return news() about all revs that are in the backend
2008-08-09T13:13:33 <ThomasWaldmann> news is just a helper to make RecentChanges effective
2008-08-09T13:13:43 <ThomasWaldmann> see the word "Recent" :)
2008-08-09T13:13:58 <xorAxAx> well, i dont see the problem
2008-08-09T13:14:29 <xorAxAx> news() is used for chronological traversal through wiki modificatiions
2008-08-09T13:14:36 <xorAxAx> this use case appears e.g. in RC
2008-08-09T13:14:41 <xorAxAx> but also elsewhere
2008-08-09T13:14:50 <ThomasWaldmann> the problem is not being able to have only the interesting news
2008-08-09T13:14:57 <xorAxAx> and news() doesnt even need to access authorative data ... its denormalised anyway
2008-08-09T13:15:03 <xorAxAx> ThomasWaldmann: i dont understand
2008-08-09T13:15:16 <ThomasWaldmann> and require news data storage to grow forever by spec
2008-08-09T13:15:31 * PawelPacana goes for a breakfast...
2008-08-09T13:15:45 <xorAxAx> ThomasWaldmann: no, its just mapping the data you have in your wiki
2008-08-09T13:15:59 <xorAxAx> so if you say "i put old revs into an archive" then news() will return less as well
2008-08-09T13:16:32 <xorAxAx> i dont see what you mean by interesting ...
2008-08-09T13:18:59 <ThomasWaldmann> xorAxAx: how news is implemented is backend specific
2008-08-09T13:19:09 <ThomasWaldmann> so you can't assume it is just pointers
2008-08-09T13:19:36 <ThomasWaldmann> (it is not for fs17, to have news there you either have to do expensive stuff or to read edit-log)
2008-08-09T13:20:09 <zenhase> ThomasWaldmann: what happens when you start moin.cgi on cmdline?
2008-08-09T13:20:15 <ThomasWaldmann> and there must be away to truncate old history in such backends
2008-08-09T13:20:21 <ThomasWaldmann> zenhase: try it
2008-08-09T13:20:32 <xorAxAx> ThomasWaldmann: news() needs to be a fast operation
2008-08-09T13:20:38 <xorAxAx> ThomasWaldmann: yes, its backend specific
2008-08-09T13:20:55 <zenhase> ThomasWaldmann: it gives me an 500 Error Page
2008-08-09T13:21:06 <ThomasWaldmann> why?
2008-08-09T13:21:07 <xorAxAx> ThomasWaldmann: and yes, you may need to cache things if you assume that news is called more than once. but actually it isnt
2008-08-09T13:21:16 <zenhase> ThomasWaldmann: well, it thinks it is a cgi
2008-08-09T13:21:28 <xorAxAx> ThomasWaldmann: note that fs17 is only used for migration, so we can live with the fact that it would be slow in normal usage
2008-08-09T13:21:29 <zenhase> and it misses wikiconfig and some CGI vars
2008-08-09T13:21:46 <zenhase> it's totally intended
2008-08-09T13:22:15 <zenhase> that's the fallback-mechanism of FCGI
2008-08-09T13:22:33 <ThomasWaldmann> xorAxAx: it is completely insane to require news giving ALL revs. Noone except clone needs that.
2008-08-09T13:23:05 <xorAxAx> ThomasWaldmann: why is that insane?
2008-08-09T13:23:08 <xorAxAx> please elaborate
2008-08-09T13:23:20 <xorAxAx> or give an example where this might lead to issues
2008-08-09T13:24:05 <ThomasWaldmann> because you usually are only interested in the recent changes, not the stuff that happened 10 years ago
2008-08-09T13:24:19 <xorAxAx> you are argueing that one use case doesnt need this feature
2008-08-09T13:24:33 <xorAxAx> but i still dont see any problem with providing this for other use cases
2008-08-09T13:25:01 <ThomasWaldmann> but if news() is required to be able to give ALL, you have to keep a table of those 10 years, and it will ever grow.
2008-08-09T13:25:05 <xorAxAx> and actually its sound - given that clone is a very important operation in the design model, it needs to have fast access to the full history
2008-08-09T13:25:18 <xorAxAx> ThomasWaldmann: it will grow much more slowly than the rest of the wiki
2008-08-09T13:25:42 <xorAxAx> every row in the news data file/table is small compared to the data file and the metadata associated with the new rev
2008-08-09T13:25:43 <ThomasWaldmann> there might be a purge policy
2008-08-09T13:25:57 <xorAxAx> yes, as i said thats totally fine in this model
2008-08-09T13:26:07 <xorAxAx> purge also needs clone, another reason to make news() fast
2008-08-09T13:26:09 <xorAxAx> and complete
2008-08-09T13:27:24 <xorAxAx> also the data store of the news relation is quickly accessible because it is guaranteed to be chronologically created
2008-08-09T13:27:28 <ThomasWaldmann> hmm, ok, I think I got it now:
2008-08-09T13:27:37 <ThomasWaldmann> new
2008-08-09T13:27:51 <xorAxAx> (except in dennda's code, grrrr)
2008-08-09T13:28:21 <ThomasWaldmann> news should be renamed to history() and give all history pointers in reverse order for all revisions we STILL HAVE.
2008-08-09T13:28:44 <xorAxAx> well, the still have part is a bit superfluous, no? :)
2008-08-09T13:28:57 <xorAxAx> imagine archive ops to be a loss of data
2008-08-09T13:29:15 <xorAxAx> so its expected to show less history. also note that you can combine an archivebackend with a working backend
2008-08-09T13:29:20 <ThomasWaldmann> there is a slight difference between what we still have and what we ever had if we purge old stuff.
2008-08-09T13:29:20 <xorAxAx> and still provide full history in RC
2008-08-09T13:29:39 <xorAxAx> imagine a tape library sitting there in case you say "RC, please show the changes in year 1995
2008-08-09T13:29:50 <xorAxAx> isnt that neat? :)
2008-08-09T13:30:24 <xorAxAx> ThomasWaldmann: purging creates a new data store with less information and less history, yes
2008-08-09T13:31:06 <ThomasWaldmann> ok, then we just rename that damn thing
2008-08-09T13:31:14 <ThomasWaldmann> it is not news, it is history.
2008-08-09T13:31:34 <xorAxAx> news is always history when you are noticing it :)
2008-08-09T13:31:39 <ThomasWaldmann> and it is doing more than it was initially designed for.
2008-08-09T13:31:58 <ThomasWaldmann> stuff from years ago is not news
2008-08-09T13:32:13 <xorAxAx> yes, given the new linear news() requirement, the design focused more on a clone based design
2008-08-09T13:32:18 <xorAxAx> it is old news :-)
2008-08-09T13:32:41 <xorAxAx> the idea of news was to signal to it which timestamp you have seen last - like the last newspaper you read
2008-08-09T13:32:48 <xorAxAx> and then it will give you the newer newspapers :)
2008-08-09T13:33:06 <ThomasWaldmann> dennda: i want to see some sane history() function including docs (see above)
2008-08-09T13:33:47 <PawelPacana> history instead of news (rename)? or history and news?
2008-08-09T13:34:10 <ThomasWaldmann> rename according to new requirements
2008-08-09T13:34:53 <ThomasWaldmann> just not fooling people by giving things misleading names
2008-08-09T13:35:11 <PawelPacana> in which package clone should live?
2008-08-09T13:35:47 <xorAxAx> PawelPacana: somewhere below storage probably
2008-08-09T13:35:53 <ThomasWaldmann> the core function could be in storage api
2008-08-09T13:36:07 <xorAxAx> its a generic function that is a level higher than the storage api
2008-08-09T13:37:04 <ThomasWaldmann> it is just copy() for storage
2008-08-09T13:40:23 <ThomasWaldmann> ok, as dennda seems to sleep rather longer, we won't wait for him
2008-08-09T13:45:29 * ThomasWaldmann renames news -> history
2008-08-09T13:48:07 * PawelPacana resurects history-clone
2008-08-09T13:55:05 <ThomasWaldmann> PawelPacana: call history(reverse=False)
2008-08-09T13:59:04 <PawelPacana> great
2008-08-09T14:04:03 <xorAxAx> hmm?
2008-08-09T14:04:16 <xorAxAx> you need to be able to supply a timestamp
2008-08-09T14:13:40 <ThomasWaldmann> PawelPacana: please merge vv
2008-08-09T14:13:58 * ThomasWaldmann slaps CIA-54
2008-08-09T14:15:15 <ThomasWaldmann> is that a new bot API? :)
2008-08-09T14:26:34 <ThomasWaldmann> PawelPacana: you will have to fix the AssertionError in hg backend then
2008-08-09T14:27:08 <ThomasWaldmann> (and if that code is mostly duplicated from fs backend, please de-duplicate)
2008-08-09T14:30:12 <ThomasWaldmann> xorAxAx: some code supports giving a time stamp, some not
2008-08-09T14:30:25 <xorAxAx> in the current api?
2008-08-09T14:30:32 <xorAxAx> yes, its not finished yet
2008-08-09T14:30:43 <ThomasWaldmann> of course also the caller can break out of a loop if it reached some timestamp
2008-08-09T14:31:09 <xorAxAx> ?
2008-08-09T14:31:24 <xorAxAx> you can implement it in O(log n) or better if you have timestamp parameter
2008-08-09T14:31:27 <xorAxAx> instead of O(n)
2008-08-09T14:31:32 <xorAxAx> thats what history is about ...
2008-08-09T14:31:37 <xorAxAx> otherwise we would scan all items
2008-08-09T14:31:56 <ThomasWaldmann> it's an iterator
2008-08-09T14:33:41 <xorAxAx> so?
2008-08-09T14:34:08 <xorAxAx> indeed, it doesnt have the timestamp
2008-08-09T14:34:12 <xorAxAx> in the current api
2008-08-09T14:34:19 <xorAxAx> dennda: why not?
2008-08-09T14:34:38 <ThomasWaldmann> you are talking about the history() base class implementation
2008-08-09T14:34:45 <xorAxAx> yes!
2008-08-09T14:34:55 <xorAxAx> i mean, this is obviously wrong
2008-08-09T14:35:04 <xorAxAx> even RC doesnt always start at now everytime
2008-08-09T14:35:09 <ThomasWaldmann> ok, for that it would be definitely required
2008-08-09T14:35:24 <xorAxAx> hmm
2008-08-09T14:35:38 <xorAxAx> or indeed, in rc you always need now -> some date in the past
2008-08-09T14:35:44 <xorAxAx> so this fits the old concept
2008-08-09T14:36:01 <ThomasWaldmann> it is required to keep the list short and sort less
2008-08-09T14:37:04 <ThomasWaldmann> PawelPacana: you are working on what exactly right now?
2008-08-09T14:40:22 <PawelPacana> im adding interrupted clone support
2008-08-09T14:40:38 <PawelPacana> so when yoy run clone once again
2008-08-09T14:40:51 <PawelPacana> it does not try to recreate existing items
2008-08-09T14:43:24 <xorAxAx> PawelPacana: if news() had a timestamp argument, that would be rather easy
2008-08-09T14:55:57 <ThomasWaldmann> xorAxAx: it's history() now :)
2008-08-09T14:56:09 <xorAxAx> yeah
2008-08-09T15:23:05 * dennda reads the log
2008-08-09T15:26:00 <ThomasWaldmann> moin dennda
2008-08-09T15:26:43 <PawelPacana> ThomasWaldmann: please review last cset
2008-08-09T15:29:04 <PawelPacana> i get en error on history(reverse=False)
2008-08-09T15:31:31 <PawelPacana> on fs backend
2008-08-09T15:32:08 <ThomasWaldmann> PawelPacana: how is that code backend related?
2008-08-09T15:37:31 <PawelPacana> well, it operates on backends, we agreed on something below storage
2008-08-09T15:38:15 <dennda> hi
2008-08-09T15:38:20 <dennda> xorAxAx: still around?
2008-08-09T15:39:18 <ThomasWaldmann> PawelPacana: iirc xorAxAx meant above storage api and i meant in storage api :P
2008-08-09T15:39:52 <dennda> Can somebody explain why clone() must create the items in some order?
2008-08-09T15:40:06 <dennda> PawelPacana: That's what I asked you yesterday
2008-08-09T15:40:37 <dennda> ThomasWaldmann: Why doesn't that work with chinese characters?
2008-08-09T15:40:41 <dennda> It does here
2008-08-09T15:41:48 <xorAxAx> dennda: because we thought that it would be useful if history() gave stuff in chronological order
2008-08-09T15:42:05 <PawelPacana> and i wrote you: because the idea of former news(), now history() is to get timestamp sorted data in optimal way
2008-08-09T15:42:09 <xorAxAx> and if you dont want it to sort, you need to create stuff in chronological order
2008-08-09T15:42:44 <PawelPacana> dennda: see at fs, this stuff is appended to file in chronological order, so its fast
2008-08-09T15:42:49 <dennda> xorAxAx: Why can't it do that by looking at the timestamp as it already does now? (history or news, that is)
2008-08-09T15:42:54 <ThomasWaldmann> dennda: because your console is utf-8?
2008-08-09T15:43:41 <PawelPacana> you can quickly get items from history up to certain ts
2008-08-09T15:43:46 <xorAxAx> dennda: that cant work
2008-08-09T15:43:52 <xorAxAx> dennda: it has to look at the timestamp, yes
2008-08-09T15:43:55 <PawelPacana> just going from top, to some point
2008-08-09T15:48:24 <dennda> I see what you mean, but isn't there a flaw in that plan?
2008-08-09T15:48:36 <xorAxAx> yes, a minor one
2008-08-09T15:48:43 <xorAxAx> but which flaw are you thinking aboutß
2008-08-09T15:48:51 <dennda> That approach assumes that once the revisions are in storage, the order is somewhat magically preserved
2008-08-09T15:49:16 <xorAxAx> yeah, in case of concurrency, the order in the file isnt strictly chronological anymore
2008-08-09T15:49:35 <xorAxAx> except if you lock, read, append, unlock
2008-08-09T15:49:37 <dennda> not only then, imho
2008-08-09T15:49:42 <xorAxAx> but when else?
2008-08-09T15:49:56 <dennda> What part of the backend guarantees the order to be preserved?
2008-08-09T15:50:45 <dennda> So you don't need to order them manually (to sort them, that is) in history()
2008-08-09T15:52:04 <xorAxAx> dennda: well, the caller has to guarantee that. the next question is if the backend needs to assert htat
2008-08-09T15:52:28 <dennda> The caller of history() needs to guarantee what exactly?
2008-08-09T15:52:48 <xorAxAx> dennda: nothing
2008-08-09T15:52:56 <xorAxAx> the caller of the mutating operations
2008-08-09T15:53:11 <dennda> And how would the caller do that?
2008-08-09T15:53:13 <xorAxAx> needs to check that the timestamps are chronological
2008-08-09T15:53:21 <xorAxAx> dennda: thats easy
2008-08-09T15:53:34 <xorAxAx> either he doesnt supply a timestamp at all
2008-08-09T15:53:58 <xorAxAx> or he orders his requests and guarantees it by design (starting from a fresh backend, --> migration/cloning)
2008-08-09T15:54:02 <dennda> with mutating operation you mean something like create_revision()?
2008-08-09T15:54:06 <xorAxAx> yes
2008-08-09T15:54:41 <dennda> How can that be _not_ ordered chronologically _for the item_ if we assume old revisions to be immutable?
2008-08-09T15:55:17 <xorAxAx> huh, you can simply do create_revision and then set a weird timestamp and try to commit
2008-08-09T15:55:19 <dennda> the problem is, that history() needs to yield a list of only revisions ordered by their timestamps
2008-08-09T15:55:29 <xorAxAx> "of only revisions"?
2008-08-09T15:55:31 <dennda> the backend is supposed to set the timestamp
2008-08-09T15:55:35 <xorAxAx> it yields more than revisions in the current spec :)
2008-08-09T15:55:47 <dennda> what else?
2008-08-09T15:56:05 <xorAxAx> item creations and renames
2008-08-09T15:57:00 <dennda> the only usecase where callers need to set the timestamp themselves is in such migration scenarios where the backend can't do that on its own, obviously
2008-08-09T15:58:16 <xorAxAx> yes
2008-08-09T15:58:27 <dennda> in productive use the backend can and should do that on its own
2008-08-09T15:58:35 <xorAxAx> yep
2008-08-09T15:58:49 <dennda> but we are off-track
2008-08-09T15:59:02 <dennda> since this doesn't answer my question :)
2008-08-09T16:01:38 <xorAxAx> which is?
2008-08-09T16:01:52 <dennda> Why must clone() sort the items before adding them to storage
2008-08-09T16:02:05 <dennda> as far as I can tell by now, that is wasted cpu-time
2008-08-09T16:02:14 <xorAxAx> it doesnt need to sort them ...
2008-08-09T16:02:20 <xorAxAx> it calls news() ...
2008-08-09T16:02:37 <dennda> of the old backend?
2008-08-09T16:02:37 <xorAxAx> and news() will retain the order
2008-08-09T16:02:48 <xorAxAx> the new one needs to be empty, yes
2008-08-09T16:04:02 <dennda> ok, doesn't matter much who sorts or orders. the problem I see is, that the order you are trying to preserve is lost as soon as you actually store the revisions in the new backend
2008-08-09T16:04:33 <dennda> at least until you convince me otherwise :)
2008-08-09T16:05:00 <xorAxAx> dennda: no, its not lost, you can set the timestamp to the revs and the items
2008-08-09T16:05:09 <xorAxAx> the latter feature isnt implemented yet
2008-08-09T16:05:56 <dennda> boy that child from the neighborhood does nothing but cry all day
2008-08-09T16:06:56 <dennda> xorAxAx: I don't see how that helps. history() of the new backend must still sort all revisions again
2008-08-09T16:07:00 <PawelPacana> dennda: why is order lost? one could append this order to historyfile
2008-08-09T16:07:28 <PawelPacana> and get order form this file, rather than sorting from backend.iteritems()
2008-08-09T16:07:57 <PawelPacana> as it is actually done in fs, hg
2008-08-09T16:08:01 <xorAxAx> dennda: no, it doesnt
2008-08-09T16:08:31 <dennda> ok it is beginning to make sense
2008-08-09T16:09:32 <dennda> but only if you have such a list keeping that order in *all* backends, not only a subset
2008-08-09T16:09:39 <xorAxAx> ?
2008-08-09T16:09:46 <xorAxAx> every backend provides history()
2008-08-09T16:10:09 <dennda> ok just tell me how it is supposed to work in the memory backend as it is at the moment
2008-08-09T16:10:19 <xorAxAx> hmm?
2008-08-09T16:10:54 <dennda> yes, and if history doesn't sort on its own, it must rely on something else to sort or some list or file that preserved the order
2008-08-09T16:11:21 <xorAxAx> in the memory backend, news() is O(n*r), not very sane :)
2008-08-09T16:11:42 <dennda> propose another way
2008-08-09T16:11:44 <xorAxAx> dennda: news() doesnt need to sort. nobody sorts
2008-08-09T16:11:50 <xorAxAx> the order is retained automatically
2008-08-09T16:11:54 <xorAxAx> dennda: look at fs.py
2008-08-09T16:12:02 <dennda> by what means??
2008-08-09T16:12:29 <xorAxAx> by the invariant that the timestamp of the current operation is larger than the timestamp of the last, completed operation
2008-08-09T16:12:49 <xorAxAx> thats a basic property of a clock :)
2008-08-09T16:13:36 <xorAxAx> we only have a small problem left - clock synchronisation in the process or machine distribution scenario ... (the clock is not really global because the append operation itself is not atomic with the time taking operation)
2008-08-09T16:13:46 <xorAxAx> but we can probably ignore that currently
2008-08-09T16:14:54 <xorAxAx> PawelPacana will explain you the reamining questions :)
2008-08-09T16:15:06 <dennda> Item A and B are created. Revision A_1 is created with timestamp 1, Revision B_1 is created with timestamp 2, Revision A_2 is created with timestamp 3
2008-08-09T16:15:21 <xorAxAx> yes
2008-08-09T16:15:28 <dennda> and the goal:
2008-08-09T16:15:40 <dennda> get all items in this order: [A_1, B_1, A_2]
2008-08-09T16:15:45 <dennda> or reversed
2008-08-09T16:16:04 <dennda> the only question I ever had was: how?
2008-08-09T16:16:33 <PawelPacana> addhistory on commit
2008-08-09T16:17:22 <dennda> yes that explains how the revisions get their timestamp
2008-08-09T16:17:34 <dennda> but not how you get a sorted list of them
2008-08-09T16:18:50 <PawelPacana> on commit(): you assign timestamp and append record to history_file
2008-08-09T16:19:14 <dennda> aha, so you agree that some kind of history file or history list or history foo is needed
2008-08-09T16:19:15 <PawelPacana> assuming commits are atomiv
2008-08-09T16:19:20 <PawelPacana> atomic
2008-08-09T16:19:56 <PawelPacana> thats how order is preserved
2008-08-09T16:20:11 <dennda> so you agree on that? :)
2008-08-09T16:20:24 <PawelPacana> yes
2008-08-09T16:20:53 <PawelPacana> you membackend would basicaly store such list
2008-08-09T16:21:32 <dennda> ok so we basically had the answer 10 minutes ago
2008-08-09T16:21:40 <PawelPacana> maybe not list of revisions
2008-08-09T16:21:57 <PawelPacana> list of (revno, name) is sufficent
2008-08-09T16:22:05 <PawelPacana> to yield proper revision
2008-08-09T16:22:15 <dennda> I blame xorAxAx that it took another 10 minutes, since his comment made me assume that I was wrong :)
2008-08-09T16:22:16 <PawelPacana> and it consumes less memory
2008-08-09T16:22:31 <dennda> repeating myself: < dennda> but only if you have such a list keeping that order in *all* backends, not only a subset
2008-08-09T16:22:53 <dennda> PawelPacana: you are perfectly right
2008-08-09T16:23:13 <dennda> it's just that such a list has never been an official part of the spec
2008-08-09T16:23:31 <dennda> which made me wonder how it's supposed to work without that
2008-08-09T16:23:53 <xorAxAx> dennda: you dont need a list
2008-08-09T16:24:00 <dennda> but..?
2008-08-09T16:24:01 <xorAxAx> an explicit one
2008-08-09T16:24:16 <xorAxAx> you only need to be able to return news() info in O(1)
2008-08-09T16:24:40 <dennda> ok and how? given the current implementation of the memory backend
2008-08-09T16:24:49 <xorAxAx> dennda: in your backend, you need a list
2008-08-09T16:24:56 <xorAxAx> but in other backends, you get it for free
2008-08-09T16:25:03 <dennda> yes
2008-08-09T16:25:09 <dennda> so this should *REALLY* be documented
2008-08-09T16:25:09 <xorAxAx> so you were wrong
2008-08-09T16:25:14 <xorAxAx> yes, of course
2008-08-09T16:25:30 <xorAxAx> history needs to run in O(1)
2008-08-09T16:25:33 <xorAxAx> thats the doc :)
2008-08-09T16:25:46 <xorAxAx> thomas can do amortimized O(1) as well
2008-08-09T16:25:46 <dennda> thats rather a requirement
2008-08-09T16:25:58 <xorAxAx> by caching it - it doesnt change anyway
2008-08-09T16:26:15 <dennda> OK, lemme have a shower first
2008-08-09T16:42:41 <dennda> ThomasWaldmann: In my rewritten clone() method, did you notice the XXX?
2008-08-09T16:44:12 <ThomasWaldmann> will look later
2008-08-09T16:44:55 <waldi> they don't need to. the compatiblity stuff mostly works
2008-08-09T16:44:59 <waldi> ups
2008-08-09T16:45:16 <dennda> ThomasWaldmann: Without much investigation, that imho is a bug in fs17
2008-08-09T16:45:43 <dennda> gnarf, desktop pc somewhat broken... fixing
2008-08-09T17:04:54 <CIA-53> Pawel Pacana <pawel.pacana@gmail.com> default * 4712:450d8eccda14 1.8-mercurialbackend-ppacana/MoinMoin/storage/backends/__init__.py: (clone) Fixed missing import.
2008-08-09T17:22:54 <ThomasWaldmann> re
2008-08-09T17:24:55 <ThomasWaldmann> dennda: file/line?
2008-08-09T17:27:43 <PawelPacana> ThomasWaldmann: http://hg.moinmo.in/moin/1.8-storage-cdenter/file/32042452d216/MoinMoin/script/migration/backend.py
2008-08-09T17:27:56 <PawelPacana> 89
2008-08-09T17:28:23 <PawelPacana> its about list to tuple sequence conversion
2008-08-09T17:28:54 <PawelPacana> new API accepts only tuples (immutable)
2008-08-09T17:29:00 <ThomasWaldmann> ah
2008-08-09T17:33:21 <ThomasWaldmann> well, i think that is just what wikiutil.split_body() gives
2008-08-09T17:34:43 <PawelPacana> as far as i see it's: ['Please edit system and help pages ONLY in the moinmaster wiki! For more', 'information, please see MoinMaster:MoinPagesEditorGroup.', 'master-page:FrontPage', 'master-date:2005-01-23'] stuff
2008-08-09T17:34:54 <ThomasWaldmann> thus, it is rather lack of testing with real data i would say :P
2008-08-09T17:35:18 <ThomasWaldmann> yes, that's the stuff I am pointing dennda at since 2 days :P
2008-08-09T17:36:22 <ThomasWaldmann> dennda: if you are still showering, watch your water bill
2008-08-09T17:38:34 <ThomasWaldmann> + logging.info('.')
2008-08-09T17:38:44 <ThomasWaldmann> i guess this wasn't tested :)
2008-08-09T17:39:21 <PawelPacana> wasn't
2008-08-09T17:39:33 <PawelPacana> i'll change for something more usable ;)
2008-08-09T17:40:17 <ThomasWaldmann> use %r for item.name
2008-08-09T17:40:21 <CIA-53> Pawel Pacana <pawel.pacana@gmail.com> default * 4713:126739efcca3 1.8-mercurialbackend-ppacana/MoinMoin/storage/backends/fs17.py: (old fs) __size fix - bogus fails in clone logger.
2008-08-09T17:42:32 <ThomasWaldmann> ah cool, generic history()
2008-08-09T17:42:55 <ThomasWaldmann> guess it's time for some practical tests
2008-08-09T17:52:10 <ThomasWaldmann> ok, converter running
2008-08-09T18:04:33 <dennda> sorry, good friend of mine arrived. we are fixing my desktop computer and will have lunch afterwards. be back in a few hours... (I am mostly night active these last days anyways)
2008-08-09T18:40:08 <CIA-53> Pawel Pacana <pawel.pacana@gmail.com> default * 4714:739214ff8a5b 1.8-mercurialbackend-ppacana/MoinMoin/ (script/migration/backend.py storage/backends/__init__.py): (migration, clone) Unbuffered py.test-like conversion output, summary, less irritating verbose option.
2008-08-09T18:42:34 <PawelPacana> ^^ is showing bogus fails because of this >> 17:34 lists being converted to tuples and then compared on second migration attempt
2008-08-09T18:44:32 <xorAxAx> PawelPacana: what are the tuples about again?
2008-08-09T18:45:06 <PawelPacana> from the beginning:
2008-08-09T18:45:22 <xorAxAx> :)
2008-08-09T18:45:38 <PawelPacana> lists are not allowed as valid metadata values
2008-08-09T18:45:53 <PawelPacana> fs17 is giving lists as meta
2008-08-09T18:46:03 <xorAxAx> ummmm
2008-08-09T18:46:13 <xorAxAx> then the backend has to fix that, not the migration code
2008-08-09T18:46:14 <PawelPacana> and this lists contain what is written about 17:34
2008-08-09T18:46:17 <PawelPacana> true
2008-08-09T18:46:25 <PawelPacana> see coment in last cset
2008-08-09T18:47:09 <PawelPacana> Thomas identified this dennda problem with wikiutil.split_body
2008-08-09T18:47:24 <xorAxAx> "dennda problem" :)
2008-08-09T18:47:52 <PawelPacana> dennda's i meant :)
2008-08-09T19:46:15 <dennda> heh
2008-08-09T19:50:15 <dennda> Why is that a dennda problem? Because I said it is one?
2008-08-09T19:50:28 <dennda> Imho it's just the backend storing a wrong type
2008-08-09T19:51:29 * dennda tries to figure out what exactly ThomasWaldmann is pointing him at since 2 days
2008-08-09T19:53:14 <dennda> bye cia
2008-08-09T20:17:49 * dennda applies a bunch of patches
2008-08-09T20:42:04 <TheSheep> see ya c-ia
2008-08-09T20:47:12 <dennda> ever considered becoming a professional clown? :D
2008-08-09T20:48:13 <xorAxAx> clowns arent funny
2008-08-09T20:49:03 <dennda> do they scare little xorAxAx? :)
2008-08-09T20:50:01 <xorAxAx> no
2008-08-09T20:51:32 <xorAxAx> clowns arent political enough
2008-08-09T20:55:03 <TheSheep> dennda: I can do more damage to young minds as an admin on the university
2008-08-09T20:55:03 <dennda> PawelPacana: the tests expect that clone() returns the number of converted pages rather than skips, fails
2008-08-09T20:55:20 <dennda> that's your job?
2008-08-09T20:55:53 <TheSheep> dennda: one of them, yes
2008-08-09T20:56:04 <dennda> install a virus :)
2008-08-09T20:56:28 <TheSheep> nah, students do it by themselves just fine
2008-08-09T20:58:29 <PawelPacana> dennda: look what clone returns now
2008-08-09T20:59:02 <PawelPacana> its in the docstring
2008-08-09T21:00:23 <PawelPacana> you can get number of converted pages from that
2008-08-09T21:00:37 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4734:d8abf2847dfb 1.8-storage-cdenter/MoinMoin/ (script/migration/backend.py storage/backends/__init__.py): Merge with 1.8-mercurialbackend-ppacana
2008-08-09T21:11:13 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4735:00a6f98e9421 1.8-storage-cdenter/ (3 files in 2 dirs): storage: applying patch to clone() method and changing flatfile-backends history method
2008-08-09T21:12:16 <dennda> I wonder why johannes has 6 failing tests left and I have 12
2008-08-09T21:13:09 <dennda> reducing it to 10...
2008-08-09T21:14:15 <xorAxAx> dennda: where did you take the "patches" from?
2008-08-09T21:14:25 <dennda> xorAxAx: from johannes
2008-08-09T21:14:47 <xorAxAx> did he send them by mail?
2008-08-09T21:15:08 <xorAxAx> i would have expected some contribution hint in the commit msg
2008-08-09T21:15:24 <dennda> contribution hint?
2008-08-09T21:15:30 <xorAxAx> "thanks johill"
2008-08-09T21:15:37 <xorAxAx> attribution hint i mean
2008-08-09T21:16:07 <dennda> I wrote (Thanks JohannesBerg) countless times, maybe I missed it
2008-08-09T21:16:12 <dennda> ah you mean in this last changeset?
2008-08-09T21:16:36 <dennda> yes, forgot that, sorry
2008-08-09T21:16:56 <dennda> I wish hg did it the git way and distinguished between author and committer
2008-08-09T21:17:28 <xorAxAx> maybe they should use moin's storage api to store metadata :)
2008-08-09T21:17:34 <dennda> http://hg.moinmo.in/moin/1.8-storage-cdenter/shortlog
2008-08-09T21:17:40 <dennda> See how often his names occurrs
2008-08-09T21:17:42 <dennda> :)
2008-08-09T21:18:07 <xorAxAx> yeah, one could be misled and assume that you are his mentor :)
2008-08-09T21:19:19 <dennda> You don't want me to throw the patches away just because I didn't write them myself, do you?
2008-08-09T21:19:26 <xorAxAx> no
2008-08-09T21:19:51 <dennda> 10 failing tests remaining
2008-08-09T21:19:55 <xorAxAx> anything that helps to converge to a common goal is good
2008-08-09T21:19:59 <dennda> at least that's the amount here
2008-08-09T21:20:06 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4736:70f87589ed38 1.8-storage-cdenter/MoinMoin/storage/backends/__init__.py: storage: pep08 fix
2008-08-09T21:23:07 <xorAxAx> hmm, your test set doesnt have syntaxreference
2008-08-09T21:23:14 <xorAxAx> but i think one test needs it
2008-08-09T21:23:28 <xorAxAx> (if it wasnt "fixed" by skipping :))
2008-08-09T21:23:38 <dennda> maybe
2008-08-09T21:23:50 <dennda> there's still something missing. johill has 6 tests failing
2008-08-09T21:24:17 <PawelPacana> dennda: cset 4735, + count[0] += 1
2008-08-09T21:24:20 <PawelPacana> what for?
2008-08-09T21:25:00 <xorAxAx> PawelPacana: please use short hashes to acknowledge your true love wrt dags and the multiplicity of the topological sorts thereof :)
2008-08-09T21:25:35 <PawelPacana> changeset 473500a6f98e9421
2008-08-09T21:25:44 <PawelPacana> changeset 4735 00a6f98e9421
2008-08-09T21:25:47 <PawelPacana> that way
2008-08-09T21:25:56 <xorAxAx> 00a6f98e9421, yeah
2008-08-09T21:27:25 <xorAxAx> i dont see why the s/F branches are useful
2008-08-09T21:27:25 <PawelPacana> dennda: you're interfering with line 87 within this file http://hg.moinmo.in/moin/1.8-storage-cdenter/file/00a6f98e9421/MoinMoin/storage/backends/__init__.py
2008-08-09T21:27:40 <dennda> just noticed
2008-08-09T21:27:40 <xorAxAx> the backend needs to be empty by definition - why would you need to skip things or see failures?
2008-08-09T21:27:41 <dennda> sorry
2008-08-09T21:28:05 <PawelPacana> you may break migration
2008-08-09T21:28:36 <CIA-54> Christopher Denter <moin GUESSWHAT the DASH space DASH station ROUNDTHING com> default * 4737:9c3faec823ae 1.8-storage-cdenter/MoinMoin/storage/backends/__init__.py: storage: remove duplicate line
2008-08-09T21:28:53 <xorAxAx> PawelPacana:
2008-08-09T21:30:18 <PawelPacana> ?
2008-08-09T21:30:43 <xorAxAx> clone() consists nearly mainly of code handling "skip" and "failure" cases. when do these happen?
2008-08-09T21:31:39 <PawelPacana> which gets executed only non-empty destination backend
2008-08-09T21:31:52 <PawelPacana> s/only/only on/
2008-08-09T21:32:46 <xorAxAx> which is not allowed by definition
2008-08-09T21:33:11 <xorAxAx> except if the oldest ts of the src backend is newer than the newest ts of the dst backend
2008-08-09T21:33:15 <xorAxAx> which is pretty seldom
2008-08-09T21:33:33 <xorAxAx> its much more sensible to add a weaving feature to clone
2008-08-09T21:33:44 <PawelPacana> weaving?
2008-08-09T21:33:45 <xorAxAx> that "clones" more than one backend at once
2008-08-09T21:34:04 <xorAxAx> yeah, mutual reads of 2 or more backend history() streams
2008-08-09T21:35:32 <TheSheep> xorAxAx: I think it was added mainly so that you can continue an interrupted migration
2008-08-09T21:35:39 <PawelPacana> xorAxAx: i have more imporant things to do (with monaday deadline)
2008-08-09T21:35:52 <xorAxAx> PawelPacana: i agree
2008-08-09T21:36:00 <PawelPacana> add this to wishlist :)
2008-08-09T21:36:01 <xorAxAx> TheSheep: thats a good point
2008-08-09T21:36:09 <xorAxAx> PawelPacana: well, its violating the spec currently :)
2008-08-09T21:36:18 <PawelPacana> show me the spec
2008-08-09T21:36:19 <PawelPacana> :)
2008-08-09T21:36:23 <xorAxAx> PawelPacana: how? :)
2008-08-09T21:36:38 <xorAxAx> there is currently no sound document that describes it
2008-08-09T21:36:44 <PawelPacana> or rather, show me 'that' written
2008-08-09T21:37:25 <xorAxAx> well, the problem is the chronological order
2008-08-09T21:40:10 <dennda> ok I am off for now, bbl
2008-08-09T21:41:56 <ThomasWaldmann> dennda: MissingPage should be shown for deleted pages
2008-08-09T21:43:14 <PawelPacana> for anything related to migration i'm off too (excluding fixes if problems show up, or one writes unitest instead of elaborating on clone wishlist :))
2008-08-09T21:43:35 <xorAxAx> PawelPacana: hmm, i think interrupted migration can be done more cleanly by reading the last item of news and scrolling the src news feed until a newer one is seen
2008-08-09T21:44:03 <xorAxAx> but that doesnt mean that you should change it :)
2008-08-09T21:45:17 <PawelPacana> assuming nothing changed the state of destination backend beetwen interrupted migrations
2008-08-09T21:46:48 <xorAxAx> PawelPacana: otherwise it would fail because of the clock condition anyway
2008-08-09T21:46:51 * xorAxAx sighs :-(
2008-08-09T21:50:07 <PawelPacana> (current implementation wouldn't)
2008-08-09T21:50:26 <xorAxAx> yes, thats another problem :)
2008-08-09T21:50:41 <PawelPacana> problem?
2008-08-09T21:50:43 <xorAxAx> that the api doesnt assert it -- but its not cheaply possible
2008-08-09T21:51:16 <xorAxAx> so i was wondering whether i should assert it in a unittest
2008-08-09T22:47:49 <ThomasWaldmann> MoinMoin.error.ConfigurationError: OSError: [Errno 2] No such file or directory: 'tests/data'
2008-08-09T22:49:31 <ThomasWaldmann> when doing py.test test_security.py
2008-08-09T23:08:45 <CIA-54> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 3935:799ba4cd0526 1.8/MoinMoin/security/_tests/test_security.py: changed page ACL tests to test with multiple ACL lines
2008-08-09T23:12:54 <CIA-54> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 4739:29d166208fe0 1.8-storage-cdenter/MoinMoin/security/_tests/test_security.py: merged moin/1.8 (test fails, please fix) MoinMoin: MoinMoinChat/Logs/moin-dev/2008-08-09 (last edited 2008-08-08 22:15:02 by IrcLogImporter)
