MoinMoin: MoinMoinChat/Logs/moin-dev/2010-01-23

   1 2010-01-23T00:21:13  *** grzywacz has quit IRC
   2 2010-01-23T01:39:24  *** TheSheep has quit IRC
   3 2010-01-23T01:39:29  *** TheSheep has joined #moin-dev
   4 2010-01-23T04:24:54  *** ChanServ sets mode: +o ThomasWaldmann
   5 2010-01-23T09:05:49  <dreimark> xorAxAx: i hope can you review
   6 2010-01-23T09:14:15  * dreimark btw. i didn't like the code as i did it in the past and may rewrite it at asap 
   7 2010-01-23T09:15:46  <dreimark> s/didn/don/
   8 2010-01-23T11:51:22  <ThomasWaldmann> moin
   9 2010-01-23T11:52:11  <TheSheep> moin
  10 2010-01-23T11:52:20  <TheSheep> ThomasWaldmann: you have a monkey at your back
  11 2010-01-23T11:53:05  <ThomasWaldmann> yes, it eats javascript bots
  12 2010-01-23T11:53:16  <TheSheep> yum yum
  13 2010-01-23T11:56:18  <ThomasWaldmann> TheSheep: btw, last night i played a bit with sqlalchemy (sql/non-orm mode) and sqlite
  14 2010-01-23T11:57:09  <ThomasWaldmann> did you ever see a speed difference with indexed columns? somehow it always took 0.2s for me, with or without index.
  15 2010-01-23T11:57:51  *** ChanServ sets mode: -o ThomasWaldmann
  16 2010-01-23T12:08:34  <TheSheep> ThomasWaldmann: some indexes are created automatically
  17 2010-01-23T12:08:43  <TheSheep> ThomasWaldmann: the id is always indexed
  18 2010-01-23T12:09:00  <TheSheep> s/indexes/indices
  19 2010-01-23T12:11:42  <ThomasWaldmann> i know, i didn't use the primary key for testing
  20 2010-01-23T12:22:02  <TheSheep> it laso shows on very large tables
  21 2010-01-23T12:22:05  <TheSheep> also
  22 2010-01-23T12:22:22  <TheSheep> and complicated queriesd
  23 2010-01-23T12:22:35  <TheSheep> you made those indices on single columns?
  24 2010-01-23T12:28:45  <ThomasWaldmann> yes
  25 2010-01-23T12:29:33  <ThomasWaldmann> selected a dozen entries from 50.000, didn't show a speed differenz
  26 2010-01-23T12:29:41  <ThomasWaldmann> ..ce
  27 2010-01-23T12:30:16  <ThomasWaldmann> # itemid, revno, metaid
  28 2010-01-23T12:30:16  <ThomasWaldmann> # metaid, metaname, metavalue
  29 2010-01-23T12:31:13  <ThomasWaldmann> columns of 2 tables to represent moin (revision) metadata - is that as good as it gets with sql?
  30 2010-01-23T12:33:03  <xorAxAx> well, its obviously not normalized, but you cannot do better
  31 2010-01-23T12:33:29  <ThomasWaldmann> (itemid is a uuid for an item, metaid is just some unique key)
  32 2010-01-23T12:34:00  <ThomasWaldmann> why is it not normalized?
  33 2010-01-23T12:35:00  <ThomasWaldmann> itemid+revno is primary key btw
  34 2010-01-23T12:35:48  <xorAxAx> because you could put the metanames into columns
  35 2010-01-23T12:36:12  <xorAxAx> what are the queries that you are trying?
  36 2010-01-23T12:36:18  <ThomasWaldmann> problematic with variable metanames
  37 2010-01-23T12:39:42  <ThomasWaldmann> i am just thinking about structure yet, but queries obviously should be able to select items / revs via their metadata
  38 2010-01-23T12:40:02  <ThomasWaldmann> e.g. find the itemid for some email
  39 2010-01-23T12:40:09  <ThomasWaldmann> (user item)
  40 2010-01-23T12:42:23  <ThomasWaldmann> is creating tables that are very long, but not wide, a problem?
  41 2010-01-23T12:43:48  <ThomasWaldmann> the first one is itemcount*revperitemcount*metaentriescount
  42 2010-01-23T13:09:52  <xorAxAx> can you show your queries?
  43 2010-01-23T14:15:55  *** grzywacz has joined #moin-dev
  44 2010-01-23T15:24:16  *** kursor has joined #moin-dev
  45 2010-01-23T15:51:38  <ThomasWaldmann> xorAxAx: there are no queries yet
  46 2010-01-23T15:55:46  <xorAxAx> then how benchmarked you the stuff?
  47 2010-01-23T15:57:06  <ThomasWaldmann> i only wrote some minimal code, like from tutorial.
  48 2010-01-23T19:11:15  *** kursor has quit IRC
  49 2010-01-23T19:37:17  * ThomasWaldmann tries to replace cdb in fs2 by sqlalchemy/sqlite
  50 2010-01-23T19:37:43  <dennda> is that a good idea?
  51 2010-01-23T19:37:43  <dreimark> :)
  52 2010-01-23T19:38:13  <dreimark> is that then a fs2_sqla so we can select?
  53 2010-01-23T19:39:10  <dennda> ThomasWaldmann: afair sqlite locks down as soon as some operation is running or something
  54 2010-01-23T19:39:16  <dennda> and cdb should scale rather well
  55 2010-01-23T19:39:23  <dennda> but it's quite some time ago since i looked into it
  56 2010-01-23T19:39:26  <dennda> ask an expert instead ;)
  57 2010-01-23T19:39:42  <ThomasWaldmann> what means "locks down"?
  58 2010-01-23T19:42:20  <dennda> dunno but quite some people said that sqlalchemy doesn't scale that well
  59 2010-01-23T19:42:28  <dennda> but oh well, you could still easily move to another db then
  60 2010-01-23T19:42:39  <dennda> but seriously, ask someone who knows what he's talking about
  61 2010-01-23T19:42:40  <dennda> :)
  62 2010-01-23T19:43:27  <ThomasWaldmann> well, the reasoning behind that was kind of:
  63 2010-01-23T19:43:59  <ThomasWaldmann> cdb is nice for r/o, but bad to update. also, it is additional code only used in fs backend
  64 2010-01-23T19:44:18  <ThomasWaldmann> sqlite is part of python meanwhile, so let's rather use sqlite
  65 2010-01-23T19:44:52  <ThomasWaldmann> sqlite needs fs access, bad for e.g. GAE, so rather use sqla with sqlite, so one can switch
  66 2010-01-23T19:45:10  <ThomasWaldmann> sqla is needed anyway for sqla backend, so no additional code
  67 2010-01-23T19:45:56  <dennda> so what is the difference between fs2 with sqla as opposed to the sqla backend?
  68 2010-01-23T19:46:12  <ThomasWaldmann> (and the bigger plan was to create an index for all metadata)
  69 2010-01-23T19:46:32  <ThomasWaldmann> authoritative data is stored in filesystem, only index is done with db
  70 2010-01-23T19:47:00  <dennda> authoritative?
  71 2010-01-23T19:47:17  <ThomasWaldmann> you can delete and rebuild the index
  72 2010-01-23T19:48:08  <ThomasWaldmann> from auth. data stored in fs (for the fs* backends) or somewhere else (for other backends)
  73 2010-01-23T21:15:03  *** franklin_ has joined #moin-dev
  74 2010-01-23T21:31:32  *** franklin has quit IRC
  75 2010-01-23T21:58:24  <dreimark> hmm that should have become WikiSandBox, now it is a bad comment
  76 2010-01-23T22:01:03  <dreimark> revert redone
  77 2010-01-23T22:13:40  <CIA-30> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 7178:f741f716008f 2.0-storage/MoinMoin/storage/backends/fs2.py: fs2 backend: use sqlalchemy/sqlite instead of cdb
  78 2010-01-23T22:13:43  <CIA-30> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 7179:59a824ed4163 2.0-storage/MoinMoin/ (4 files in 3 dirs): merge main
  79 2010-01-23T23:02:21  <CIA-30> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 7180:a64e8274d407 2.0-storage/MoinMoin/storage/backends/fs2.py: fs2 backend: replace self-made history file by sqla table
  80 2010-01-23T23:58:04  *** grzywacz has quit IRC
  81 2010-01-23T23:58:41  <CIA-30> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 7181:e3701fb63e8b 2.0-storage/MoinMoin/storage/backends/fs2.py:
  82 2010-01-23T23:58:44  <CIA-30> fs2 backend: misc. cleanups, see below
  83 2010-01-23T23:58:46  <CIA-30> cleanup some docstrings
  84 2010-01-23T23:58:48  <CIA-30> remove unused imports, parameters
  85 2010-01-23T23:58:50  <CIA-30> use index_history.db because it contains index and history
  86 2010-01-23T23:58:52  <CIA-30> create temp files in a directory near the final destination, use *.tmp
  87 2010-01-23T23:58:54  <CIA-30> for meta revisions, use *.rev