1 2011-09-17T00:21:07 *** MattMaker
2 2011-09-17T00:30:08 *** raignarok
3 2011-09-17T00:52:59 *** MattMaker
4 2011-09-17T01:41:17 *** asaman
5 2011-09-17T06:32:49 *** MattMaker
6 2011-09-17T07:20:08 *** Marchael
7 2011-09-17T07:55:56 *** Marchael
8 2011-09-17T09:41:36 <ThomasWaldmann> moin
9 2011-09-17T10:31:52 <ronny> moin
10 2011-09-17T10:32:13 <ronny> ThomasWaldmann: any success with tycoon?
11 2011-09-17T10:35:17 <ThomasWaldmann> just read some docs, no new code yet
12 2011-09-17T10:36:51 *** raignarok
13 2011-09-17T10:39:36 <ThomasWaldmann> ronny: can you check the stuff there (testing stuff), is it all done or something needed to be kept there still? http://etherpad.osuosl.org/moin2-storage
14 2011-09-17T10:40:22 <ronny> ThomasWaldmann: its done
15 2011-09-17T10:41:00 <ThomasWaldmann> ok
16 2011-09-17T10:41:26 <ThomasWaldmann> is it unclear when to use open/close?
17 2011-09-17T10:41:50 <ThomasWaldmann> https://bitbucket.org/thomaswaldmann/storage-ng/issues?status=new&status=open
18 2011-09-17T10:42:19 <ThomasWaldmann> i mean, "use open before using, close it after using" is a common concept
19 2011-09-17T10:42:47 <ThomasWaldmann> what we make out of the is another problem (how long we keep it open)
20 2011-09-17T10:42:53 <ThomasWaldmann> that*
21 2011-09-17T10:43:49 <ronny> ThomasWaldmann: the scpe of how long one uses us unclear
22 2011-09-17T10:44:10 <ronny> ThomasWaldmann: do backends keep open the whole runtime, or do we repopen per request
23 2011-09-17T10:44:26 <ThomasWaldmann> is that any different from "file"
24 2011-09-17T10:44:33 <ThomasWaldmann> you open a file, use it, close it
25 2011-09-17T10:45:00 <ThomasWaldmann> how long you can keep it open is not a problem of file, but of the code layer using it
26 2011-09-17T10:45:58 <ronny> ThomasWaldmann: the question is what is the scope of the ressource usage
27 2011-09-17T10:46:56 <ronny> i.e. per request or per process
28 2011-09-17T10:48:40 <ronny> i think for the backend per process makes sense
29 2011-09-17T10:51:45 <ronny> ThomasWaldmann: btw, merge my fork?
30 2011-09-17T10:51:49 <ThomasWaldmann> backend is related to 1 wiki currently
31 2011-09-17T10:54:59 <ThomasWaldmann> hmm, strictly taken, there is no requirement that 1 backend is storing 1 wiki only
32 2011-09-17T10:55:33 <ThomasWaldmann> as long as one want search results showing up in other wikis also
33 2011-09-17T10:57:15 <ThomasWaldmann> i have some kt stuff with len and iter failing
34 2011-09-17T10:58:01 <ThomasWaldmann> how do i make it not test that because it is currently unsupported?
35 2011-09-17T11:04:43 *** MattMaker
36 2011-09-17T11:14:54 * ThomasWaldmann tries the rpc interface for iter
37 2011-09-17T11:21:58 *** raignarok
38 2011-09-17T11:28:39 *** raignarok
39 2011-09-17T11:46:57 <ronny> ThomasWaldmann: those arent skippable nicely yet
40 2011-09-17T12:09:24 *** Marchael
41 2011-09-17T12:15:09 * ThomasWaldmann implements len and iter
42 2011-09-17T12:31:39 * ThomasWaldmann committed, pulled, merged, pushed
43 2011-09-17T12:39:04 <ThomasWaldmann> ronny: ^^
44 2011-09-17T12:39:06 <ThomasWaldmann> " #XXX log info?
45 2011-09-17T12:39:26 <ronny> ThomasWaldmann: i wasnt sure what to do the case where we dont do anything
46 2011-09-17T12:39:28 <ThomasWaldmann> do you mean for the else branch? (comment from router.py)
47 2011-09-17T12:40:00 <ronny> yes
48 2011-09-17T12:40:16 <ThomasWaldmann> well, if the stuff is read-only, we can't do anything. but if it is read-only, it is either configured that way or not writable due to fundamental reasons.
49 2011-09-17T12:40:42 <ronny> true
50 2011-09-17T12:41:36 <ThomasWaldmann> you need to add your name to router.py
51 2011-09-17T12:43:30 <ronny> ThomasWaldmann: pushed
52 2011-09-17T12:44:20 *** raignarok
53 2011-09-17T12:44:43 <ThomasWaldmann> https://bitbucket.org/thomaswaldmann/storage-ng/issue/1/backend-uses-term-revision i think we should fix that
54 2011-09-17T12:45:21 <ThomasWaldmann> i think that is a valid point. as we don't have parent code in storage, we don't have revisions
55 2011-09-17T12:45:59 <ThomasWaldmann> we just deal with bunches of meta and data, but they are not related to each other except by sharing some commen metadata values
56 2011-09-17T12:46:05 <ThomasWaldmann> common*
57 2011-09-17T12:46:50 <ThomasWaldmann> so, revision are a concept on a higher level, right?
58 2011-09-17T12:49:27 <ronny> ThomasWaldmann: pretty much
59 2011-09-17T12:49:34 <ronny> backends only get data and metadata
60 2011-09-17T12:49:48 <ronny> the terms store/retrive/discard seem reasonable
61 2011-09-17T12:49:55 <ronny> maybe we can lean a bit more on sets
62 2011-09-17T12:50:36 <ronny> hmm
63 2011-09-17T12:51:57 <ThomasWaldmann> we could also use dict methods :)
64 2011-09-17T12:52:49 <ThomasWaldmann> btw, a quick thing first: serializer.deserialize -> use sourcefile as 1st arg instead of io?
65 2011-09-17T12:53:07 <ThomasWaldmann> (see targetfile in serialize)
66 2011-09-17T12:53:50 <ronny> ThomasWaldmann: i think src/dst may be better
67 2011-09-17T12:54:03 <ThomasWaldmann> or that, ok, just something consistent
68 2011-09-17T12:59:09 <ronny> ThomasWaldmann: for backend im going to use store, read and remove as terms, ok?
69 2011-09-17T13:00:15 <ronny> ThomasWaldmann: btw, we should have some way to fail tests if the backend cant provide
70 2011-09-17T13:00:26 <ronny> (eh skip i mean)
71 2011-09-17T13:00:33 <ThomasWaldmann> i am still in favour of dict method names :)
72 2011-09-17T13:00:53 <ThomasWaldmann> all we need is some class for (meta, data)
73 2011-09-17T13:01:53 *** raignarok
74 2011-09-17T13:02:32 <ThomasWaldmann> or just define that values are (meta, data) tuples
75 2011-09-17T13:07:57 <ThomasWaldmann> ronny: ^^
76 2011-09-17T13:09:19 <ThomasWaldmann> brunomartin: still busy?
77 2011-09-17T13:10:51 <ronny> ThomasWaldmann: http://paste.pocoo.org/show/477360/
78 2011-09-17T13:11:23 <ronny> ThomasWaldmann: i dont consider backend a subset of mapping its semilar, but different
79 2011-09-17T13:11:44 <ThomasWaldmann> read sounds file-like
80 2011-09-17T13:12:02 *** MattMaker
81 2011-09-17T13:12:06 <ThomasWaldmann> what exactly is different?
82 2011-09-17T13:12:54 <ronny> meta and data dont have a strict 1:1 relation
83 2011-09-17T13:13:08 <ThomasWaldmann> just imagine (meta, data) as one thing, call it dm
84 2011-09-17T13:13:12 <ThomasWaldmann> or md
85 2011-09-17T13:13:59 <ThomasWaldmann> and that we may optimize data storage by reducing redundancy isn't visible to upper layers
86 2011-09-17T13:14:26 <ronny> it would be nice to see it in serialize for example
87 2011-09-17T13:15:02 <ThomasWaldmann> btw, serializer is no middleware
88 2011-09-17T13:15:17 <ronny> i wasnt sure where to put it
89 2011-09-17T13:15:24 <ThomasWaldmann> we could make it a mixin maybe
90 2011-09-17T13:15:39 <ThomasWaldmann> so we call it as backend.serialize(targetfile)
91 2011-09-17T13:15:39 <ronny> why make it a mixing?
92 2011-09-17T13:15:54 <ronny> its completely distinct right now
93 2011-09-17T13:15:58 <ronny> which is a nice property
94 2011-09-17T13:16:16 <ThomasWaldmann> just an idea, we can also keep it separate
95 2011-09-17T13:17:12 <ThomasWaldmann> backend.__init__?
96 2011-09-17T13:17:32 <ThomasWaldmann> i mean backend/__init__.py
97 2011-09-17T13:17:47 <ronny> i think it deserves a own file
98 2011-09-17T13:18:31 <ThomasWaldmann> in any case it is rather related to backend than to middleware
99 2011-09-17T13:18:48 <ThomasWaldmann> otoh, router is also
100 2011-09-17T13:18:53 <ThomasWaldmann> %)
101 2011-09-17T13:19:14 <ronny> can we consider router as multiplexing backend?
102 2011-09-17T13:19:40 <ronny> it also only changes meta[NAME]
103 2011-09-17T13:19:47 <ronny> so its pretty nice for various things
104 2011-09-17T13:20:45 <ronny> ThomasWaldmann: ok, i would prefer to keep the distinction between mapping and backend for now, the api is smaller
105 2011-09-17T13:22:41 <ThomasWaldmann> ok, but don't call it read(), that's too file-like
106 2011-09-17T13:22:52 <ronny> hmm, retrieve?
107 2011-09-17T13:23:30 <ronny> get will probably work, too
108 2011-09-17T13:24:54 <ThomasWaldmann> get/set/(remove|delete) is used by stores
109 2011-09-17T13:27:01 <ronny> ThomasWaldmann: stores are mappings
110 2011-09-17T13:32:41 <ronny> ThomasWaldmann: i wonder how we should go about the difference between dataid in normal backends and in fileserver
111 2011-09-17T13:37:38 <ThomasWaldmann> well, they are different, but as long as iter yield something get can process, we don't have a problem, do we?
112 2011-09-17T13:38:40 *** raignarok
113 2011-09-17T13:43:00 <ronny> ThomasWaldmann: i suppose
114 2011-09-17T13:46:52 <ThomasWaldmann> store/restrieve/remove sound consistent
115 2011-09-17T13:46:55 <ThomasWaldmann> -s
116 2011-09-17T14:04:18 <ronny> ok, finished renaming
117 2011-09-17T14:04:25 <ronny> and pushed
118 2011-09-17T14:05:27 <ronny> bbl
119 2011-09-17T14:08:11 *** greg_f
120 2011-09-17T14:21:32 <ThomasWaldmann> waldi: https://bitbucket.org/thomaswaldmann/storage-ng/issue/2/open-close-usage-unclear set it to invalid? need more docstrings?
121 2011-09-17T14:29:33 <ronny> ThomasWaldmann: we forgot something critical - handling of closing data
122 2011-09-17T14:38:28 <ThomasWaldmann> i added some comments a while ago. maybe we could think about whether context managers make sense.
123 2011-09-17T14:39:16 <ThomasWaldmann> if a caller gives an open file to a callee, caller is reponsible for closing
124 2011-09-17T14:39:43 <ThomasWaldmann> if a caller calls a callee and gets an open file as a return value from callee, caller is reponsible for closing.
125 2011-09-17T14:40:16 <ronny> ThomasWaldmann: true
126 2011-09-17T14:40:18 * ThomasWaldmann currently adds missing tests for indexer
127 2011-09-17T14:40:34 <ronny> ThomasWaldmann: we might want a contextmanager for revision objects
128 2011-09-17T14:46:40 <ronny> ThomasWaldmann: btw, is fs the only store that has files?
129 2011-09-17T14:46:48 <ronny> (as in a real file)
130 2011-09-17T14:46:59 <ThomasWaldmann> there is a fs storage
131 2011-09-17T14:47:20 <ThomasWaldmann> but i had open files in mind
132 2011-09-17T14:47:29 <ThomasWaldmann> so talk before fixing :)
133 2011-09-17T14:49:22 <ronny> ThomasWaldmann: do we want a backend type that stores files over a certain size in a different store?
134 2011-09-17T14:49:53 <ronny> (requires to pass it in, but we usually get it from uploads/mimetypes)
135 2011-09-17T14:50:29 <ronny> (i think its neat to have text items in sqlite, and stuff like image uploads in the fs)
136 2011-09-17T14:52:06 <ronny> ThomasWaldmann: aslo we might want to have some strategy for deferin opening backend files
137 2011-09-17T14:52:12 <ronny> we need o make some notes i think
138 2011-09-17T14:52:28 <ronny> i have to run a erand, bbl
139 2011-09-17T14:52:53 *** Marchael
140 2011-09-17T16:02:05 *** Marchael
141 2011-09-17T16:14:46 *** MattMaker
142 2011-09-17T16:47:25 * ThomasWaldmann filed a whoosh bug
143 2011-09-17T17:06:28 <ronny> re
144 2011-09-17T17:08:18 <ThomasWaldmann> wb ronny
145 2011-09-17T17:08:47 <ThomasWaldmann> that special case of destroying a revision that is in latest revs index now also works correctly
146 2011-09-17T17:11:51 <ronny> ThomasWaldmann: it would be neat if whoosh managed to have the winners of a certain search in a own index on its own
147 2011-09-17T17:12:33 <ThomasWaldmann> heh
148 2011-09-17T17:17:24 <ThomasWaldmann> hmm, are we running out of todo before implanting storage-ng? :)
149 2011-09-17T17:19:29 <ThomasWaldmann> ronny: what do you think about giving clear_revision new metadata dict (as in create_revision), so it overwrites some of the old values with the new ones?
150 2011-09-17T17:19:45 <ThomasWaldmann> would replace that partial solution for COMMENT/reason
151 2011-09-17T17:22:05 <ronny> ThomasWaldmann: kwargs might make sense for that
152 2011-09-17T17:22:59 <ThomasWaldmann> but giving meta is more consistent with the other code
153 2011-09-17T17:23:18 <ThomasWaldmann> that meta would even include the REVID
154 2011-09-17T17:31:37 <ThomasWaldmann> hmm, in fact i think we are move flexible with a overwrite_revision()
155 2011-09-17T17:47:06 <ronny> ThomasWaldmann: overwrite_revision(revid, **kw)?
156 2011-09-17T17:49:27 <ronny> hmm
157 2011-09-17T17:49:34 <ThomasWaldmann> wait a sec
158 2011-09-17T17:50:09 *** Marchael
159 2011-09-17T17:50:25 *** Marchael
160 2011-09-17T17:50:51 <ronny> ThomasWaldmann: actually
161 2011-09-17T17:51:07 <ronny> ThomasWaldmann: its only a api on item, in the backend, all we do is a store
162 2011-09-17T17:51:21 <ronny> ThomasWaldmann: do we want to be able to store meta items with no data attached?
163 2011-09-17T17:51:44 <ronny> (that would be neat for alias pages and things like that)
164 2011-09-17T17:52:45 <ThomasWaldmann> http://paste.pocoo.org/show/477456/ somehow like that
165 2011-09-17T17:52:55 <ThomasWaldmann> scroll to store_revision
166 2011-09-17T17:54:22 <ThomasWaldmann> could also work without the flag, just with the acl
167 2011-09-17T17:57:05 <ronny> ThomasWaldmann: seems kind of reasonable
168 2011-09-17T17:57:20 <ronny> ThomasWaldmann: the acl stuff needs some more changes i suppose?
169 2011-09-17T17:57:32 <ronny> ThomasWaldmann: i'll have to do some other stuff for a while, bbl
170 2011-09-17T17:58:54 <ThomasWaldmann> yes, overwrite would be new
171 2011-09-17T18:02:21 *** raignarok
172 2011-09-17T18:02:28 <ronny> ThomasWaldmann: i ave a feeling that it'll be a mess to integrate acl directly into middleware
173 2011-09-17T18:02:38 <ronny> would it make sense to layer it on top?
174 2011-09-17T18:03:21 <ronny> (so we can construct a AclMiddleware(inexer, current_user)
175 2011-09-17T18:12:44 <ThomasWaldmann> i thought about that, but most acl code will be in a separate module
176 2011-09-17T18:12:59 <ThomasWaldmann> also, we can get the current revision acl from index easily
177 2011-09-17T18:13:30 <ThomasWaldmann> a bit more complicated will be the hierarchical acl support though, that needs multiple index accesses
178 2011-09-17T18:14:10 <ThomasWaldmann> ok, the store_revision stuff works now with tests
179 2011-09-17T18:19:38 * ThomasWaldmann pushed
180 2011-09-17T18:19:56 <ThomasWaldmann> coders should be payed by negative LOC
181 2011-09-17T18:21:29 <ThomasWaldmann> we can directly use username = flaskg.user.name later btw, so maybe we don't need a layer for this
182 2011-09-17T18:23:24 <ronny> ThomasWaldmann: plugins might want to behave as system user
183 2011-09-17T18:26:19 *** Marchael
184 2011-09-17T18:31:00 <ThomasWaldmann> hm, yes
185 2011-09-17T18:36:08 <ronny> ThomasWaldmann: i think acl can be a 1:1 set of decorators for middleware, items and revisions
186 2011-09-17T18:38:01 <ThomasWaldmann> like?
187 2011-09-17T18:42:01 *** raignarok
188 2011-09-17T18:46:06 *** greg_f
189 2011-09-17T18:54:27 <ronny> ThomasWaldmann: basicall, it would just wrap around the unrestricted api as decorator, and do the checks
190 2011-09-17T19:00:07 *** raignarok
191 2011-09-17T19:05:07 <ThomasWaldmann> show some code
192 2011-09-17T19:06:36 <ThomasWaldmann> i have rather bad experiences with decorators. they are pretty, but sometimes also pretty useless/unusable for the desired scenario.
193 2011-09-17T19:09:17 <ronny> ThomasWaldmann: cant do today
194 2011-09-17T19:12:15 <ThomasWaldmann> ronny: i just found search/... are a bit unefficient
195 2011-09-17T19:12:50 <ThomasWaldmann> the search the index, get result docs from that search, then use self[doc[NAME]]
196 2011-09-17T19:13:15 <ThomasWaldmann> and the latter does another index lookup just to find the same doc again
197 2011-09-17T19:13:41 <ronny> oO
198 2011-09-17T19:14:23 <ThomasWaldmann> (that's the case for searching in current revs)
199 2011-09-17T19:14:34 <ThomasWaldmann> for past revs, it might be even worse
200 2011-09-17T19:14:42 <ronny> ThomasWaldmann: item construction should be able to also take a revid
201 2011-09-17T19:14:59 <ThomasWaldmann> or a result doc? :)
202 2011-09-17T19:15:32 <ronny> hmm, basically whatever works, just choose a reasonable api coupling
203 2011-09-17T19:15:42 <ThomasWaldmann> otoh, for acls, we need the current doc, not a past one
204 2011-09-17T19:16:08 <ThomasWaldmann> i'll try to refactor that...
205 2011-09-17T19:23:41 *** Marchael
206 2011-09-17T19:38:11 *** Marchael
207 2011-09-17T20:02:21 <ThomasWaldmann> ronny: http://paste.pocoo.org/show/477521/ do you see why this makes one test fail?
208 2011-09-17T20:04:01 <ThomasWaldmann> or anyone else?
209 2011-09-17T20:16:53 <ronny> re
210 2011-09-17T20:20:31 <ronny> ThomasWaldmann: seems like Item._current can end up as None
211 2011-09-17T20:20:48 <ronny> eh no
212 2011-09-17T20:21:01 <ronny> ThomasWaldmann: the CONTENT pop on _current looks problematic
213 2011-09-17T20:23:23 <ronny> ThomasWaldmann: http://paste.pocoo.org/show/477538/ seems to fix
214 2011-09-17T20:23:31 <ronny> (moved the pop to only if gotten from index
215 2011-09-17T20:25:07 <ThomasWaldmann> yeah, just seen it also
216 2011-09-17T20:25:47 <ronny> ThomasWaldmann: i think that mappig might be problematic
217 2011-09-17T20:25:57 <ronny> its mutated and reused in various places
218 2011-09-17T20:26:16 <ronny> so its hard to reason about
219 2011-09-17T20:27:05 <ThomasWaldmann> did you change anything else than the pop?
220 2011-09-17T20:29:42 <ronny> ThomasWaldmann: no, which is why im suspicious
221 2011-09-17T20:30:01 <ThomasWaldmann> ok, that fixes it
222 2011-09-17T20:31:44 <ronny> ThomasWaldmann: i think it'll cause issues later
223 2011-09-17T20:31:59 <ThomasWaldmann> i think I'll solve slightly different
224 2011-09-17T20:32:41 <ThomasWaldmann> the Revision._current shall never have content, because that's not needed there
225 2011-09-17T20:33:29 <ThomasWaldmann> hmm, no
226 2011-09-17T20:33:37 <ThomasWaldmann> i guess i better do not pop
227 2011-09-17T20:33:52 <ThomasWaldmann> we won't create a ton of Revision objects
228 2011-09-17T20:34:56 <ThomasWaldmann> ehrm, Items
229 2011-09-17T20:35:58 <ThomasWaldmann> btw, that is not the final state, I'll remove the name fixation now and use **kw
230 2011-09-17T20:36:53 <dreimark> re
231 2011-09-17T20:36:54 <ThomasWaldmann> because using name is wrong
232 2011-09-17T20:41:21 <ronny> ThomasWaldmann: what do we do wrt content indexing on __large__ items?
233 2011-09-17T20:41:59 <ThomasWaldmann> what exactly is the problem?
234 2011-09-17T20:42:44 <ronny> ThomasWaldmann: pull a 4gb iso image into the stored content, watich it all burn?
235 2011-09-17T20:42:47 <ronny> *watch
236 2011-09-17T20:43:23 <ThomasWaldmann> point to the problematic place :)
237 2011-09-17T20:51:28 <ronny> ThomasWaldmann: see convert_to_indexable
238 2011-09-17T20:51:54 <ronny> it has a todo
239 2011-09-17T20:52:06 <ronny> but i think we should tak care for some checks
240 2011-09-17T20:54:24 <ThomasWaldmann> let me just finish the current stuff :)
241 2011-09-17T21:06:07 * ThomasWaldmann pushed
242 2011-09-17T21:06:48 <ThomasWaldmann> ronny: we don't need to fix that convert_to_indexable, we need to use a real one later anyway
243 2011-09-17T21:07:20 <ThomasWaldmann> after implanting into moin2
244 2011-09-17T21:08:08 <ThomasWaldmann> def __getitem__(self, item_name):
245 2011-09-17T21:08:13 <ThomasWaldmann> def create_item(self, item_name):
246 2011-09-17T21:08:18 <ThomasWaldmann> def existing_item(self, item_name):
247 2011-09-17T21:08:33 <ThomasWaldmann> those could also get **query rather - except the first one...
248 2011-09-17T21:08:48 <ThomasWaldmann> so one can give name=u"foo" or itemid="...."
249 2011-09-17T21:09:12 <ThomasWaldmann> or whatever other unique field=value for latest-revs index
250 2011-09-17T21:12:20 *** raignarok
251 2011-09-17T21:12:42 <ThomasWaldmann> or email=foo@example.org for user items :)
252 2011-09-17T21:16:14 * ThomasWaldmann likes the new api :)
253 2011-09-17T21:25:23 *** raignarok
254 2011-09-17T21:28:46 <ThomasWaldmann> http://paste.pocoo.org/show/477586/ < how about that?
255 2011-09-17T21:46:31 <ronny> ThomasWaldmann: looks reasonable at first glance
256 2011-09-17T21:48:31 <ThomasWaldmann> ok, done
257 2011-09-17T21:59:08 *** eSyr
258 2011-09-17T22:07:34 *** eSyr
259 2011-09-17T23:14:45 *** raignarok
260
