1 2011-04-16T00:05:01 *** MattMaker
2 2011-04-16T02:29:37 <CIA-25> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 178:2f1708707657 2.0/quickinstall: quickinstall now uses curl or wget (fix for no wget on Mac OS X, thanks to Graham Dumpleton)
3 2011-04-16T02:29:40 <CIA-25> Thomas Waldmann <tw AT waldmann-edv DOT de> default * 179:939013ce69fe 2.0/quickinstall: quickinstall: simplify
4 2011-04-16T03:15:33 *** cosmo_
5 2011-04-16T03:17:09 *** cosmo
6 2011-04-16T05:28:48 *** MattMaker
7 2011-04-16T05:29:19 *** z^ck
8 2011-04-16T05:29:30 *** z^ck
9 2011-04-16T06:36:21 *** MattMaker
10 2011-04-16T07:08:03 <dreimark> moin
11 2011-04-16T08:13:28 *** Marchael
12 2011-04-16T08:13:36 <Marchael> moin
13 2011-04-16T11:16:22 *** Marchael
14 2011-04-16T11:32:55 *** Marchael
15 2011-04-16T11:46:10 *** ronny
16 2011-04-16T12:52:48 <ThomasWaldmann> moin
17 2011-04-16T13:03:01 *** greg_f
18 2011-04-16T13:34:55 *** m4k3r
19 2011-04-16T13:47:48 *** ronny
20 2011-04-16T14:01:38 *** ronny
21 2011-04-16T14:13:32 *** ronny
22 2011-04-16T14:39:00 *** m4k3r
23 2011-04-16T14:40:05 *** m4k3r
24 2011-04-16T14:41:20 *** m4k3r
25 2011-04-16T14:49:12 * dreimark greetings from cologne
26 2011-04-16T14:49:41 <ThomasWaldmann> ah, right. have fun!
27 2011-04-16T14:50:03 <ThomasWaldmann> if something moin2ish comes up, point people to this channel.
28 2011-04-16T14:50:17 <dreimark> i may be have a talk at 16:45
29 2011-04-16T14:50:27 <dreimark> and try to run the client too
30 2011-04-16T14:50:41 <ThomasWaldmann> you don't know yet? :)
31 2011-04-16T14:50:50 <dreimark> currently we are on track
32 2011-04-16T14:50:57 <dreimark> may be a bit later
33 2011-04-16T15:23:52 *** andialbrecht_
34 2011-04-16T15:51:11 <ThomasWaldmann> Marchael: btw, what qp class would you use for the search box input field processing?
35 2011-04-16T15:53:54 <Marchael> ThomasWaldmann: What does it mean "qp class"?
36 2011-04-16T15:54:21 <ThomasWaldmann> whoosh offers misc. queryparser classes
37 2011-04-16T15:54:32 <Marchael> aah
38 2011-04-16T15:56:20 <Marchael> If I understand right you mean something like this http://packages.python.org/Whoosh/api/qparser.html#parser-object ?
39 2011-04-16T15:58:18 <ThomasWaldmann> yes
40 2011-04-16T16:01:34 <Marchael> So I will use it of corse, but I`m worried about parsing queries like "mimetype:text/html rev_no:1 search query"
41 2011-04-16T16:02:51 <Marchael> by default I will find all documents which names matches given query
42 2011-04-16T16:04:30 <ThomasWaldmann> btw, i asked for the class name.
43 2011-04-16T16:08:14 * Marchael reading description of all classes
44 2011-04-16T16:10:58 <Marchael> whoosh.qparser.QueryParser
45 2011-04-16T16:12:02 <Marchael> I`m also reading information about plugins for qp http://packages.python.org/Whoosh/api/qparser.html#plug-ins
46 2011-04-16T16:18:32 *** MattMaker
47 2011-04-16T16:19:10 <Marchael> Althought, for searching in multiple fields I`m need whoosh.qparser.MultifieldParser
48 2011-04-16T16:35:20 <Marchael> ThomasWaldmann: Am I answered to your question?
49 2011-04-16T16:38:48 <Marchael> I also think what it will be good idea to add FieldAliasPlugin. For example "mimetype" == "mime" or "revision" == "rev"
50 2011-04-16T16:47:36 <Marchael> xorAxAx: hi
51 2011-04-16T16:47:43 <Marchael> My apologies what I couldn't answer to your comment on melange soon. Last 3 days I actively working on my coursework, so I had no time to think about answer to your comment.
52 2011-04-16T16:47:48 <xorAxAx> Marchael: hi
53 2011-04-16T16:47:56 <Marchael> > How do you think you want to implement the search language? Will you reuse the parser of Whoosh? Will you use a parsing toolkit like they did?
54 2011-04-16T16:47:56 <Marchael> Search language will be look like http://moinmo.in/HelpOnSearching. I don't want to write my own search language, because whoosh already has whoosh.qparser.QueryParser for this.
55 2011-04-16T16:47:59 <xorAxAx> thats ok
56 2011-04-16T16:48:39 <xorAxAx> but woosh doesnt have this language ... do youi want to convert the queries, Marchael?
57 2011-04-16T16:48:52 <xorAxAx> and how do you want to parse the language explained on that wiki page?
58 2011-04-16T16:55:01 <xorAxAx> Marchael: ping
59 2011-04-16T16:55:22 <Marchael> whoosh have many different plugins for parsing search quieries http://packages.python.org/Whoosh/api/qparser.html#plug-ins. Whoosh have similar query language.
60 2011-04-16T16:55:22 <Marchael> For example if you want to use phrase searching then whoosh will be use whoosh.qparser.PhrasePlugin.
61 2011-04-16T16:55:22 <Marchael> If you want to all documents starting with "help" but didn`t want to find "helponsearch" expression will be look like "NO helponsearch AND help*". For this whoosh will using whoosh.qparser.WildcardPlugin
62 2011-04-16T16:56:10 <xorAxAx> so what?
63 2011-04-16T16:56:16 <xorAxAx> please answer my questions
64 2011-04-16T16:56:36 <Marchael> etc
65 2011-04-16T16:57:55 <Marchael> I do not want to convert quieries if whoosh can understand it without convertion
66 2011-04-16T16:59:13 <Marchael> if you type "query string" on search box whoosh will recognize this as phrase without convertion
67 2011-04-16T17:02:45 <xorAxAx> but the whole language is NOT understood by whoosh, so what is your plan?
68 2011-04-16T17:06:53 <Marchael> xorAxAx: you meant language describing on HelpOnSearch?
69 2011-04-16T17:11:46 <xorAxAx> yes
70 2011-04-16T17:23:57 <Marchael> Whoosh can understand queries like "content:help". So I`m can design schema with this fields(exclude case and regexp of course)
71 2011-04-16T17:24:09 <Marchael> sorry for long answers
72 2011-04-16T17:26:50 <xorAxAx> yeah, you will have cases which whoosh cannot understand. how do you want to handle them?
73 2011-04-16T17:30:42 <ThomasWaldmann> xorAxAx: btw, i think moin2 is rather different from moin1 anyway, so it is not strictly required that search queries are 100% compatible. of course if it can be made easily to understand the same queries, one would do that.
74 2011-04-16T17:31:11 <xorAxAx> ThomasWaldmann: yes, its no requirement, indeed, but he wanted to achieve that goal, so he should explain how
75 2011-04-16T17:31:20 *** franklin
76 2011-04-16T17:32:03 <Marchael> xorAxAx: can you provide some example of query which whoosh can not understand?
77 2011-04-16T17:32:43 <xorAxAx> Marchael: you just gave an example
78 2011-04-16T17:35:27 <ThomasWaldmann> Marchael: yes, MultifieldParser looks good for a combined title search and text search box
79 2011-04-16T17:37:17 <Marchael> xorAxAx: I do not find how whoosh can perform case sensitive searching and searching by regexp(althought '*' and '?' will be enough I guess)
80 2011-04-16T17:37:26 <Marchael> s/find/found/
81 2011-04-16T17:38:03 <xorAxAx> Marchael: then you need to extend whoosh
82 2011-04-16T17:39:48 <Marchael> althought whoosh support some regexp which contains <, >, >=, <=, =>, or =<
83 2011-04-16T17:40:39 <ThomasWaldmann> Marchael: in case you mean comparisons / range stuff: that is not regexp
84 2011-04-16T17:41:27 <ThomasWaldmann> please read some stuff about regular expressions so you'll know better what is one and what is not
85 2011-04-16T17:42:44 <Marchael> ThomasWaldmann: I`m just copypaste this from there http://packages.python.org/Whoosh/api/qparser.html#whoosh.qparser.GtLtPlugin
86 2011-04-16T17:49:26 <ThomasWaldmann> they use a regex there for the plugin, but that doesn't make regexes part of the query language
87 2011-04-16T18:05:49 <Marchael> ThomasWaldmann: I think it will be good idea to write feature request to whoosh ml because I don't know how much time takes writing regexp and case sensitive search support
88 2011-04-16T18:07:43 <xorAxAx> Marchael: i think that it rather makes sense to implement regex on the moin side of things
89 2011-04-16T18:08:13 <xorAxAx> same for case sensitive things
90 2011-04-16T18:10:55 <xorAxAx> e.g. case sensitive search is application of a whoosh search operator and a moin search operator that does a plain fulltext search
91 2011-04-16T18:11:50 <Marchael> xorAxAx: and what if we have something like this '^[^o0-9]*' ?
92 2011-04-16T18:12:19 <xorAxAx> then you do a plain fulltext search with that regex
93 2011-04-16T18:14:46 <ThomasWaldmann> what makes more sense maybe depends what whoosh author wants to have in whoosh and what not
94 2011-04-16T18:23:09 <Marchael> for case sensitive search I can use following algoritm:
95 2011-04-16T18:23:09 <Marchael> 1) Search as usual text
96 2011-04-16T18:23:09 <Marchael> 2) get list with search results
97 2011-04-16T18:23:09 <Marchael> 3) Use re for each element in list
98 2011-04-16T18:26:15 <xorAxAx> yes, for example
99 2011-04-16T18:26:33 <ThomasWaldmann> yes, that'ld work. requires the stuff for re :)
100 2011-04-16T18:30:28 <Marchael> I'm need to go back home, get back soon :)
101 2011-04-16T18:30:33 *** Marchael
102 2011-04-16T19:12:44 *** Marchael
103 2011-04-16T19:14:22 <Marchael> xorAxAx: Am I answered to your question?
104 2011-04-16T19:16:09 <xorAxAx> no
105 2011-04-16T19:16:25 <Marchael> xorAxAx: What is unclear for you/
106 2011-04-16T19:16:28 <Marchael> ?
107 2011-04-16T19:16:43 <xorAxAx> you still need to extend whoosh's query syntax to accomodate regex and case searches, you didnt tell us how you plan to do it
108 2011-04-16T19:18:22 <Marchael> I already wrote about case sensitive
109 2011-04-16T19:19:01 <xorAxAx> you only described the algorithm, not the query language
110 2011-04-16T19:21:14 <Marchael> queries will be look like in moin1(case:word)
111 2011-04-16T19:21:27 <xorAxAx> and how do you parse them?
112 2011-04-16T19:22:37 <Marchael> using re for analyzing request coming from search box
113 2011-04-16T19:28:29 <Marchael> For finding regexp in request I also will be using re and if it found regexp then re will be using this regex on search results
114 2011-04-16T19:30:17 <Marchael> s/re will be using/re will use/
115 2011-04-16T19:33:31 <xorAxAx> why not a parser toolkit?
116 2011-04-16T19:33:44 <xorAxAx> easier, easily extensible
117 2011-04-16T19:35:50 <Marchael> xorAxAx: qparser module?
118 2011-04-16T19:36:19 <xorAxAx> hmm?
119 2011-04-16T19:36:37 <Marchael> http://packages.python.org/Whoosh/api/qparser.html#module-whoosh.qparser
120 2011-04-16T19:36:37 <xorAxAx> a parsing toolkit takes a grammar e.g. in BNF form and generates a parser from it
121 2011-04-16T19:36:48 <xorAxAx> whoosh uses pyparsing
122 2011-04-16T19:36:58 <Marchael> aah
123 2011-04-16T19:40:15 * Marchael reading about BNF
124 2011-04-16T19:42:32 <ThomasWaldmann> we maybe really first look at how whoosh parses queries before putting a whole new thing on top of this
125 2011-04-16T19:43:19 <xorAxAx> ThomasWaldmann: yes, hte best idea is to extend the pyparsing desc of it
126 2011-04-16T19:43:32 <xorAxAx> Marchael: read the pyparsing docs
127 2011-04-16T19:43:41 <Marchael> ok
128 2011-04-16T19:43:47 <xorAxAx> we learn about BNF in first semester in germany
129 2011-04-16T19:44:20 <Marchael> second semestr of 3th curse in Russia :)
130 2011-04-16T19:44:55 <xorAxAx> "3th curse"?
131 2011-04-16T19:45:08 <Marchael> *course
132 2011-04-16T19:45:11 <xorAxAx> 3rd course? what is a semester of a course?
133 2011-04-16T19:47:21 <Marchael> second
134 2011-04-16T19:50:33 <ThomasWaldmann> xorAxAx: why are you talking about whoosh using pyparsing?
135 2011-04-16T19:52:44 <ThomasWaldmann> and are we both talking about whoosh 1.8.2?
136 2011-04-16T19:54:40 *** greg_f
137 2011-04-16T19:57:18 <ThomasWaldmann> xorAxAx: looks like you are looking at whoosh < 1.0
138 2011-04-16T19:57:50 <ThomasWaldmann> "New hand-written query parser based on plug-ins. Less brittle, more robust, more flexible, and easier to fix/improve than the old pyparsing-based parser."
139 2011-04-16T19:58:05 <ThomasWaldmann> from 1.0 changelog entry
140 2011-04-16T19:58:33 <ThomasWaldmann> looks like he wasn't that happy with pyparsing
141 2011-04-16T19:59:49 <xorAxAx> ThomasWaldmann: i was looking at the website
142 2011-04-16T20:00:16 <ThomasWaldmann> don't use whoosh.ca
143 2011-04-16T20:00:41 <Marchael> http://packages.python.org/Whoosh/index.html
144 2011-04-16T20:00:44 <ThomasWaldmann> bitbucket and pypi are much more current
145 2011-04-16T20:04:34 <ThomasWaldmann> Marchael: xorAxAx: (or anybody else): if you find spelling errors or other minor thing in the whoosh docs, please tell me
146 2011-04-16T20:05:10 <Marchael> ok
147 2011-04-16T20:05:30 * ThomasWaldmann wants to feed back such stuff upstream
148 2011-04-16T20:05:31 <xorAxAx> ThomasWaldmann: i looked at bitbucket
149 2011-04-16T20:13:13 <ThomasWaldmann> xorAxAx: i fixed the wiki page
150 2011-04-16T20:13:32 <xorAxAx> ok
151 2011-04-16T20:28:53 <Marchael> xorAxAx: for parsing queries I can use something like this MultifieldParser(["title", "regexp",...], ix.schema).parse(u'title:none regexp:^[^\!]*')
152 2011-04-16T20:28:53 <Marchael> this will return parsed request
153 2011-04-16T20:30:21 <Marchael> looking like And([Term(u'title', u'shit'), Term(u'regexp', u'^[^\!]*')])
154 2011-04-16T20:30:43 <Marchael> ooops
155 2011-04-16T20:35:38 <Marchael> I meant u'title', u'none' of corse :)
156 2011-04-16T20:37:28 <Marchael> *of course
157 2011-04-16T20:41:24 *** Marchael
158 2011-04-16T20:41:31 *** Marchael
159 2011-04-16T21:01:31 *** m4k3r
160 2011-04-16T21:04:36 <Marchael> xorAxAx: so what you think?
161 2011-04-16T21:20:55 <ThomasWaldmann> btw, I'ld suggest reading once through the whoosh docs and also through the whoosh src code
162 2011-04-16T21:41:59 *** Marchael
163 2011-04-16T21:50:15 <ThomasWaldmann> xorAxAx: still there?
164 2011-04-16T22:29:37 <ThomasWaldmann> xorAxAx: waldi: i am currently thinking about whether it makes sense to run revision data through dom conversion (data->dom, dom->simple text) before giving it to the search indexing analyser (tokenizer, filters)
165 2011-04-16T22:32:28 *** humbu
166 2011-04-16T22:32:35 *** humbu
167 2011-04-16T22:39:37 <waldi> ThomasWaldmann: well. you don't want to feed html data into it usually
168 2011-04-16T22:40:29 <ThomasWaldmann> i didn't say html
169 2011-04-16T22:41:03 <ThomasWaldmann> the point is that if we directly try to index revision data, we have 2 issues:
170 2011-04-16T22:41:34 <ThomasWaldmann> a) different kind of text items (wiki markup, rst, maybe even html, docbook/xml, ...)
171 2011-04-16T22:41:57 <ThomasWaldmann> b) non-text data (word docs, pdfs, images, ...)
172 2011-04-16T22:42:15 <waldi> you said "data"
173 2011-04-16T22:42:26 <ThomasWaldmann> for a) it would be nice to get out text only information
174 2011-04-16T22:42:34 <ThomasWaldmann> (no markup)
175 2011-04-16T22:42:58 <ThomasWaldmann> for b) we need text extraction anyways (see MoinMoin.filter in 1.9)
176 2011-04-16T22:58:08 <dreimark> re
177 2011-04-16T22:58:10 <dreimark> Rietveld Code Review Tool
178 2011-04-16T22:58:25 <dreimark> is an interesting tools to get clean commits
179 2011-04-16T22:58:45 <dreimark> and also less changesets
180 2011-04-16T22:59:07 <dreimark> at least less fix a fix changesets
181 2011-04-16T23:07:59 <ThomasWaldmann> the docs tell it is for use with subversion
182 2011-04-16T23:16:28 <ThomasWaldmann> ah, docs outdated, seems to support mercurial also
183 2011-04-16T23:19:24 <dreimark> it supports mercurial and it does not need only app engine
184 2011-04-16T23:20:35 <dreimark> some images from today http://www.flickr.com/photos/onyame/
185 2011-04-16T23:22:47 <dreimark> http://openetherpad.org/coderview
186 2011-04-16T23:24:51 <dreimark> at the dinner andi told that there is also a version available which independent from app engine
187 2011-04-16T23:25:03 <dreimark> need to ask him tom. about the url
188 2011-04-16T23:41:57 *** MattMaker
189
