Searching text is very important search, and a major way of navigating a wiki. It is also a slow search as we don't use index search currently
This test the very small wiki with only the system and help pages and about 100 new pages. In typical wiki the results will be much slower.
All tests run a search for "help config" without the quotes, which mean the page text is searched twice, once for help and once for config.
patch 467
ab -n 11 -c 1 'http://localhost:8000/HelpOnConfiguration/EmailSupport?action=fullsearch&context=180&value=help+config&fullsearch=Text'
Requests per second: 0.59 [#/sec] (mean)
729017 function calls (682597 primitive calls) in 31.802 CPU seconds
Ordered by: internal time, call count
List reduced from 250 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
53120/22810 9.334 0.000 16.588 0.001 Page.py:78(get_rev)
10547 5.877 0.001 12.555 0.001 search.py:222(search)
59770 2.515 0.000 2.515 0.000 posixpath.py:171(exists)
32240 1.855 0.000 1.855 0.000 wikiutil.py:154(quoteWikinameFS)
32240/16130 1.764 0.000 12.245 0.001 Page.py:120(getPagePath)
177520 1.269 0.000 1.269 0.000 posixpath.py:56(join)
12367 1.268 0.000 9.316 0.001 Page.py:488(get_raw_body)
6580 1.105 0.000 7.069 0.001 security.py:52(<lambda>)
6580 1.058 0.000 1.058 0.000 security.py:45(__getattr__)
10 1.005 0.100 1.130 0.113 search.py:577(sortByWeight)
7370 0.602 0.000 0.602 0.000 codecs.py:232(read)
6540 0.345 0.000 0.418 0.000 wikiutil.py:187(unquoteWikiname)
6580 0.284 0.000 0.446 0.000 wikiacl.py:185(may)
7370 0.283 0.000 0.404 0.000 codecs.py:531(open)
3640 0.238 0.000 0.300 0.000 search.py:421(_unique_matches)
7370 0.224 0.000 0.224 0.000 Page.py:1382(decodeTextMimeType)
6460 0.210 0.000 0.225 0.000 search.py:121(sortByCost)
20 0.206 0.010 11.535 0.577 Page.py:1316(listPages)
15970 0.154 0.000 0.154 0.000 Page.py:28(__init__)
6580 0.126 0.000 5.456 0.001 Page.py:1266(getACL)
Ordered by: internal time, call count
List reduced from 250 to 20 due to restriction <20>
Function was called by...
Page.py:78(get_rev) Page.py:120(getPagePath)(30350) 12.245
Page.py:199(_text_filename)(13980) 11.877
Page.py:332(exists)(2060) 2.066
Page.py:1316(listPages)(6730) 11.535
search.py:222(search) search.py:126(search)(10547) 12.847
posixpath.py:171(exists) Page.py:78(get_rev)(53120) 16.588
Page.py:120(getPagePath)(20) 12.245
Page.py:332(exists)(50) 2.066
Page.py:1266(getACL)(6580) 5.456
wikiutil.py:154(quoteWikinameFS) Page.py:120(getPagePath)(32240) 12.245
Page.py:120(getPagePath) AttachFile.py:50(getAttachDir)(10) 0.005
Page.py:78(get_rev)(16040) 16.588
Page.py:120(getPagePath)(16110) 12.245
Page.py:228(_last_edited)(10) 0.005
Page.py:332(exists)(50) 2.066
Page.py:427(getPageList)(10) 11.538
editlog.py:74(__init__)(10) 0.001
posixpath.py:56(join) Page.py:78(get_rev)(106240) 16.588
Page.py:120(getPagePath)(64480) 12.245
Page.py:427(getPageList)(10) 11.538
Page.py:1316(listPages)(6730) 11.535
caching.py:13(__init__)(10) 0.001
caching.py:34(_filename)(10) 0.000
wikiaction.py:836(getPlugins)(40) 0.006
Page.py:488(get_raw_body) Page.py:1183(getPageHeader)(910) 0.032
search.py:222(search)(10547) 12.555
search.py:702(formatContext)(910) 3.151
security.py:52(<lambda>) Page.py:1316(listPages)(6530) 11.535
__init__.py:488(shouldShowPageinfo)(10) 0.014
__init__.py:750(shouldShowEditbar)(20) 0.017
__init__.py:910(editbar)(10) 0.062
request.py:366(getAvailableActions)(10) 0.017
security.py:45(__getattr__) Page.py:1316(listPages)(6530) 11.535
__init__.py:488(shouldShowPageinfo)(10) 0.014
__init__.py:750(shouldShowEditbar)(20) 0.017
__init__.py:910(editbar)(10) 0.062
request.py:366(getAvailableActions)(10) 0.017
search.py:577(sortByWeight) fullsearch.py:7(execute)(10) 31.749
codecs.py:232(read) codecs.py:378(read)(7370) 0.630
wikiutil.py:187(unquoteWikiname) Page.py:1316(listPages)(6540) 11.535
wikiacl.py:185(may) security.py:52(<lambda>)(6580) 7.069
codecs.py:531(open) Page.py:488(get_raw_body)(7370) 9.316
search.py:421(_unique_matches) search.py:389(get_matches)(3640) 0.319
Page.py:1382(decodeTextMimeType) Page.py:488(get_raw_body)(7370) 9.316
search.py:121(sortByCost) search.py:126(search)(6460) 12.847
Page.py:1316(listPages) Page.py:427(getPageList)(20) 11.538
Page.py:28(__init__) AttachFile.py:50(getAttachDir)(10) 0.005
Page.py:1252(getParentPage)(20) 0.010
request.py:670(run)(10) 31.802
search.py:702(formatContext)(910) 3.151
search.py:945(searchPages)(6460) 24.534
security.py:52(<lambda>)(6580) 7.069
text_html.py:212(pagelink)(1820) 2.326
wikiutil.py:432(getSysPage)(150) 0.078
wikiutil.py:909(send_title)(10) 0.230
Page.py:1266(getACL) security.py:52(<lambda>)(6580) 7.069
patch 460 + experinamtal refactoring
Note about caching pages:
- this code cache pages in request. This cache include the page text, if its read from disk. So here after a match is found, we don't have to open the page and read the text again, and convert from crlf to lf, just to get the context of each match. Search results are done in memory. Running 100 search raise the memory footprint to 65MB, and it seems that python does not want to return this memory back. This could be optimized to delete from memory the text of a page that does not match, or to use another limit on the page cache.
ab -n 11 -c 1 'http://localhost:8000/HelpOnConfiguration/EmailSupport?action=fullsearch&context=180&value=help+config&fullsearch=Text'
Requests per second: 1.13 [#/sec] (mean)
466592 function calls in 16.296 CPU seconds
Ordered by: internal time, call count
List reduced from 285 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
9194 2.605 0.000 6.074 0.001 search.py:222(search)
6460 2.455 0.000 2.455 0.000 codecs.py:232(read)
18970 1.684 0.000 1.684 0.000 posixpath.py:171(exists)
20 1.202 0.060 7.704 0.385 Page.py:1337(listPages)
6650 1.096 0.000 1.239 0.000 Page.py:1287(getACL)
10 1.050 0.105 1.057 0.106 wikidicts.py:300(scandicts)
1830 1.020 0.001 1.020 0.001 warnings.py:59(warn_explicit)
12360 0.787 0.000 3.047 0.000 Page.py:185(getInfoForDomain)
6730 0.357 0.000 0.424 0.000 wikiutil.py:187(unquoteWikiname)
6460 0.323 0.000 0.407 0.000 codecs.py:531(open)
6600 0.304 0.000 0.304 0.000 wikiutil.py:154(quoteWikinameFS)
6650 0.278 0.000 1.438 0.000 wikiacl.py:185(may)
25730 0.230 0.000 0.230 0.000 posixpath.py:56(join)
3640 0.218 0.000 0.283 0.000 search.py:421(_unique_matches)
6460 0.190 0.000 0.203 0.000 search.py:121(sortByCost)
6460 0.166 0.000 3.217 0.000 Page.py:141(readTextFile)
6460 0.150 0.000 0.150 0.000 Page.py:1401(decodeTextMimeType)
6670 0.133 0.000 0.133 0.000 posixpath.py:144(getmtime)
11014 0.122 0.000 3.366 0.000 Page.py:496(get_raw_body)
21990 0.107 0.000 3.154 0.000 Page.py:169(getInfo)
Ordered by: internal time, call count
List reduced from 285 to 20 due to restriction <20>
Function was called by...
search.py:222(search) search.py:126(search)(9194) 6.332
codecs.py:232(read) codecs.py:378(read)(6460) 2.476
posixpath.py:171(exists) Page.py:185(getInfoForDomain)(18960) 3.047
Page.py:242(_last_edited)(10) 0.001
Page.py:1337(listPages) Page.py:435(getPageList)(20) 7.707
Page.py:1287(getACL) security.py:52(<lambda>)(6650) 2.752
wikidicts.py:300(scandicts) request.py:555(initdicts)(10) 1.058
warnings.py:59(warn_explicit) warnings.py:24(warn)(1830) 1.057
Page.py:185(getInfoForDomain) Page.py:169(getInfo)(12360) 3.154
wikiutil.py:187(unquoteWikiname) Page.py:1337(listPages)(6730) 7.704
codecs.py:531(open) Page.py:141(readTextFile)(6460) 3.217
wikiutil.py:154(quoteWikinameFS) Page.py:105(getStorageName)(6600) 0.347
wikiacl.py:185(may) security.py:52(<lambda>)(6650) 2.752
posixpath.py:56(join) Page.py:185(getInfoForDomain)(25590) 3.047
Page.py:435(getPageList)(10) 7.707
Page.py:530(getEditLogPath)(20) 0.000
Page.py:536(getAttachmentPath)(10) 0.000
Page.py:542(getPagesPath)(10) 0.000
caching.py:13(__init__)(10) 0.001
caching.py:34(_filename)(10) 0.000
multiconfig.py:267(__init__)(30) 0.004
wikiaction.py:836(getPlugins)(40) 0.006
search.py:421(_unique_matches) search.py:389(get_matches)(3640) 0.297
search.py:121(sortByCost) search.py:126(search)(6460) 6.332
Page.py:141(readTextFile) Page.py:496(get_raw_body)(6460) 3.366
Page.py:1401(decodeTextMimeType) Page.py:141(readTextFile)(6460) 3.217
posixpath.py:144(getmtime) Page.py:289(lastEditInfo)(10) 0.002
Page.py:1287(getACL)(6650) 1.239
logfile.py:160(date)(10) 0.000
Page.py:496(get_raw_body) Page.py:1210(getPageHeader)(910) 0.026
search.py:222(search)(9194) 6.074
search.py:702(formatContext)(910) 0.323
Page.py:169(getInfo) Page.py:217(_text_filename)(10) 0.000
Page.py:316(isWritable)(10) 0.000
Page.py:329(isUnderlayPage)(10) 0.000
Page.py:337(isStandardPage)(10) 0.000
Page.py:345(exists)(8800) 3.178
Page.py:496(get_raw_body)(6460) 3.366
Page.py:530(getEditLogPath)(20) 0.000
Page.py:536(getAttachmentPath)(10) 0.000
Page.py:542(getPagesPath)(10) 0.000
Page.py:1287(getACL)(6650) 1.239
patch-468
ab -n 11 -c 1 'http://localhost:8000/HelpOnConfiguration/EmailSupport?action=fullsearch&context=180&value=help+config&fullsearch=Text'
Requests per second: 0.62 [#/sec] (mean)
777957 function calls (747857 primitive calls) in 31.430 CPU seconds
Ordered by: internal time, call count
List reduced from 255 to 20 due to restriction <20>
ncalls tottime percall cumtime percall filename:lineno(function)
36850 7.099 0.000 8.296 0.000 Page.py:89(get_current_from_pagedir)
10547 4.848 0.000 10.557 0.001 search.py:222(search)
43500 2.290 0.000 2.290 0.000 posixpath.py:171(exists)
128460 1.912 0.000 1.912 0.000 posixpath.py:56(join)
36850 1.622 0.000 4.724 0.000 Page.py:105(get_rev_dir)
15970 1.307 0.000 2.657 0.000 Page.py:62(reset)
3640 1.239 0.000 1.304 0.000 search.py:421(_unique_matches)
7370 1.089 0.000 1.121 0.000 codecs.py:360(__init__)
6460 1.072 0.000 11.848 0.002 search.py:126(search)
2070 1.067 0.001 1.067 0.001 wikiutil.py:75(quoteWikinameURL)
5280 1.012 0.000 1.012 0.000 wikidicts.py:145(has_member)
15970 0.711 0.000 0.711 0.000 wikiutil.py:154(quoteWikinameFS)
7370 0.620 0.000 0.620 0.000 codecs.py:232(read)
46160/16060 0.597 0.000 12.456 0.001 Page.py:122(get_rev)
6540 0.366 0.000 0.440 0.000 wikiutil.py:187(unquoteWikiname)
7370 0.310 0.000 1.431 0.000 codecs.py:531(open)
6580 0.284 0.000 1.443 0.000 wikiacl.py:185(may)
32000 0.282 0.000 12.698 0.000 Page.py:241(getPageStatus)
12367 0.259 0.000 6.243 0.001 Page.py:590(get_raw_body)
32000 0.237 0.000 12.257 0.000 Page.py:176(getPageBasePath)
Ordered by: internal time, call count
List reduced from 255 to 20 due to restriction <20>
Function was called by...
Page.py:89(get_current_from_pagedir) Page.py:105(get_rev_dir)(6730) 4.724
Page.py:122(get_rev)(30120) 12.456
search.py:222(search) search.py:126(search)(10547) 11.848
posixpath.py:171(exists) Page.py:105(get_rev_dir)(36850) 4.724
Page.py:241(getPageStatus)(20) 12.698
Page.py:429(exists)(50) 2.754
Page.py:1368(getACL)(6580) 6.531
posixpath.py:56(join) Page.py:62(reset)(15970) 2.657
Page.py:89(get_current_from_pagedir)(36850) 8.296
Page.py:105(get_rev_dir)(36850) 4.724
Page.py:241(getPageStatus)(32000) 12.698
Page.py:1418(listPages)(6730) 11.462
caching.py:13(__init__)(10) 0.001
caching.py:34(_filename)(10) 0.000
wikiaction.py:836(getPlugins)(40) 0.006
Page.py:105(get_rev_dir) Page.py:122(get_rev)(30120) 12.456
Page.py:1418(listPages)(6730) 11.462
Page.py:62(reset) Page.py:26(__init__)(15970) 2.848
search.py:421(_unique_matches) search.py:389(get_matches)(3640) 1.322
codecs.py:360(__init__) codecs.py:531(open)(7370) 1.431
search.py:126(search) search.py:945(searchPages)(6460) 24.218
wikiutil.py:75(quoteWikinameURL) Page.py:664(link_to)(1890) 3.936
__init__.py:131(logo)(10) 0.005
__init__.py:910(editbar)(10) 0.046
wikiutil.py:909(send_title)(160) 0.166
wikidicts.py:145(has_member) wikidicts.py:271(has_member)(5280) 1.078
wikiutil.py:154(quoteWikinameFS) Page.py:62(reset)(15970) 2.657
codecs.py:232(read) codecs.py:378(read)(7370) 0.649
Page.py:122(get_rev) Page.py:176(getPageBasePath)(30120) 12.257
Page.py:296(_text_filename)(13980) 9.825
Page.py:429(exists)(2060) 2.754
wikiutil.py:187(unquoteWikiname) Page.py:1418(listPages)(6540) 11.462
codecs.py:531(open) Page.py:590(get_raw_body)(7370) 6.243
wikiacl.py:185(may) security.py:52(<lambda>)(6580) 9.837
Page.py:241(getPageStatus) Page.py:122(get_rev)(15930) 12.456
Page.py:272(getPagePath)(16070) 0.485
Page.py:590(get_raw_body) Page.py:1285(getPageHeader)(910) 0.032
search.py:222(search)(10547) 10.557
search.py:702(formatContext)(910) 2.139
Page.py:176(getPageBasePath) Page.py:241(getPageStatus)(32000) 12.698