MoinMoin 2.0

MoinMoin 2.0 (short: moin2) will be a big step in moin development, we are rewriting / heavily refactoring some core components of moin.

Repo: http://hg.moinmo.in/moin/2.0-dev

Minefield: http://test.moinmo.in:8080/

See also:

• the Talk at the DLR (text in German, pictures in English).

• the Europython 2010 Lightning Talk Slides (en)

• the Europython 2010 Lightning Talk Video (en)

The big news

Storage API

moin 1.x: the storage code was mixed into Page, PageEditor and AttachFile modules (yes, quite messy).

moin 2: storage code will be in MoinMoin.storage package and misc. storage backends will be supported, all conforming to the same "storage API".

Storage Layers, Middlewares, Backends

• there are "final" storage backends that really store stuff
  • fs - filesystem storage (cdb for index)
  • fs2 - filesystem storage (sqlalchemy/sqlite for index/news, hash-addressing for content)
  • fs19 - filesystem storage like moin 1.9 (read-only, only for migration of old content)
  • SQLalchemy based backend (TODO: memory/performance tuning)
  • hg - mercurial DVCS storage
  • fileserver - expose parts of server filesystem (read-only)
  • flatfile - very simple flat-file storage (only for special applications)
  • memory - memory backend (only for special applications, like debugging)
• there are some kinds of middlewares / layers above them:
  • acl middleware - checks ACL constraints
  • router middleware:
    • enables you to "mount" different backends into the namespace
    • indexing mixin indexes important metadata to speed up stuff (see below)
    • translate item names to item UUIDs to make simpler final backends possible (WIP)

Mimetype items

In moin 1.x, there were 3 main types of stored objects in the wiki: pages, attachments and user profiles

In moin 2, there will be only 1 type of stored object: an "Item".

Some facts:

• a storage backend stores named items
• an item may have metadata
• an item may have revisions
• a revision has metadata
• a revision has data
• metadata can be any key: value pair
• data can be of any type (text, binary)

Metadata

As seen above, moin2 does much more with (own) metadata and even enables storage of user-defined metadata.

To speed up processing, metadata of items and revisions will be indexed (using sqlalchemy).

Index can be used for:

• selecting wiki items/revisions by some criteria
  • by current item name, by uuid, ...

  • TitleIndex like item list

    • more difficult than in moin 1.x as we have all sorts of item mimetypes
    • but we either want to show just some specific mimetypes (== "wiki pages")
    • or we only want to show toplevel items?
  • theme: when showing some item, list sub-items in some panel
  • mimetype-specific listings

  • RecentChanges == items/revisions changed since date

  • "Items edited by me/by some editor"
  • Items linking to some specific item
• selecting user items by some criteria
  • by uid, by user name, by email, by jid, by ldapdn, ...
  • users subscribed to some wiki item
• sorting such lists (by name/date/...)

TODO:

• indexed search in metadata

Item OO user interface

moin 1.x: pages and attachments had separate user interfaces (deleting a page worked differently than deleting an attachment, uploading an attachment worked differently than uploading page content, pages had revisions, attachments not).

moin 2: unified user interface for items:

• show item (render wiki page, render image/video with some additional data - all done in wiki content area)
• item infos / history
• modify item (create, edit, upload)
• rename (delete == move to trash)
• download item content (raw wiki page, raw image)

Page, PageEditor, AttachFile code modules are dead.

TODO:

• move remaining stuff from Page / PageEditor to Item (or whereever it should be).

• UI still needs much polishing / theming before end user can use it, see themeing.

Code cleanup

Lots of crappy code was deleted, still some left.

Tree based transformations

(See /Converters for more information)

moin 1.x: parse wiki markup line-by-line, create html on the fly (problems: crap html, no xhtml, crap include, crap TOC, ...)

moin 2.0: input -> converter -> DOM tree -> converter -> output

• wiki -> DOM

• creole -> DOM

• reST -> DOM

• pygments -> DOM

• docbook-xml -> DOM

• image / audio / video -> DOM

• DOM -> html5

moin 2.0: reverse direction (e.g. for GUI editor)

• html -> DOM

• DOM -> wiki

• DOM -> reST

• ...

Features:

• generating well-formed x(ht)ml(5) output is easy now
• we use emeraldtree, which is a fork of elementtree
• operations on the dom tree make some stuff much cleaner / easier:
  • TOC
  • include
  • section edit (TODO)

• html -> DOM -> html filters out dangerous stuff

TODO:

• more practical testing of dom code, fix bugs

• some of the stuff has been worked on in GoogleSoc2010

GUI editor update and different approach

• ckeditor 3.x
• currently just stores html items

  • no conversion problems

  • but: no collaboration between gui editor and text editor users on the same item

• conversion is WIP (see Tree based transformations)

Themeing with Jinja2 templating engine

moin 1.x: Output generation was mostly done by python code with embedded html. Some of this was within pluggable themes, some was hardcoded.

moin 2.0: use Jinja2 templating engine for theme and user interface.

some of the stuff has been worked on in GoogleSoc2010.

HTML / CSS / Images / ...

• html5 (xmlish variant), we now can really do that

• jquery as base for cool (and less self-made) javascript stuff
• maybe svgweb for supporting svg stuff on crap browsers
• reorganize / cleanup css and images
• make css / js / html / images more extensible, modular, pluggable

TODO:

• CSS still needs lots of cleanup / work / a rewrite

XML serialization

Revisions, items, backends can be serialized / unserialized to XML.

Applications:

• backup / restore content of some or all storage content
• import from / export to other apps
• replace wiki xmlrpc with this (wiki xmlrpc is not powerful enough for moin2)

TODO:

• more tests
• replace page package with xml files
  • packagepages, i18n packages
  • interactive export/import for single/multiple pages

New plugin system

The old plugin system is not powerful enough and too complicated.

Just rough ideas, nothing final:

• use WSGI for action-like stuff (actions have full control over http response)?
• WSGI won't be enough as a general plugin mechanism (it is too low level, only for web stuff, not usable for internal stuff like macros, converters, etc.
• plugins should be able to have own i18n, static files (css, js, html, templates)

Framework / Libs / Python

Already done:

• we require Python 2.6 now, so need to test with older pythons any more
• We use Flask now as the framework.
  • restful URLs
  • werkzeug for WSGI stuff (since moin 1.9, will get even more and better used in moin2)
  • jinja2 for templating
  • flatland (+ j2 macros) for forms generation / validation of entered data
• use sqlalchemy (by default with sqlite backend) for some indexing / selection / speedup

Just rough ideas, not final yet:

• use some flask extensions: flask-babel, flask-script, flask-themes?

• use babel for i18n, pytz for timezone support

How to help

As you see above, all this is a quite big task and we need help:

• good Python, Javascript, HTML/CSS developers
• people with a good artistic skills and taste for theme development

How to start:

• Talk with us on MoinMoinChat.

• Read and try the new code.
• Expect brokenness/incompleteness and help fixing it.
• Bring in your ideas about how to improve. Some stuff is not specified yet.

Release date

As moin2 is still in heavy development, we can't give any release date yet.

"Not this year, maybe next year." is maybe not totally unrealistic.

But we can give some criteria that help you to choose:

choose moin2 if ...

• you are a developer and

  • you want to hack it / fix it / improve it

  • you want to write a wiki in python (don't start from scratch, we already have quite some useful stuff and there's lots to play with)

  • you plan to do a bigger software development project related to moin that won't go into production soon anyway. if you would do it for moin 1.9, you'ld likely have to rewrite it for moin2. while you develop your stuff, moin2 will get nearer to release and moin 1.9 will get more outdated.

• you are a user and
  • you like to play with cool stuff, even it it is partly broken
  • you like less, but more powerful features

choose moin 1.9 if ...

• you are a user and
  • you need something stable for "production"
  • you need functionality that is not (yet?) present in moin2

  • you need something that is highly compatible with moin 1.9

  • have lots of important data in moin 1.x format
• you are a developer and
  • you want to do no or only small changes or extensions for moin (note that you can use some stuff we also use for moin2, like jinja2, flatland, sqlalchemy, ...)

Note: Some old content can be still seen here: here