Help on Storage Configuration

Introduction

The new storage system introduced in MoinMoin version 2.0 is quite different from the one used up to version 1.9 (which we will call the old storage system). With the new storage system, your content does not necessarily reside on disk as plain text files, but may be stored in any form imaginable that is supported by a storage backend. This does not only allow you to use the backend that best fits your needs, but you can also interchange different storage backends without much effort should your requirements change. You can also use several (different) backends at the same time, each taking care of some part of your wiki. Along with this change there has come a change in terminology: We don't speak of storing 'pages' or 'attachments' anymore but uniformly address them as 'items'. Simplified, an 'item' has a name and - perhaps revisioned - data.

Storage Backends Overview

A backend is a collection of items. There are multiple kinds of storage backends:

• final backends (they really do store data somehow)

• middleware backends (not storing data themselves, but usually wrapping some final backend(s), providing additional functionality.)

• special backends (these are usually final backends themselves that are only intended for some special use-case)

Final backends:

• fs backend - storing data into files/directories on your disk (not compatible with old storage system)

• hg backend - storing data into a Mercurial DVCS repository.
• (... - more to come as new backends are written)

Middleware backends:

• ACL middleware - Wraps a final backend and checks if the user has the necessary permissions to perform the desired action.
• Routing middleware - Wraps several final backends and routes items to them based on their item name. I.e., an item named 'Fruit/Banana' is routed to the backend mounted at 'Fruit/'. (cf. fstab on UNIX systems)
• ...

Special backends:

• memory backend - just keeps data in memory, do not use this for production!
• fs19 backend - just used once for migration of existing moin 1.7, 1.8 and 1.9 compatible data storage (read-only)
• ...

Starting a new wiki from scratch

Configuring storage backends for a new wiki setup is easy. In its simplest form you just do namespace_mapping = create_simple_mapping('fs:' + data_dir) (the default), which then automatically uses three filesystem backends for data, user and trash storage.

Advanced Configuration

The default configuration is sufficient for many use cases. For more complex usage scenarios, there's much more you can configure, which is the focus of this section.

The namespace_mapping

Let's take a closer look at this powerful configuration value. When MoinMoin performs a storage operation (like getting or creating an item) it usually needs the item name. This name is, for example, provided by some user who tries to delete a wikipage, say 'Fruit/Banana'. Now you, as the admin, may not want all those delete pages to end up in the same data storage backend (e.g. the same folder) alongside your important (and not deleted) business information. There is a simple solution to that: You just select another backend that will store everything below (the so-called 'namespace') 'Fruit/'. You do that in the namespace_mapping (so what you do is 'mapping' a backend to its namespace, hence the name.). Now this is in no way limited to just separating deleted pages from the other content. You can mix backends in the mapping and specify them for every namespace you like. By default, every item is stored in the default backend, unless there is some backend mapped to a namespace matching the beginning of the item's name. The default backend is the backend that is mapped to '' (or '/', since trailing slashes are ignored) which is specified last in the namespace_mapping. 'Fruit/Banana' would hence be stored in the default backend, unless there is another backend mapped to 'Fruit/'.

Example

So you have chosen to set up a more complex storage configuration. Let's look at an example of how you'd do that. What you need to do is provide a namespace_mapping yourself (not using the create_simple_mapping helper). To be precise, what you need define is a list of tuples, each tuple containing the following (in order): A point in the namespace where the backend should be mounted, the backend itself, ACLs to protect the contents of the backend.

The default configuration, manually specified:

   1                     # Namespace,        Backend,                ACL rules
   2 namespace_mapping = [('Trash/',         trash_backend,          content_acl),       # Deleted items end up here.
   3                      ('UserProfile/',   userprofile_backend,    userprofile_acl),   # Private user data (passwords, email) are stored here.
   4                      ('/',              content_backend,        content_acl), ]     # Everything else is stored here.

This is the general structure. For clarity, this uses some values that must have been defined before.

   1 content_acl = dict(default=u'All:read,write,create')   # Everybody should be able to read, modify and create new items
   2 userprofile_acl = dict(before=u'All:')                 # Only accessible by Moin internally
   3 
   4 # Define backends. Folders must already exist.
   5 trash_backend = FSBackend('trash_folder')              # All deleted items are moved to this folder
   6 userprofile_backend = FSBackend('userprofile_folder')  # Confidential user profile data is stored in this folder
   7 content_backend = FSBackend('content_folder')          # Folder where everything else is stored

Note that you could, for some or all of the backends, just as well use MercurialBackend in very much the same way.

Storage Backends Detailed

Filesystem Backend (fs)

Backend properties:

• requires POSIX file system semantics (works on Linux, BSD, Mac OS X, ... - but not on Windows, at the moment)
• uses one directory per item id and an item name to item id mapping (allows for easy renaming of items)
• uses one file per item revision that stores both metadata and data in a specific, binary format

To use a filesystem backend you do the following:

• import it first
• create it, giving it the path to a folder where you'd like it to store its contents

   1 # Configuration for filesystem backend
   2 from MoinMoin.storage.backends.fs import FSBackend
   3 
   4 my_fs_backend = FSBackend("path/to/the/folder")  # Now you can use my_fs_backend in the namespace_mapping as a backend.

Before running your wiki, make sure the folder you specified does exist, with proper permissions (i.e., writable by the user who runs the wiki/web server).

Mercurial Backend (hg)

Backend properties:

• uses a Mercurial repository for storage

• allows easy backup on-line (you can simply hg clone your data)

• allows simple wiki synchronization (run hg serve and pull data to other instance of wiki)

• allows off-line edits using command line

The easiest way to use the Mercurial Backend is provided below. Add this to your wikiconfig:

   1 from MoinMoin.storage.backends import create_simple_mapping
   2 
   3 # Use the mapping helper to set up a namespace_mapping with content/, userprofiles/, trash/ in data_dir automatically:
   4 namespace_mapping = create_simple_mapping('hg:' + data_dir)

If you need more fine-grained configuration, you can create MercurialBackends for the namespace_mapping like so:

   1 from MoinMoin.storage.backends.hg import MercurialBackend
   2 
   3 # Make sure that the folder exists and that you have the permissions necessary
   4 my_hg_backend = MercurialBackend("path/to/folder")

Backends prerequisites

MercurialBackend

In order to run this backend you need to have Mercurial installed. This can be achieved various ways depending on your operating system.

Linux

If you have setuptools installed, you can always grab latest version from PyPI:

easy_install -U mercurial

Otherwise, if you prefer to install using your systems packages tool, installation instructions depend on distribution you use.

• Debian, Ubuntu

   aptitude install mercurial

Windows

• You can grab the latest version of precompiled binary from Mercurial binary packages for Windows.

• There is also bundled version of mercurial in TortoiseHg installer.

• If you rather prefer to compile it yourself, please read http://mercurial.selenic.com/wiki/WindowsInstall.

Mac

Mercurial installation depends on your Python setup. If you use virtualenv it is better to use setuptools:

 easy_install -U mercurial

If you rather prefer system-wide install use Mac ports:

 port install mercurial

Migration

If you already had a wiki running before upgrading, you will already have some content you want to migrate. That content exists in the old storage system's format.

In order to help you migrate your contents, there is a conversion script that can read this old format. All you need to do is specify the paths to your old data and user storage folders (usually '.../data' and '.../data/user') in your new wikiconfig like so:

   1 data_dir_old = 'path/to/your/old/datadir'
   2 user_dir_old = 'path/to/your/old/userdir'  # Can be None if you don't want to migrate users.

Now issue the following command to actually perform the conversion:

moin --config-dir=/path/to/new/config migration backend

Depending on the size of your wiki, conversion may take some time to finnish. Now run your wikiserver and load a page to confirm that it worked. You can remove data_dir_old/user_dir_old now.

That's it! You are now running a wiki on top of the new storage system.