Help on Storage Development
Contents
Prerequisites
This guide assumes you have already read and understood Storage2009/HelpOnStorageConfiguration as the terminology used there will be used here as well.
A Complete Tour
When MoinMoin receives a request from a browser it looks up the action that should be taken (e.g. most commonly 'show page') from the request arguments. It then creates a so called context (similar to old request object) that contains information about the user who accessed the page in the browser and other configuration parameters (such as what is used for storage). The action in question is invoked and the context is passed to that action. Dependant on the mimetype of the item the user wants to access, an appropriate high-level item is created in the show action. We then get the content from that high-level item and ask our theme to render the content for the user.
Now, where does the content come from? As explained in Storage2009/HelpOnStorageConfiguration it may come from an instance of any final backend type supported. Before the content is accessed (or in the case of the modify action, modified), the ACL middleware (that belongs to the backend the item comes from) automatically performs permission checks and raises an AccessDeniedError if appropriate. This means that you usually do not need to take care of checking permissions yourself.
In the early steps of processing the request the namespace_mapping that was defined in the wiki's configuration is used to assemble two instances of the router middleware which are the central points from where storage data is accessed throughout the wiki. One of those routers takes the ACL rules that were specified for each individual backend into account (hence protecting the backend's contents), the other doesn't. The former can be accessed as request.storage, the latter as request.unprotected_storage. You can use these middlewares just as if they were normal backends and need not take care of putting items into the correct backend yourself.
The Storage API
The storage API of MoinMoin >2.0 is defined in MoinMoin/storage/__init__.py. It is well documented in the docstrings. It might still be helpful for you to get a better idea about the components it's comprised of. We will explain that in the following so you can start using the API in your own code.
What data is handled by the storage API?
All data. Pages, Attachments (Movies, Soundclips, Images, ...) and even users and their confidential data.
The Players
- Backend - A backend is a collection of zero or more items. It manages them and makes sure that item names are unique. It allows to iterate over items and revisisions and to create, get and search for items.
- Item - An item is a collection of zero or more revisions. The item is the only player that has a name associated with it. It contains metadata about itself. It allows to create, get and iterate over its revisions.
- Revisions - A Revision can contain arbitrarily large amounts of data. This could be a page's content at a given point in time or even your movie's data. Revisions also have metadata.
StoredRevision - Revisions that are only used for reading content. These are immutable.
NewRevision - Revisions that are used when new content should be stored. These can be modified until they are committed to storage.
How it is used by Moin
MoinMoin needs to store several different things. In the following it will be explained how these things are stored, so you will get an idea of how the storage API is used and how it can be used.
Storing Pages
A page in Moin consists of a name, meta-information (such as the page authors) and obviously the page's contents. We now store a page as an item that makes use of revisions. If a non-existing page is modified and saved, a new item (with the page's name as item name) is created along with a first revision of that item. The contents the user entered for the page are then written to the revision. The meta-information is stored separately in the revision's metadata. After all data has been pumped into the revision, the item is committed and the new revision ends up in storage. If the same or another user now comes along and modifies the already existing page, Moin realizes that there already is an item with the name of the page and hence does not create but get the already existing item. A second revision is created and the new page contents are written to the new revision alongside the meta-information of this second modification. Let's say another user wants to look at the changes that have been made in revision 2 of the page (Note that the term 'Revision' is used in page and item context alike). In order to construct the differences between the two revisions the diff action obviously needs to get both revisions of the item. It then reads both revision's data and compares it. The result of the comparison is then rendered in the browser.
Storing Attachments
Attachments are stored in the very same way as pages. They have been special-cased in MoinMoin <= 1.9 but are now treated as items as well.
Storing Users
Obviously you need to store your users' profile data somehow as well in order to send emails to users or to allow them to log in with their password. It might not be clear at first glance how one would store users with the approach described above, but the answer is quite simple: We don't need to revision a user's data, so we can just use a normal item (with the user's id as item name) and store the user's data in the item's metadata. That's it. Whenever the user updates his data, the item's metadata is simply updated reflecting the change made by the user.
Using it
Below we will provide a few simple examples on how you can use the storage API to store and retrieve data. Note: For all of the examples below we will assume that a backend object is present already. In order to get used to the API, you can construct it as follows:
If you want prefer to work with a persistent backend, you can choose between the fs and hg backends. A snippet for a persistent environment is given below:
1 >>> from MoinMoin.storage.backends import fs, hg
2
3 # If you want to use hg, use:
4 backend = hg.MercurialBackend("path/to/already/existing/folder")
5
6 # And for fs it would be:
7 backend = fs.FSBackend("path/to/already/existing/folder")
8
9 # Note that you should use only one of those, as the second assignment to the name backend will overwrite the reference to the first backend.
Creating an Item with Revision
This is the most common case. We will just create an item and a revision and store some contents.
1 # We create an item with a unique name. No other item in the backend has this name.
2 >>> item = backend.create_item("MyDiary")
3
4 # There's a handy attribute on every item object that gives you its name:
5 >>> item.name
6 "MyDiary"
7
8 # Note that you must pass the number of the revision you want to create as argument.
9 # This is done for concurrency reasons. You can also pass item.next_revno, which would
10 # be 0 in this case because the item does not have any revisions yet.
11 >>> rev = item.create_revision(0)
12
13 # There is a similar helper for revisions. This gives you the revision's unique number (item-wide, not backend-wide)
14 >>> rev.revno
15 0
16
17 # We can modify the revisions metadata. Revisions support dict-like access for this.
18 >>> rev["Date"] = "Today!"
19
20 # We now write some data to the revision:
21 >>> rev.write("Beloved diary, today I learned how to use MoinMoin's great storage API!")
22
23 # In order to actually store our changes in the storage backend, we need to commit the item:
24 >>> item.commit()
Adding a Second Revision
In order to add a revision to an already existing item, you would do the following:
1 # Since we have already created an item once, we *get* it this time and do not create it again:
2 >>> item = backend.get_item("MyDiary")
3
4 # Create the next revision. Again: You must pass the next revision number. We do that with the
5 # item.next_revno helper this time, so you don't need to compute it manually.
6 >>> rev = item.create_revision(item.next_revno)
7
8 # Naturally, we can again store something in the revisions metadata...
9 >>> rev["Date"] = "Yesterday + 1"
10
11 # ... and write to the revision....
12 >>> rev.write("Hey Diary! Today I'm gonna write my first piece of code using the storage API!")
13
14 # ... and commit the item again so we can be sure that our data was actually saved.
15 >>> item.commit()
Getting an Old Revision
If you want to take a look at revisions that have already been added to storage, you can do it like this:
1 # Since revisions are bound to their item, we first need to get the item that contains the revision
2 >>> item = backend.get_item("MyDiary")
3
4 # We can now get the revision we want...
5 >>> rev = item.get_revision(0)
6
7 # ... take a look at its metadata...
8 >>> rev["Date"]
9 "Today!"
10
11 # ... and of course read its contents:
12 >>> rev.read()
13 "Beloved diary, today I learned how to use MoinMoin's great storage API!"
Accessing an Item's Metadata
As was already mentioned when explaining how Moin stores users, item can have metadata as well. Accessing it is similar to how a revisions metadata is accessed.
1 # We need the item obviously, so get it if we don't have it around anymore
2 >>> item = backend.get_item("MyDiary")
3
4 # We need to acquire a lock on the item so that only one person is allowed to change the
5 # metadata at any given point in time. We do that with a call to change_metadata().
6 >>> item.change_metadata()
7
8 # We can now set new metadata key/value pairs as we see fit
9 >>> item["Warning"] = "This diary is private! Take your filthy hands off!"
10
11 # We need to release the lock that we acquired before. This also flushes the metadata to
12 # the storage system.
13 >>> item.publish_metadata()
14
15 # We can now access the key we set before and get the value back. (This is read-only now as
16 # we don't have a lock anymore).
17 >>> item["Warning"]
18 'This diary is private! Take your filthy hands off!'
Error Handling
There are several storage specific exceptions that may be raised while you are working with the storage system. For a complete list, please see MoinMoin/storage/errors.py. We will show a few of those errors in practice:
1 # If we try to change an item's metadata without having acquired a lock, it will raise an error:
2 >>> item["Warning"] = "This diary is a wiki! Feel free to modify."
3 Traceback (most recent call last):
4 File "<input>", line 1, in <module>
5 File "./MoinMoin/storage/__init__.py", line 650, in __setitem__
6 raise AttributeError("Cannot write to unlocked metadata")
7 AttributeError: Cannot write to unlocked metadata
8
9 # We will also see an error if we try to create a revision that has already made its way to the storage system:
10 >>> rev = item.create_revision(1)
11 Traceback (most recent call last):
12 File "<input>", line 1, in <module>
13 File "./MoinMoin/storage/__init__.py", line 782, in create_revision
14 self._uncommitted_revision = self._backend._create_revision(self, revno)
15 File "./MoinMoin/storage/backends/memory.py", line 179, in _create_revision
16 "The revision number must be latest_revision + 1.") % (item.name, last_rev, revno))
17 RevisionNumberMismatchError: The latest revision of the item ''MyDiary'' is 1, thus you cannot create revision number 1. The revision number must be latest_revision + 1.
18
19 # Due to the uniqueness of item names we cannot create an item with an already occupied name:
20 >>> item = backend.create_item("MyDiary")
21 Traceback (most recent call last):
22 File "<input>", line 1, in <module>
23 File "./MoinMoin/storage/backends/memory.py", line 100, in create_item
24 raise ItemAlreadyExistsError("An Item with the name %r already exists!" % (itemname))
25 ItemAlreadyExistsError: An Item with the name 'MyDiary' already exists!
26
27 # We cannot get items that do not exist:
28 >>> item = backend.get_item("MyOtherDiary")
29 Traceback (most recent call last):
30 File "<input>", line 1, in <module>
31 File "./MoinMoin/storage/backends/memory.py", line 71, in get_item
32 raise NoSuchItemError("No such item, %r" % (itemname))
33 NoSuchItemError: No such item, 'MyOtherDiary'
34
35 # The same is true for revisions:
36 >>> item.get_revision(404)
37 Traceback (most recent call last):
38 File "<input>", line 1, in <module>
39 File "./MoinMoin/storage/__init__.py", line 734, in get_revision
40 return self._backend._get_revision(self, revno)
41 File "./MoinMoin/storage/backends/memory.py", line 153, in _get_revision
42 raise NoSuchRevisionError("No Revision #%d on Item %s - Available revisions: %r" % (revno, item.name, revisions))
43 NoSuchRevisionError: No Revision #404 on Item MyDiary - Available revisions: [0, 1]
Processing Several Items/Revisions Iteratively
There are several ways you can access multiple items or revisions iteratively. We will demonstrate how:
1 # By now you should be familiar with the following call:
2 >>> item = backend.get_item("MyDiary")
3
4 # You can get a list of the item's revisions' numbers:
5 >>> item.list_revisions()
6 [0, 1]
7
8 # You can also iterate over all revisions that the *backend* has. Note that the following is just the same as what we
9 # had in the step before because there only is one item in the backend in this case.
10 >>> [rev.revno for rev in backend.history()]
11 [0, 1]
12
13 # In a similar way you can iterate over the items a backend has:
14 >>> [item.name for item in backend.iteritems()]
15 ["MyDiary"]
16
17 # IMPORTANT: Note that you should NOT do something like list(backend.iteritems()). While revisions, content and metada
18 # may be loaded lazily, some backends need to perform some additional bookkeeping and such a call would make
19 # the backend explode. The fs backend, for example, would struggle since it keeps an open file handle for each item.
Extending the Storage Eco-System
The most common way to extend the storage API is to write a backend yourself. That backend can then be used by yourself or others with the API presented above. It will automatically fit into the storage system and can be used in combination with the already existing middlewares.
If you want to write your own backend, all you have to do at the very least is to implement the Backend class (MoinMoin.storage.Backend). Generic items and revisions will then automatically be available for your backend, so you don't need to implement those yourself as well (although you can do that if you want).
The storage system internally is constructed in a way that allows to easily write and add new backends. Most method calls you can perform on an item or a revision call an internal method on the backend by default. This is why it is sufficient to implement the backend (including the internal methods, obviously) to get something functional. For reference, please see the simple MemoryBackend implementation in MoinMoin.storage.backends.memory. In order to get started you can just copy MoinMoin/storage/__init__.py to a new file in MoinMoin/storage/backends and implement the methods that raise a NotImplementedError. (If you have something that you would like to share with us and perhaps even see included in MoinMoin, please do not hesitate to contact us.)
Happy Hacking!