How to handle twikidraw stuff

Twikidraw stores 3 files, not only 1.

We usually have 1 revisioned file in item storage.

Then the item storage is not general enough. If we want to keep revisions for drawings,a and each one contain 3 files, we need to keep 3 files for each revision. The simplest way to do this would be to have a directory for each revision, containing all files. This can also work for regular pages, which have 2 files per page, the page text and the page meta data.

Here is a general structure:

Item
    current - current state
    revisions/
        00000001/
            meta
            data
            more-files-as-needed
        00000002/

It will use more directories, but it's simple and can handle anything, and also easy to purge any amount of revisions by deleting the directories. For example, keep only last 10 revisions.

-- NirSoffer 2005-02-15 22:54:26

I just realised that it TWikiDraw will not combine well with such a structure ... TWikiDraw mainly issues 3 transactions. And we do not really want to have some of them without meta data in the system. So my next suggestion: either we store 3 different items and implement additional delete, modify, rename logic (this should be easy in the new model, at least I hope so) or we modify TWikiDraw to submit a compound file that stores the 3 data domains and has to be unpacked on page view etc.

• Unclear what "will not combine well"!?

That way we loose the capability to provide direct access to the image + map. It would be possible to store the map and the draw file within the image png (using non-critical private chunks -- png is a very open an extensible format). This won't break viewer compability and we could still provide direct access to the image itself. The map is integrated into the page, so there is no need for a direct access here. The draw file is not accessed that often, so its no issue. -- OliverGraf 2005-02-16 06:49:26

• That is possible. But having a type renderer that unpacks the file is no problem either, in any case this is not a speed issue.

Even if we can solve the problem for TwikiDraw by saving custom png format, we still have the problem of separate data and meta files for any other items, which the solution of revisions and revisions_meta directories is quite ugly. We need a generic way that works with everything we will ever need. -- NirSoffer 2005-02-16 12:27:24

• TWikiDraw issues 3 transactions, so you cannot use parallel data storage like you described above.
  • With one or more transactions, you end with multiple files per page state - revision. You need a way to keep the multiple files for each revision.
    • Maybe you even need a group state. I. e. it should not be possible to change the state of a single file except for replacing or adding a file (that is what TWiki does).

New stuff

ItemName
    current/
        meta
        data
        more-files-as-needed

    revisions/
        00000001/
            meta
            data
            more-files-as-needed
        00000002/
            meta
            data
            more-files-as-needed

    sub/
        SubItemName/
            ...

• direct, fast access to current revision data and metadata
• fixed path of current stuff (makes script hacks easier)
• page.exists() == os.path.exists(ItemName/current/data)
• creating a new rev is easy:
  • create a writelock for item
  • read meta dict from current/, revision = meta['revision']
  • create new/ item rev
  • meta['revision']++, do other modification to meta as needed
  • store meta, store data into new/
  • create a readlock for item

  • mv current revisions/<revision>, mv new current

  • release readlock, release writelock
• maybe later: hierarchical storage of subitems under sub/
• maybe later: store all stuff related to a revision in a single 00000001.zip file. Would be much less space usage on disk, but speed and cpu load impact would have to be seen (compression could be 0).

Problem: if you have a bigger file item and change meta-data multiple times, it would have to store the same data file multiple time.

Solutions:

• just do it and have some purge policy
• use hardlinks (win32file and win2000+ has them)
• do separate data file revisioning and point from a metadata entry to the right data file revision

Using separate metadata and data revisioning

ItemName
    itemmeta (contains item level metadata, including pointer to current meta rev)
    metarevs/
        00000001
        00000002
        00000003 (contains revision level metadata, including pointer to current data rev)
    datarevs/
        00000001
        00000002

• purge has to do refcounting for the datafiles (by reading all metarevs) before purging a data file

• item.exists() is best solved by some WikiTrash namespace (fast, no deleted items in other namespaces) or 2nd best by some entry in itemmeta (quite slower, we would have to open and parse X/itemmeta when rendering links pointing to X)