See MetaData, FacetedClassificationInMoin, ResourceDescriptionFramework, SectionParser

Contents

  1. Initial implementation
    1. #!meta section
    2. Integration with MoinMoin
  2. Comments and discussion
    1. A Full Implementation
    2. Applications
    3. Meta-Data and XML
    4. Is Meta-data only Semantic Data?
    5. YAML and inheritance
  3. Other related implementations
    1. Diamond wiki
    2. Graphing wiki

Initial implementation

It seems that the best way to do it is with a custom parser to get the data into the page, and custom search macros and actions to implement searching in meta data.

#!meta section

Its nice we have one way to enter meta data into a page. The current way is #key value, and used only in the page header. SectionParser use same way to add meta data to section of the page. We can reuse same code here: 

{ { {
#!section
#class meta
#acl BadGuy: EditorsGroup:read,write,revert All:
#Author Plony Almony
#Description How to use section in wiki page
#Status in progress
#Type idea
#Topic development
#Topic plugins
#Release 1.3
} } }

It can displayed like this:

Meta data

ACL

you may Read, Write

Author

Plony Almony

Description

How to use section in wiki page

Status

in progress

Type

idea

Topic

development

Topic

plugins

Release

1.3

Search similar pages

See the page source to see how much simple is the markup of the section compared to the needed wiki table markup that achieve the same effect.

The values can be saved and cached by the wiki, and used for a facet system. We can do the same in the page header, the section only give us a way to move the data to the end of the document, where we put the categories today.

This can be also in a sidebar.

In this way meta data could be at the start of a page or at the end, as the user likes. Page metadata could then be rendered in a table, with a special class="metadata". In this way, we could format it easily with CSS and JavaScript; for example, folding display at the top or at the side of a page, rendering it with smaller fonts, aligned right, only horizontal lines, etc.

Searching can look like this: 

[[MetaSearch(type:bug state:fixed)]]

We can do custom FullSearch macro that you install in your wiki, that override the standard FullSearch macro, and add meta searching. But I'm not sure about using key:value in the search, as keys can be anything and it can collapase with our search modifiers, e.g linkto: or title:. Instead, maybe syntax like meta:key=value: 

[[FullSearch(meta:type=bug)]]

We can cache page meta data in memory, and update the data each time a page is saved. This is not trivial as we have multiple processes accessing the same wiki.

Integration with MoinMoin

To be able to update the meta cache when a page changes, we need some hooks in MoinMoin. Currently we don't have hooks in PageEditor, but, we have the SecurityPolicy, which suppose to handle security, but it is actually a nice way hook to many operations: view, edit, delete, etc.

Comments and discussion

How does the scheme above enable data from established Meta-data schemes such as Dublin Core (http://dublincore.org/). There needs to be a more general attachment scheme, for example to annotate a page with RDF.

In the previous example, one can see that almost all keys of meta data begin with capitals as [A-Z][a-zA-Z]+. I suggest that to insert metadatas on page, insert capitalized PIs on top of the pages are regarded as metadata keys. e.g.) 

#Acl BadGuy: EditorsGroup:read,write,revert All:
#Author Plony Almony
#Description How to use section in wiki page
#Status in progress
#Type idea
#Topic development
#Topic plugins
#Release 1.3

And it can be implemented with Metadata parser as mentioned in the above section; lines with capitalized PIs are processed by metadata parser.

A Full Implementation

I have written a set of four plugins (parser/meta.py, action/metasearch.py, macro/MetaSearch.py, macro/BrowseMeta.py) to implement facet browsing, tagging, and meta search in MoinMoin 1.3. I use the following syntax for tagging: 

{{{#!meta
author: Charlie DeTar
status: beta
topic: development; plugins
release: 1.3
}}\} (minus the \, of course)

This is searchable using a search string either in a search form or in an in-page macro like this: 

[[MetaSearch(meta:"key = value")]] ... or ... [[MetaSearch(meta:"key")]] ... or ... [[MetaSearch(meta:"=value")]]

The search function simply extends the existing full search, so it also supports keywords, title:, regex:, etc.

The BrowseMeta macro creates a browser somewhat akin to Diamond Wiki's browser for this meta data.

It can be downloaded here: http://theendmusic.org/programming/MetaDataPlugin
warning - it hasn't been tested heavily yet. If you find bugs, let me know (<contact@theendmusic.org>). 

 key:: value

Applications

Besides semantic classification and navigational benefits for Wiki experience, metadata opens the door for creating web-based applications leveraging the Wiki platform.

Metadata turns document based Wiki pages into data entities by providing the attributes dimention. Think of many information management applications: corporate mail, bug tracking, process planning and project management, etc., they all have in common those attributed document entities as the core storage structure. That's exactly what metadata provides.

Having the data attributes in place, what's left to turn up actual applications is a set of tools that can operate on the domain of the attributed page entities. theendmusic design and implementation which is very economical and efficient, has already a couple of such tools: MetaSearch and MetaBrowse. A few other such tools can be added.

Tabular search result

like MetaSearch but result is presented in a table with specified columns whose values are taken from corresponding meta attributes, e.g. new macro MetaTable(type:card author:PersonOne, "Name;Color;Other Key") will return a table below. (Note, resemblence to relational select "Name","Color","Other Key" where type='card' author='PersonOne'

  • Name

    Color

    Other Key

    Page 1

    Red

    123

    Page 2

    Red

    13

    Page 4

    Blue

    23
    • Chart
    similar to table but attributes used to create a graphical view
    Templates

    with NewPage macro, already existing mechanism to populate new page entities with predefined attribute schema

    Data Binding

    a template page can be used as a stylesheet with placeholders corresponding to attributes, so that the result of search can be filled in. This would allow different presentations for the same metadata records. As in tabular example, a similar macro MetaBind(type:card author:PersonOne,Layouts/MyTable) will use search results and format them according to the layout in Layouts/MyTable page. 

    ||        ||'''Name'''||'''Color'''||'''Other Key'''||
#header
||{@index}||[:{@Name}]||{@Color}   ||{@Other Key}   ||
#footer
Total {@count}

    -- OlegKobchenko 2006-12-01 09:19:51 -- OlegKobchenko 2006-12-02 07:42:20

    Meta-Data and XML

    I think for the future it would be nice to use some kind of XML for this. because HTML metadata will always be minimal. We already can produce DocBook. I still think saving data as XML (Docbook) would be a good idea. Ok. you have some extra data, but I think it would enable many, many things. With plain text you always have to work with different markup. With XML every tag is meta data. it only depends how we use it. And I think it is one step further if meta data is not only for a whole page but for all content.

    What's also nice about Docbook is, that you can easily produce any other output. And it is the basis of many documentation standards (like in GNOME and KDE) -- ThiloPfennig 2005-12-18 09:50:25

    see also WikiDomFormatter

    Is Meta-data only Semantic Data?

    The above discussion shows the normal way, meta-data-issues are dealt with in the programming community: people are fixiated on semantical stuff and meta-data, that characterizes the content of a page like author, topic, status. On FeatureRequests/RethinkingPageOrganization I made the promise, not only to have "semantic" but also "pragmatic" metadata. Yet ACLs are not semantic metadata but of some other kind. On FeatureRequests/AssociatedPageFeature I further came forward with the proposal also to have some metadata "PageAssocation" that defines assocations between pages not on their content but on the aims of the user. This could be an easy solution maybe for having an associated comment page to a certain wiki page and/or implemeting tab-controls more deeply in themes, leaving the way, how exactly comment pages and/or tab-controls are implemented up to the theme programmer and thus not spoiling the code of Moin. But not to misunderstand me: I agree, that semantic metadata might be the most important type of metadata. But it is not the only one. There are also other types, which must be dealt differently and could not simply be displayed in a table like above. Themes could use e.g. pragmatic metadata to build the user-interface and offer additional tab controls. But all in all: I am not that deeply in moin to judge, whether my idea of pragmatic metadata makes sense. I just wanted to add it here so that you may think of that, too. -- OliverSiemoneit 2006-08-31 16:29:25

    YAML and inheritance

    What speaks against placing metadata in a subpage in YAML format? (in the future even javascript editors could appear for YAML http://yaml.org) YAML parses to python datastructures and supports schemas. It is also a very intuitive yet powerful syntax. -- AlvaroTejero 2006-09-08 12:59:48

    YAML seems OK. Why a subpage and not in itself?

    But metadata doesn't have to shift content down if we give it a class and control the display with CSS. Imagine (just as an example), a little box, flowed right.

    No, I wasn't thinking on the user interface right now. Certainly, the 2 window approach could simplify things a bit in the editing action. I just didn't feel that it would scare users to see metadata in the beggining, but it can happen, so let's work on this issue.

    We should start to design the UI for MetaData. We should integrate the UI with the storage in brief, but let's start with open, empty minds. It seems we don't have a page about the next major moin release with native metadata support... What are the developers thinking about that? Could/should we refactor all this to present philosophy approach, whole-moin impact of metadata, UI ideas, storage, etc.?

    Other related implementations

    Diamond wiki

    See DiamondWiki (also in Python).

    Graphing wiki

    See GraphingWiki for a working implementation since oct/2006.

    CategoryDiscussion

    MoinMoin: MetaDataInMoin (last edited 2007-10-29 19:10:17 by localhost)