Moin for mixed environment development in non-software projects
Initially proposed for GSoC 2009, now put on hold until I have time. Suggestions to tune the spec are welcome.
Introduction
Development in non-software projects requires some added flexibility not provided by tools such as trac and redmine. While moin has potential in this direction, there are a number of usability enhancements possible that recognize the typical workflow and allow easier integration with the rest of the environment.
These are the ways non-software projects are different from software ones :
- Specification documents are generally not in wiki-friendly format. They are generally maintained in doc/odp/pdf.
- Milestones cannot be tracked with number of open bugs or tickets (actually, they can, but I am yet to find a team that is willing to move in that direction)
- Lots of extra types of files are involved, which are not possible to be represented adequately on a wiki such as diagrams, simulations, etc.
- The team is not always well-versed will computers and programming, so even the simplicity wiki-markup is not adequate unless you factor in some drop in productivity.
- Specification and status reports need to be compiled from a number of different sources - reports from various subsystems, etc.
- Code and data version control for non-text documents is important.
Here are some of the problems that seem to require addressing :
Storage of files as attachments does not enable simple version control. In order to achieve this, the scheme we sometimes use is to make the files available on a separate VCS or simply an FTP, and then link to it from the wiki. This workflow can be automated, by adding a separate kind of attachment which will basically pull the file from the appropriate source, and allow it to be re-uploaded with a commit message. The fact that these files are typically binary means that a meaningful diff is not trivially possible, but the infrastructure integration will certainly enhance productivity. Current MoinMoin development in this direction is by ChristopherDenter.
- Timelines and Milestones : Sometimes it is advantageous have a system which looks after the work that has been done, the work that should be done, and the time frames associated with that. Some pages could contain information about the targeted completion date and the persons/teams responsible, similar to the way in which Categories are recorded or with a macro. The list of tasks could then be sorted according to target time and priority. To the best of my knowledge, Ubuntu and other projects currently achieve this by having a release team that maintains the page manually. Small teams such as mine dont have the resources to put in such effort, and in any case, it is time better spent actually _working_ on the project instead, and distributing the management work. In addition, the page itself can contain information about how far the spec is fulfilled, and if there are any related tasks that it depends on. The data from all the pages can be automatically pooled to obtain a complete picture of the status of various pending tasks.
- Better integration with auth systems : LDAP groups support, integration with trac userDB, integration with redmine userDB, for example. Still need to iron this one out to figure out a workable deliverable specification, but the bottom line will be to make it easier for Moin to share its users with other systems.
- Improve RSS feed generation : Allow creation of RSS feeds for changes in a category / a set of related pages. Can allow for the creation of announcement systems.
- Related pages : Allow one page to tag another. Works similar to the category system, where each page can contain one or more macros pointing to another page, and the list of pages pointing to it can be displayed as a list of related pages. I would use it to allow a group to tag another when making certain spec documentation, etc.
- Generate full documents by pulling from various other specified pages and compiling it together. Basically, this would make use of Moin's feature of importing a page from another.
Proposed Implementation & Deliverables
Storage
Storage is already being worked on by ChristopherDenter. Using the storage system that now exists in Moin, we could write a new backend (and middleware) that knows how to communicate with a version control system. For this, I would work with bzr and hg, mainly because that is what I use / will use. It should be simple to extend it to svn. Once the backend is written, we would be able to do the following :
- Import a file from the VCS onto a wiki page. Display it like a link. If the file can be displayed by Moin (text only, image only) trivially, then the contents can also be displayed in a codebox.
- Choose whether the diff for the attachment comes from the VCS's diff (text only) or from the VCS's commit log.
- Performing a diff on the page will pull the changes to the file from the VCS.
Milestones
Every milestone is actually a wikipage, containing macros or new markup that specify when it is due, what it is about (category, but not in the moin sense of categories). Macros would require, well, writing new macros. New markup would require extending the DOM specification. Additional markup/macro parameters specify degree of completion and tag a group (this time in the moin sense) for its implementation, and dependencies on other milestones. On one, or multiple, pages, a macro is used to list all the milestones in one category, assigned to one group, or in the whole group in chronological or degree of completion order.
Each milestone page can thus :
- Declare when it is due, who is responsible, and what it depends on.
- Have a list of dependencies still unfulfilled
- Declare how much is complete
- Declare that something specific has halted/is impeding progress.
Any arbitrary page containing a macro can list :
- List of pending and/or completed milestones
- Dependencies of those milestones, specific reason for halting.
- Percentage completion of a larger, pseudo milestone.
- Potentially help identify tasks that need to be completed immediately (lots of thing depend on the milestone to be reached) and tasks that need to be postponed (depends on a lot of tasks)
Auth
LDAP groups integration is most easily achieved, although its somewhat a hack, by generating the appropriate groups pages from an LDAP group. The wiki will have to regenerate this page automatically at some sane interval, as well as allow for manually forcing an update. This could be done in a somewhat more elegant way with an LDAP backend that can generate said pages. The ideal way, of course, would be to rewrite the Groups part of the Auth and ACL system to accept LDAP groups as well. At this, I really dont know which of them is doable in the given timeframe, or if any of them is doable.
Related Pages
Related Pages are recognized not by an automatic system that generates the list, although that would be very nice. Unfortunately, the nature of the relationship is complex and different in different cases (two specification pages are related in that they are specifications. on the other hand, a specification page of section X is related to an implementation page of section X in that they are of the same section). Each page can declare that it is related to some set of specific pages. This system would either be independent of Moin's builtin Category system, or would extend it. I'm leaning heavily towards independent, since the RegEx for the Category system is baggage that is not something I'd like to have.
The typical use case would be having a hierarchial system of 'Categories' in a form of a 'tree' that starts from a point and extends downwards, and at each node branches but never merges between branches. Over such a tree, some nodes, mostly specification related nodes, would be linked to the counterpart nodes from another branch. Each page is then set to be related to all the pages it is directly connected to in the resulting graph. (These things probably have a formal name in graph theory. I'm afraid I dont know what they are).
Once implemented, every page will be able to :
- Declare the pages it is related to. In two pages of a relationship, either or both of the two must declare the other to create the relationship.
- With a macro, display a list of pages it is related to.
- With a macro, display the recent changes in all its related pages (or all the related pages of a specified page)
- Sort the related pages according to date of creation or date of last edit. It may be possible to sort according to importance, but how the importance of a page is declared or calculated is something I can't come up with a good solution for at this time.
- Generate RSS feeds that contain changes in the page's related pages, and containing entries from up or down the tree for n-hops (where n is specified by a macro)
Document Generation
Some clever collection of related pages should be able to, in theory, generate the required documents. However, since a number of these pages are likely to contain binary file formats, I'm not even going to try.
Broad Timeline
May :
- Creation of the specification for all the components, including the specific macro names, parameters.
- Familarization with the Moin core system including storage system and DOM / Wiki Markup conversion.
June (before Mid Term Evaluation) :
- Initial implementation of VCS storage backend.
- Initial implementation of LDAP groups support.
- Implementation of the Milestones system.
June (after Mid Rerm Evaluation) :
- Starting work on the implementation of the Related Pages : be able to describe pages as related and generate a crude list.
July :
- Finishing up the VCS storage backend and implementation of the modification of Page Info behaviour.
- Improving the LDAP Groups integration. Supported by creation of a test suite.
August :
- Finishing up the Related Pages System.
- Ironing out all the problems that came up in the last few months.
- Testing
- Merge with Moin (?)