Real-Time Editing

Based on: FeatureRequests/RealTimeEditing

Allow pages to be edited by several people at the same time.

Kinds of TODOs:

• SOCTODO means that this is part of the SOC 2009 project of AlexandreMartani (support him, but leave the coding to him)

• TODO means: help needed, anybody is welcome to help with this (please coordinate here or on #moin-dev)

Status

• Repository: http://hg.moinmo.in/moin/2.0-storage-editor-amartani/

Development is being done on 2.0-storage branch. Simple synchronization is working, using mobwrite's daemon methods and an action that calls them directly, storing the pages on a Berkley DB. A simple list of users who is currently editing is also displayed.

To try it:

hg clone http://hg.moinmo.in/moin/2.0-storage-editor-amartani/
cd 2.0-storage-editor-amartani/
python wikiserver.py

Access localhost:8080 and edit the same page in two (or more) windows.

Mobwrite debug mode is activated, so in compatible browsers (Firefox with Firebug, Safari and Chrome) there will be a lot of mobwrite messages on console. To disable it, change mobwrite.debug on MoinMoin/web/static/htdocs/applets/Mobwrite/mobwrite_core.js.

Next steps:

• Usability improvements and bug fixing
• Possible text highlighting
• Write docs/changes and wiki help

Documentation (SOCTODO)

• Mobwrite javascript files are in MoinMoin/web/static/htdocs/applets/Mobwrite/.

• A modified Mobwrite daemon (python) is in MoinMoin/support/mobwrite/. The modification is mainly to get all information (temporary pages and views) from a BDB file supplied by the mobwrite worker.

• Action mobwrite acts as a simple proxy to mobwrite daemon, adapting some requests. The MobwriteWorker class has the methods that converts the requests from the clients to mobwrite server, and some high level methods (download and upload). The get_worker classmethod instanciates one of the subclasses, MobwriteWorkerProxy or MobwriteWorkerInProcess, according to configuration.

• Action ajax_info is used for ajax features in the editor (save and continue, locking renew). It reads post data, and returns json with information to be displayed for the user.

• /HelpOnRealTimeEditing

Server (SOCTODO)

Adapt the chosen real-time editor server to MoinMoin.

• Initially, just start Mobwrite server script in parallel to MoinMoin, and do all requests to mobwrite server directly.

• Adapted mobwrite to run in-process:

  • Modified mobwrite daemon, separating the core parts of the daemon and calling them directly.

• Make a preference to choose how to run it (as a daemon or in-process)?

  • High-traffic wikis can prefer use it as a daemon, for performance reasons (don't need to open db files every request, as they are kept in memory).

Integration (SOCTODO)

Code for this project will be based on 2.0-storage branch, as all the high-level actions (view, edit) was moved to items.

• Create template with mobwrite lib on client-side, with a save button to "commit" changes to the wiki.

  • Need to create an AJAX interface for saving items.

• Display users editing each page.

  • Editors are being stored in a picked dict, using caching api.

  • Perhaps use also the itemlock, so we don't have this information almost duplicated.

• Choosing editors on 2.0: There should be a way to choose using the simple text editor or mobwrite (and graphical one).

  • Enable while editing:

    • could it be made in a way that:

    • you do not need to choose beforehand whether you want "simple" or "mobwrite"
    • it detects whether mobwrite could work (js enabled, communication with mobwrite server works)
    • if yes, it offers to "share this editing session"
    • if no, it does not offer this, but you can just use the simple text editor
    • "unshare this editing session"?

  • Choosing before:

    • Override do_modify on Text class, and choose the template (editor) to use from some post info.
    • Other editors should be locked if mobwrite is being used.

    • Simple UI to choose the editor: "You can either create the item from scratch or select a template. [[mobwrite]] [[Simple]] [[GUI]]"

  • Just enable it if it is supported?

    • If it is the first user, enter the editor and enable mobwrite on the background, if possible.
    • If it is not, enter a "locked" page, and try to load mobwrite on the background.
  • Done using locking with messages offering to use mobwrite. Need some usability feedback.

  more stuff to think about:

• how to handle concurrent edits from simple and mobwrite editor?
  • probably, simple editor should be locked during mobwrite sessions (only available through mobwrite)
• is there some danger that someone gets access to ACL protected content via joining some mobwrite session?
  • request.user.may is being checked

Preferences:

• What preferences do we need?
  • activate mobwrite? use by default?
  • how should mobwrite work with lock? currently:

    • When lock="warning", display warn message and offer using mobwrite

    • When lock="lock", display only lock message and button to activate mobwrite

    • To have a real lock, would need to make lock="lock" and editor_mobwrite_mode=None

  • Should we have mobwrite enabled by default?
    • I think not, so we don't change the way users are used to, but we can "advertise" the possibility
  • Should we have some user preference? Like, if some user never wants to use mobwrite

Interface (TODO)

Polish editor interface. Try to make it similar to current FCKEditor.

• Text-only interface.

• "Preview" feature.
  • An always reloading preview can be too bandwidth consuming.

  • Always show last saved version?

    • So you know how it is currently saved, and it is easy to correct it and re-save
    • Can expand variables
    • Client would be asking periodically for the last saved version, and on change, ask for the entire page.
  • Have a "preview" button to preview the actual content of this page (as in previous versions)?
    • Can be used by both traditional editor or mobwrite.
    • Can spot mistakes before saving (although if there is anything wrong you can just correct and save again)
  • Have both
    • Too many actions to choose from can confuse users
• Syntax highlight (and would be useful to highlight incoming changes): available editors

  • CodeMirror and CodePress:

    • Creates the editor inside an Iframe, so it is more difficult to handle

  • EditArea

    • Don't create iframe, mantains textarea Missed it before, it does also creates an iframe, textarea contents are not synchronized with editor's content.

  • DuckWizard

    • Textarea content is always the same as the real content = much easier to integrate
    • Slow and unmanteined
    • Licensed under CC attribution (author wants a link)
  • Bespin
    • Uses canvas
  • After studding them a bit, I think it will be a bit difficult to do this with a good performance, especially with big texts.

Use cases

• When pages will be saved: periodically, or only when client clicks a "save" button? Using the save button for now.
• Who will be the author of the commit? Who saved the document, but other editors will be stored also.

Issues

• Reloading a page or going back and forward duplicates the editor at the editor's list on some browsers (eg. Opera and Chrome).
  • Problem is that some browsers don't trigger onunload event on reload or back. Inactive editors are cleaned up after some time.

Diary

Discussion

Add your thoughts, concerns and use-cases here.

Bugs/usability issues:

• the last save display pushes down the "upload file" button quite a bit, maybe it should be also in the row with the other buttons (it is also sort of a "save" button)

  • I think it would make the "save" bar too crowded.
    • When your merge moin/2.0-storage, there will be more space.
• All editors could leave the shared editor (assuming that someone else will have saved) and all new data would be lost.
  • Would it be better to have it "cached" somehow, like it works now with preview?
    • the old editor warns if you try to navigate away without saving
• Without Javascript:
  • "Save and Continue" is also disabled. I guess this is because that functionality is based on JS. Just thinking about whether this could also be done without JS.
• generalized state syncing (json?), that includes:
  • "who is editing" list, per editor info includes:

    • editor name

    • last change timestamp by this editor (or "active" / "inactive since ...")
  • "needs saving" flag (or even some metric about amount of change that needs saving)
  • "last save time"
  • "last save revno"
  • "last save editor who did the save"
  • editors' chat contents

  • "edit lock until <time>" - maybe it could also prolong the edit lock as a side effect of this exchange

• Pressing "Enable Sharing" removes current text instead of merging if it is done for the second or any later time

  • Needs testing - now it is merging contents. Perhaps offer to save before activating mobwrite, in case merging gives a wrong result.
• The editors list could use some CSS

• this sequence overwrites data:

    1. press enable sharing
    2. press enable sharing
    1. edit
    2. edit
    1. press save and continue
    1. edit
    2. edit
    1. press save
    2. press save
    expected: edit conflict

  Please test more such sequence, data loss may never happen!

• save button state (indication of "save needed") is not synced (for one of two cases, it is, but not for the other [if one editor has saved, the other still sees "save needed" by a still active button]) - "save and continue" doesn't disable globally if someone has saved
• conflict detection and merge works sometimes, but I had the impression that sometimes it also just overwrites stuff without merging/noticing
  • I had this impression too. I will test how it is doing on 1.9.

• release_lock should be a POST operation and redirect to the normal view

I like this GSOC project very much: it seems to me very advanced, the cutting edge in wiki development. A cool feature other wikis definetly miss! Good luck with it! However, I have also a little concern about the project. Guess what? Yes, right, there might be an accessibility issue! As far as I can see, bespin uses html canvas elements? But they are prone to problems, see http://esw.w3.org/topic/HTML/AddedElementCanvas. Furthermore real-time editing could be also a problem for the blind, since they have no chance to get lively inserted changes quickly: A screenreader has to read each line for them. To put it short: I do welcome this feature very much as also requested by me here: http://moinmo.in/FeatureRequests/TextEditorWithSyntaxHighlighting. On the other hand a would plead that the old text-box is keept in some way and also the way to have marked edit conflicts to preserve accessibility. This could mean: In the userpreferences you can choose between three edit modes (text, gui, realtime) with default edit mode=text?? That's it! Keeep going and good luck! ThomasWaIdmann, no sorry, just kidding -- OliverSiemoneit 2009-05-30 20:33:31

• Thanks for the comment. Bespin uses canvas, but mobwrite don't, and I don't think this will be necessary (although this can make some fancy features possible). Also, the old plain text editor will still be available. -- AlexandreMartani 2009-06-02 02:59:28

i18n issue in http://hg.moinmo.in/moin/2.0-storage-editor-amartani/rev/9ce3768d3b5c ?: "Save and continue"-button seems not to be translateable

• 2.0 branch is moving all HTML code to jinja templates, but this is still in early stages, and there is no i18n work on it yet. This button is equal to all other buttons on the template.

21:30  * dreimark tries amartanis repo
21:31 < dreimark> texts_db = bsddb.hashopen(mobwrite_daemon.DATA_DIR + 
                  "/texts.db") 
21:31 < dreimark> DBNoSuchFileError: (2, 'No such file or directory')
21:32 < dreimark> any hints what i need to configure?

• Sorry, I had stored mobwrite's database on a temporary dir I created here. Had moved it to cache dir, try again now. -- AlexandreMartani 2009-06-17 01:27:48

This will make an ideal experiment log file for our facility - several people working on our machine can enter details of their experiments, settings, comments about data. The other features of MoinMoin are excellent for existing pages, when it is less likely to have simultaneous editors. This solves the problem of several scientists working on the same page (today's log file) at the same time. Tried your 26/5 version; very nice, easy to install according to your three lines at top of page. I think text-only entry is a reasonable compromise, with the possible exception of putting other contributor's input in a different colour/font.

What are your plans for identifying editors apart from joint editorship tags on a save? I have had two thoughts.

1. markups in the form of comments inside a jointly edited document marking the beginning and end of changes by each author. This would be the easier option. Problem is the multi-editor save would look ugly in text mode if comments were "on", but this could be worked around if all multi-editor markup comments were removed when reading a page in for further edits. The information would still be available in the older save.
2. Markups on every changed phrase may be cumbersome. I was thinking about detecting trivial cases where there is little overlap, and two editors. Consider two type of save - one a normal save (single editor), the other a joint save. If one (Anne) saves, the system could save two copies; one attributed to Anne which is just her edits relative to the most recent single editor save, and a one with a later version number attributed to Ben which is current state of the document (at the instant of save). Then if Ben continues to edit, the situation is just like an ordinary save. Both are marked as single editor saves, and read (almost) as if the edits were performed in sequence (and are easier to follow without in-line markups). If there are three editors, then for the first to save (Anne) the system just save her changes to the most recent single editor save, and an automatic second save is marked as joint, and the session reverts to a two editor session. I guess induction gives you the rest.

So there would be a sequence of saves, some multi-editor, some single editor. The end result would be identical to the present system, all the differences between single author save would make sense or be easier to understand, and the only confusing part would be if you compared multi-editor saves with single editor saves, you might see apparent deletions that were in fact insertions. <-- BoydBlackwell 2009-06-27 06:27:01>

• For now, I don't have plans to add this kind of identification, but this can come in the future. Mobwrite's algorithm makes a bit more difficult to identify who has done each change (it's not a simple three-way merge, see details), but it can be done, on the trivial cases. But I don't know if behaving differently on a not so exact criterion is good. -- AlexandreMartani 2009-07-04 03:07:36

I do feel a bit that the ui is somehow too complicated. e.g. if I edit a page It does not make sense to allow someone else to open the page for editing too. I do think a lock message to the second editor and a invitating message for the first editor would be much better (User: ABC likes to edit this page too. Do you like to invite him?). if the first one (XYZ) enables sharing then I am asked at the time I select the page for editing if I want to edit it in shared mode together with user XYZ. If I deny I can't edit. -- ReimarBauer 2009-08-16 10:39:04

• We don't have mandatory locks anyway, and also don't want it either. So it would be just not be allowed into the editing session. -- BastianBlank 2009-08-17 10:09:24

  • Ok, that makes the ui much simpler, unless someone has selected sharing the page can't be opened for editing from another one. So not two or more could start editing and have to decide later that they want to share "now". And if one has enabled sharing all the others get shared mode when they try to edit this page. That will also avoid:
    • This page is opened for editing by localhost. You should refrain from editing this page to avoid editing conflicts. To leave the editor, press the Cancel button.

    • This page is opened for editing by ReimarBauer. You should refrain from editing this page to avoid editing conflicts. To leave the editor, press the Cancel button.

    -- ReimarBauer 2009-08-21 19:05:24

• Data loss in the following sequence:
  • Original text: "test1\ntest2\n"
  • User 1 adds "test3\n" so the text now reads "test1\ntest3\ntest2"
  • User 2 removes the complete text before the change is synced
  • Result, more or less: "test2\n", not "test3\n" as expected or completely empty.

-- BastianBlank 2009-08-17 10:06:46

• release_lock should be a POST operation and redirect to the normal view

• Save always take the version from mobwrite, which is one update behind the local view. A save request should first trigger an explicit update or use the version from the client.

  • Fixed save request only after syncing is done.

-- BastianBlank 2009-08-23 14:39:38

Plan for integration

Licensing

Problem: Apache 2.0 license is incompatible with GPL v2 (but not with GPL v3).

Easiest way for now is to keep it separate:

• plugin actions
• misc. directories with mobwrite code

TODO

1. extract generally useful code (and css, templates, ...) that is not mobwrite specific, keep it under GPL v2+
   • do that while implementing templating / new framework
2. try to make the rest plugins, distribute separately

Anyone helping with this?

CategoryGsocProject