Breaking the wiki borders by adopting pages from other wikis

This page is part of NetworkOfMoinMoinWikis group. See alternative IrcEventSystem for alternative implementation.

By now the borders between wikis are quite strong. Although we have InterWiki links, cooperation across wikis is more or less impossible. This is more or less because external pages don't show up in the local RecentChanges and people don't check RecentChanges in all wikis that have probably interesting content.

There are some InterWiki features that have tried to improve the situation but with no real success. To name some:

• SisterSites

• MetaWiki

• UnifiedRecentChanges

I think they have failed because their authors did not understand what a wiki is: A community that communicates about a specific topic. I think we need features that keeps the wiki as a bundle of pages about a specific topic, but don't stop at the server borders.

We add another magical page (AdoptedPages) that contain a list of InterWiki links. Changes of these pages show up in the local RecentChanges. That's all.

This allows the wikizens to add external pages to the focus of the wiki and makes discussion with other wiki communities possible.

-- FlorianFesti 2004-08-25 10:10:04

To make this more clear: Here is an example list of Pages that I would like to subscribe to: /ExamplePages and /WhyAdoptedPagesIsUseful

The name "AdoptedPages" does not tell much about how it works. So a better name would be appreciated. Perhaps something with RecentChanges?

Wiki Embassy

Each wiki instance could have an "embassy" in neighboring wikis. This is a kind of home page like a user home page but for a whole wiki. As this page is of course an adopted page the other community can talk to us directly in their wiki.

XmlRpc Server

First try:

XMLRPC interface wiki side

• notify(String from_url, String message)

  • tell wiki that it is time to look for new changes
  • may be used for any other push mechanism
  • don't push data directly - too dangerous

  • message is an arbitrary string used to decide what to do. Additionally it may serve as password to authenticate the caller.

XMLRPC interface server side

• array getRecentChanges(Date timestamp) gives unfiltered list, see WikiRPC v2

• array getFilteredRecentChanges(String rpcurl, String message, Date timestamp) gives a filtered list - subscribed pages only

• subscribe(String rpcurl, String message, utf8 interwikiname)

  • message is sent back to the clients notify method and is also used to authenticate for unsubscribe and listsubscriptions

• setSubscriptions(String rpcurl, String message, array interwikinames)

  • interwikinames is a array of UTF-8 encoded interwikinames

• unsubscribe(String rpcurl, String message, utf8 interwikiname)

• array listsubscriptions(String rpcurl, String message) returns a list of utf8 interwiki names

• notify(String from_url, String message)

  • act as client for wikis pushing RCs to the server

Wikis are encouraged to support the server interface, too.

see also: WikiRPCInterface2

Alternatively the server can offer a RSS feed.

I started implementing this XMLRPC server. It will support WikiRpc v2 and RSS feeds as sources (see MoinMoinSyndication) and offer the above XMLRPC interface.

We have to avoid that changes of AdoptedPages don't get back to the server as own changes! The wiki instance should offer an RSS feed or XmlRpc interface that does only include the own changes or we'll really mess up...

• alternatively we could tag the external RC entries.
  • xmlrpc
    • ???
  • RSS
    • ???

Implementation Log

• translate page names to interwiki names

• save state

  • automatically on every change

• load

  • on start

• config file

• polling

• more backends:

  • RSS via Universal Feed Parser

    • UseMod style

      • MoinMoin needs fix

    • add more
• more value checking!!!!

• allow subscription regexes

• improve performance of "is_subscribed" check

MoinMoin side implementation

• Configuration variables
  • XRC_Servers
    • dict URL: password
  • seperate incomming and outgoing Servers?

• AdoptedPages processor (for subscription)

  • resend to server link

  • still breaks if there are several areas per server!

• notify XMLRPC method

  • integrate into standard MoinMoin

  • use plugin to dispatch (notified?)

  • divide message into two parts by :

    • use first part as plugin name
  • perhaps change signature to something more general...
• send notify on edit to server

  • dispatcher (plugin mechanism?) - EventSystem would be cool...

• external RC storage (may be recycle from server implementation)

• new RecentChanges macro that merges external RCs

Server Web Interface

• Show available Wikis

• Show UnifiedRecentChanges

• Interface for Editing subscriptions

• Showing Personal RecentChanges.

Can this be done by a wiki? Do we really need this?

Wikiside Interface

Wiki engines may implement any user interface to allow AdoptedPages. Here are some ideas:

• Adopt all pages that are linked via InterWiki links

• offer special markup to make target InterWiki links adopted

• offer text input box with add button
  • offer a list of adopted pages and "remove" buttons/links for each page
• use a special page with one page per line

• use a processor (see MoinDev/PluginConcept)

Personal RecentChanges

The AdopedPages server could offer a human user interface to offer personalized RecentChanges. Functionality is more or less the same as for AdoptedPages, but the audience are individuals and not wiki instances/communities.

Performance Requirements

Scenario: AdoptedPaged are widely used.

• Each wiki has AdoptedPages from n wikis in average - m AdoptedPages from each

  • n ~ 10-30
  • m ~ 10-20

  • -> 100-600 AdoptedPages

• average wikis have 30-100 edits/day
• Wikipedia has 40 edits/minute right now (En 18/min) - 60.000 edits/day, 1.800.000 edits a month (November 2004)

  • and Wikipedia ist still fast growing... (Wikipedia EN Stats)

  • on the other hand is wikipedia not that interesting for AdoptedPages

• without Wikipedia we have 300-3000 external edits of interest/day that have to be filtered.
• with simple Polling we have n * 1/polling intervall polls per wiki
  • 10-30 / 15 min = 1-2 polls/min
  • 15 delay (7.5 min average) is quite long for a "real time" discussion
  • 2 polls/min is ok, but noticeable
• central servers can reduce polls to 1/polling interval

Implementation Ideas

• perhaps we should add "Type" information to the InterWikiMap (What wiki engine runs that site?)

Subscription

Make the foreign wiki notify us if the page we are interested in is changed.

• much more difficult, because the other wiki has to support a proper mechanism

Mail notification:

• relatively widely supported
• Mail is a slow medium
• need a user account in the foreign wiki
• need to use the web interface - very wiki engine (version) specific (could be done by hand)
• need mailaccount and mailprocessing

HTTP Protocol:

• WikiTrackBack is another alternative that works with blogs as well.

  • not widely implemented

  • If I understand WikiTrackBack right it is missing one direction for our purpose.

Discussion

This is great idea. A nice example will be a link pattern pages. A page like this is:

• Duplicated on many wiki engine wikis, discussed by the specific wiki engine developers and users
• Duplicated on wiki about wikis, discussed by people who like to talk all day about wiki things

We can adopt such a page - but don't stop with RecentChanges - we should be able to edit the page, as if it was a page on our server. First we should implement this in moinmoin, so every moinmoin wiki can adopt other moinmoin wiki pages. This will be a big mess and great fun. Next, other engines will join.

Here is a related example I wrote yesterday while learning Twisted, it's a remote wiki server - you run this on your machine and it serves pages from another wiki. It's just a stupid example, but it's amusing to use. remote.tac -- NirSoffer 2004-08-25 10:39:52

Emulating external pages as locals is not that easy. Main problem are the local links that point to the remote site. I was just talking about listing them in the RC (of cause marked as external pages) and setting links to remote wiki from the RC. -- FlorianFesti 2004-08-25 12:22:07

Editing the wiki page as a local page is indeed impossible, as it can not have two namespaces for the same links. They are either links to one wiki or another. So maybe we should start with the easy solution of displaying the changes in RC, with links to the original wiki for viewing and editing.

Another candidate for adopted pages are all the help pages. Instead of keeping copies of the help pages in every wiki, we can just share MoinMaster pages. We can use local caching, updated automatically by a message from MoinMaster. -- NirSoffer 2004-08-25 12:49:52

How about an InterWikiInclude?

Btw: Shouldn't we use URC to insert the MoinMaster RecentChanges in MoinMoin, too?

Can this feature be OFF by default for at least the first version? I'm worried about scaling issues, giving spammers an amplified channel, and getting nasty emails from my hosting site. -- JohnWhitlock 2004-09-21 22:16:07

It will be off by default because you will have to configure which servers to use. -- FlorianFesti 2004-09-22 06:56:41