'''Unify parsers and macros for both simpler user interface and code'''

See also UnifyMarkup

<<TableOfContents>>

= Unify Parsers and Macros =

Now that we have started to UnifyParsersAndProcessors, cause they are just the same, it's time to take a look at parsers and macros.

A macro is an extension plugin that can be called from wiki markup to perform some output depending on arguments given to it. In contrast a parser is an extension plugin that performs some output depending on some parameters ''and a block of lines'' given to it.

'''Conclusion:''' A macro is a parser that gets an empty block of lines.

If you now take a look at the discussion about LinkPattern``s, and the idea to use `[[link]]` for linking, it would be just nice to get this `[[` free (currently used by macros).

So if we could unify parsers and macros, we can get `[[` free and (as bonus) get a nice class based api for all (former) three types of extension plugins: Processor, Macro and Parser.

We are dealing with two different problems: the code handling plugins and the wiki markup. Unifying the code that handle both macros and parsers and plugins in general is good, but not related to the markup. Macro and parser are not the same - parser has content.
 * Who says that a macro couldn't have content? If we can handle it by the same code, there is not much point making a different user interface for it and speaking of 2 different thing "macros" and "parsers". And less items are easier for users to learn, so instead 3 things parsers,macros,processors like in 1.2, we only will have 1 item in 1.4 (no matter how we decide to call that thing).

We can use one namespace for macros,parsers,processors e.g. `PythonParser`, `CPPParser`, `FullSearch` - makes sense as all are page extensions and doing the same thing "process some text and output some page content". OTOH actions, formatters and themes are wiki engine extensions.
 ''I am not sure if actions in some cases wouldn't be page extensions. Because we could write user interfaces which do interact with the page body. And an action could be called from a macro or parser too. So you could write a macro which would start an user interface e.g. NewPage. We have two kinds of action macros now. One are of the type of !DeletePage and the other one like the new !SlideShow. '' -- ReimarBauer <<DateTime(2005-04-29T21:18:21Z)>>
 

= The name of the thing =
... is hard to determine. We need a sounding name instead of saying "that thing formerly known as macro, parser or processor".

Until we have a better idea, we name it just "PPM" - thanks to FlorianFesti :)

== Discussion ==

How will be the PPM Syntax? It should be something that could live in a future WikitextStandard...

{{{
{{ExtensionName(arg[, arg]*)}}

{{ExtensionName(arg[, arg]*)
line
line
line
}}
}}}

On API level it could be just:
{{{
def FormerMacro(request, args, **kw)

def FormerParser(request, args, **kw)

with the Parser just getting a kw arg of lines with the text lines it should process.
}}}

More like:
{{{#!python numbers=none
class ExtensionMacro:
    def __init__(self, raw, request, **kw):
        # raw is empty!
        ....

class ExtensionParser:
    def __init__(self, raw, request, **kw):
        # raw has the lines!
        ....
}}}

But even better would be:
{{{#!python numbers=none
class Macro(PluginBase):
    # __init__ is inherited and does all parameter parsing automagically
    # only custom stuff needs a overriden __init__
    def format(self, formatter):
        # do stuff
        ....

class Parser(PluginBase):
    ....
}}}

-- OliverGraf <<DateTime(2004-10-29T08:54:11Z)>>

== PluginBase Class ==

{{{#!python numbers=off
class PluginBase:

    # file extensions this plugin may apply to...
    _extensions = None
    # mimetypes this extension may apply to...
    _mimetypes = None
    # they are either None (== no parser like extension)
    # or a list of strings (extensions or mimetype globs)

    # what dependencies (caching) does this plugin have?
    _dependencies = ["time"]

    # how to parse the arguments for this plugin?
    _parameters = None

    def __init__(self, _request, _lines, **kw):
        # underscore, so we won't collide with kwargs
        self._request = request
        self._lines = lines
        # just save the kwargs as self attributes
        for key in kw:
            setattr(self, key, kw[key])

    def help(self):
        ''' Describe what your plugin can do. 

        The base class can have code to produce automatic help from 
        available instance attribtues.
        '''
}}}

== Plugin Protocol ==

Instead of defining calls that raise, or return nothing, we can define a informal protocol. Plugins that want to answer this calls from the parser will implement them. We can do the same by puting these methods in the base class, but that can make it harder when the base class return something that you don't want. If we define {{{raise NotImplementedError}}} for each such method, the poor developer that sub class this will have to override ALL methods which she does not care about, just to remove that raise.

  That was an example. If a raise or a return value is the best has to be determined. But defining a base class which is not a base class but only a documentation is kind of stupid. Why do we use a object oriented language at all, if we don't use its features? -- OliverGraf <<DateTime(2004-11-05T09:44:44Z)>>

   This was only an example. You could define the protocol without a class, its just a lilst of methods. ObjC has a formal protocol syntax, but many protocols are not formal, and just describe as interface. Formal protocols are used when time can be saved by testing if an object conform to a protocol, instead of checking for each method separately. Informal protocols used when object might implement some methods, and we don't care what. Python lack the concept of "protocol" or "interface", but its versatile enough to use the concept.

In the case of help(), its is usefull if the base class can produce some automatic help for you, base on your methods docstrings and names, so its better to have this in a base class. Same for automatic parsing code of arguments etc.

By using a protocol, any object can be a plugin. parser and plugins just have to use the same protocol, they don't care what is the object type, just what protocol methods it likes to implement.

{{{#!python
""" Draft for Informal protocol for plugins.

When you write your plugin, copy and pase the methods
prototoyes from here.

This protocol might be extended in future versions. You don't have
to implement any of these mehthods - just what make sense for your
plugin.
"""

def executeInParagraph(self, formatter):
    """ What the plugin should do in a pargragrph context  
    
    Example: 
        This is text [[ThisMacro]] more text

    In this context, the parser will call you with
    executeInParagraph(). If you don't implement this, the
    parser will ignore you. You can return an error message
    if it does not make sense to put the macro in this
    context.

    @rtype: unicode
    @return: formatted content
    """
    
def executeInPage(self, formatter):
    """ What the plugin should do in a page context  
    
    Example: 
        This is a wikii paragraph 

        [[ThisMacro]]

        Another paragraph...

    In this context, the parser will call you with
    executeInPage(). If you don't implement this, the
    parser will ignore you. You can return an error message
    if it does not make sense to put the macro in this
    context.

    @rtype: unicode
    @return: formatted content
   """
}}}

How the parser can use such object?

{{{#!python
def _repl_plugin(self, text):
    # Get plugin instance (using oliver/fabi parser and import plugin)
    plugin = self._getPlugin(text)
    if in_p:
        execute = getattr(plugin, 'executeInParagrph', None)    
    else:
        execute = getattr(plugin, 'executeInPage', None)

    if execute:

            # the parser does not have to open a pargraph or div or
            # whatever for a plugin. The plugin writer is responsible for
            # correct html element in either a page context or paragraph
            # context.
            
            output = execute(self.formatter)
            self.request.write(output)
    else:
        # Maybe error message, or plugin markup in red, or just ignore
}}}

== Can we add this in 1.3? ==

'''In short words: No'''

== How do we add PPM in 1.4? ==

Use `{{` and `}}` for PPM, so we can 
 * either have the old stuff as compability options, to be removed in 1.5.
 * or have an automatic markup converter, migrating the content

''Not providing a converter will let people go crazy who have a pre-1.4 wiki (like this one) and want to use 1.5. And writing one should not be difficult anyway. Macro and parser calls are easy to spot.''

=== markup conversion ===
We should have some markup conversion rules ready for a mig script:
{{{
old syntax (parser, processor or macro)
----
new syntax (PPM)
}}}

{{{
{ { { plaintext } } }
----
`plaintext`    # hmm, does this line show we need 2 quoting methods anyway?
                     # needs backtick_meta = 1 hardcoded
}}}

{{{
{ { {
... pre section ...
} } }
----
{{       # no specification = default to Parser(text/plain, addpre=1)
... pre section ...
}}
}}}

{{{
{ { {#!python
... py src
} } }
----
{{#!python        # or PythonParser !? - shortcut for Parser(python) / Parser(text/x-python)
... py src ...
}}
}}}

{{{
[[BR]]
----
{{BR}}    # implicitely this is BR()
}}}

{{{
[[OtherMacro(arg1,arg2)]]
----
{{OtherMacro(arg1,arg2)}}
}}}

==== Syntactic sugar ====
Some parsors/processors have its own arguments. For example [[ParserMarket/bibtex|bibtex]] in ParserMarket and SimpleTable in ProcessorMarket.

{{{
{‏{‏{!#SimpleTable 4
... contents ...
}‏}‏}
----
{{SimpleTable 4
... contents ...
}}
}}}

Or we can allow another syntax for arguments.
{{{
{{SimpleTable(4)
... contents ...
}}
}}}

We can extend this idea to macros.
{{{
[[OtherMacro(arg1,arg2)]]
----
{{OtherMacro arg1 arg2}}

or

{{OtherMacro(arg1,arg2)}}
}}}

And for arguments, we can design an escape mechanism.
{{{
{{Macro1(arg1,second arg with space)}}
{{Macro2 arg1 second,arg,with,comma}}
{{Macro3("arg 1", "arg 2, with comma")}}
}}}

  I think we should always go for quoted argument lists. My new OliverGraf/ArgumentParser already supports python like argument lists, with some extensions for bools. -- OliverGraf <<DateTime(2005-03-11T06:55:27Z)>>

If we do quoted argument lists, then lets use SGML like attributes, which are easier to use then positional arguments, better known then Python conventions, and will allow us to merge css attributes naturally, e.g. let us set a css class to a wiki element.
{{{
{{TableOfContents level="2" class="sidebar"}}
{{ParserName line-numbers="on" class="folding"
Text...
}}
}}}

 Would make argument lists longer, but it is still well defined, so I have no Problem with this. Making just the OP(,) in the parser optional would accept SGML like attribute lists as well as python style args including positional arguments, but I think allowing this mix would result in more confusion. So we need to settle for one syntax. -- OliverGraf <<DateTime(2005-03-11T11:32:50Z)>>

 In any case the syntax should not forbid using `"` or `'` in the arguments like the search engine currently does. -- AlexanderSchremmer <<DateTime(2005-03-11T16:27:36Z)>>

== {{ }} for "include"? ==
While looking for new ideas on another wiki, I found that they use `{{imgage.png}}` for showing ("including") images.

I thought that `{{ItemName or URL}}` could in general mean "include the content from that place here", while `[[ItemName or URL]]` means "link to that content".

That conflicts with usage of `{{text}}` for plain / pre / verbatim text (see section above), but maybe we could just define that default to that "include" processing and choose something more verbose for verbatim.

On first look it also isn't quite intuitive why "include" is similar to PPM (or why it is even the default of it), but on the second look one notices, that this just means "include the output of that PPM here" or "include the output we get when doing default processing on the mimetype we find there", so this is ''not'' that far-fetched.

(HeatherStern here) hmm.  I am for ideas that make it easier, as a wiki-user rather than coder, to remember which markup is which.  The meme that [[ means hotlinks and {{ means included objects doesn't seem so bad.  We could see: `[[newlocalpage]]` `[[new page]]` `[[picture.png]]` (ooh, would it even demand the attachment for me?) `[[otherwiki:TheirPage optional plaintext]]` `[[:SomePage alternate text for local link]]` (the presumptive : meaning wiki, but no wikiname, so it must be this wiki, right?  maybe this could be used to invoke looking for whatever subpage it is really).  {{ could mean basically the same thing but in included rather than hotlinked form, so those would in order be... `{{unknown text, must want to be pre?}}` `{{include text of wikipage}}` `{{actually display the image, demanding it as attach if needed}}` `{{including text from an otherwiki, after some defensive transformations}}` ... you get the idea.
It would be further easy to think of {{ as taking single character "parameters" so that `{{{` possibly used for extensions, simply means to the human wikizen, "an included thing, that had code do some of the work".  Other {{ or [[ thingies are easily imagined then, e.g. `[[*icon* link]]`  mark the link special somehow, `{{| big box around my inclusion |}}` `{{[lang] run parser on my attachment for display }}` ... ok, so these are not consistent where the "end mark" of the inner parameter is, but the human sees the usability much better than cumbersome `{{Include(pagename)}}` `{{Parser(language,linkname)}}` -- why should wikizens need to know what we call these thing?  They should be able to concentrate on the effects.

The markup must be clear and handle all the hard cases, like page or file names with spaces. Therefore, {{:pagename label}} does not work. {{ThisName}} is also not clear; does it run ThisName macro, or include ThisName page content, or include ThisName image?

 This is why I have said in IRC we must have precedence for operators/operations -- Heather Stern
  * {{: taking the rest of its content one way, we could have a different
  * {{= maybe, which takes the whole item as a page name no matter how many spaces.

   '{{:', '{{=', '{{!' or '{{$' is not clear markup, unless you code in perl.

 Remember that part of this is how code behaves while part of this is how humans will behave when they see it!  To code, {{: and {{= are not the same thing.  They might use a same baseclass, perhaps.
 
  Exactly, people should use clear and consistent markup. The stuff they use many times, like page names, should be short and easy to type, the stuff they use less can be more verbose.

All markup can use the same Smalltalk-like syntax:
{{{
{{object:value variable:value}}
}}}

Which happen to be like our acl syntax:
{{{
#acl: Name:rightlist AnotherName:rightlist
}}}

And like our search syntax:
{{{
title:This regex:That AnotherTerm
}}}

Where object can be ''http'', ''page'', ''image'', and ''plugin'' which are a Renderer, that render the contents of the resource in the page. All of them can be implemented as macros which can be overridden by private wiki macros. So, if someone want to add custom icons for http links, he can simply use its own dancing and singing http renderer.

 making wandering wikizens have to know all those things about how it works under the hood is crazy.  We should want them to type less and concentrate on their real topic, not on how moin works! -- HeatherStern
 
  Why do  you think the users have to know about that? they will continue to use http://example.com just like they did before, they will not have any idea that http is just a plugin like UserPreferences. This stuff is important only to the people who want to customize moin to their needs, without any effort on our side.

Here how common markup will look like:
 * `{{http://example.com/image.png title:Example com amazing image}}` - an image with tooltip
 * `{{page:Page Name}}` - include all page content
 * `[[page:Page Name]]` - link to local page, equivalent to `[[Page Name]]`
 * `[[page:Page Name title:Different Title]]` - link with title
 * `[[page:Page Name image:Image.png]]` - link with an image
 * `[[page:Page Name title:Different Title image:Image.png]]` - link with an image and text
 * `{{page:Page Name from:"^= Here =$" to:"^= There =$"}}` - include part of a page
 * `{{wiki:WikiName/PageName}}` - include a page from another wiki
 * `[[wiki:WikiName/PageName]]` - link to a page from another wiki (current markup)
  * both confusable with Page/Subpage syntax
  * or `[[wiki:WikiName page:PageName]]` - consistent, little more typing
  * or `[[wiki:WikiName:PageName]]` - without page: key, little strange.
  * or just use a URL like scheme: `wiki://WikiName/Page/SubPage`
   * Advantages: easy to parse, a well known syntax, no confusion anywhere
    * But then why page: and plugin: does not use //? It would be very strange to use plugin://name...
 * `{{plugin:UserPrefernces}}` - macro call
 * `{{plugin:Python lineNumbers:off \n code...}}` - code area using python parser
 
We can use one namespace as the default, maybe the pages namespace, as its the most used namespace in a wiki. Variables names can be omitted to save typing: `[[:Page Name :Different Title]]` - this syntax already exists: `[:Page Name :Different Title]` -> [[Page Name |Different Title]].

If attached image is just like a regular page, for example, saved as SomePage/ImageName.png, then including an image is just `{{SomePage/ImageName.png}}`, and a link to that image is `[[SomePage/ImageName.png]]`

This will simplify the parser, as there will be no more magic words to parse. Only `text`, `''text style''`, `= heading =`, ` * list`, ` #. ordered list`, `[[link]]` and `{{include}}`. Both include and link will use same parser for the content.

 (!) Also rename `MoinMoin` to `Moin:Moin` :)

I do not like it because it feels kinda bloated. `{{plugin:BR}}` It rather looks like a parser simplification than a syntax simplification because the long prefixed macro/parser calls are not very intuitive.

The name PPM does describe the techniques are used to write the code but normally I use the words ''extension'' or ''plugin'' or ''module'' to describe someone who does like to use the wiki and its PPMs. If the code is unified we should use in my eyes a common name like firefox and others are doing.
Just call it ''extension'' -- ReimarBauer <<DateTime(2005-11-19T07:25:54Z)>>
 The problem just calling it "extension" is that this is too generic. This rather describes the class of this thing, but not what it exactly does. And there are other types of extensions/plugins: actions, themes, maybe soon languages,  server adaptors (Request*), authentication plugins, ...

----
Just an other idea for the syntax invoking a parser or macro call. {{{
{[ImageLink(arguments)]}
}}}

or 
{{{
{[python(arguments)
text
]}
}}}
Then it is pretty clear that it is a new synatx and for some time the old syntax could be used for comtibility issues. In our cases we may be won't be able to update all wikiservers at the same time. -- ReimarBauer <<DateTime(2006-09-05T10:16:33Z)>>
