UnifyPagesAndAttachments/Discussion

Please read the toplevel page and this page completely before commenting.

Not doable (performance): direct links to raw attachment file as default

Some users want to directly start the target item download when clicking on a link.

This will likely be possible, but it won't be the default, but needing some link parameter like [[target|label|&do=get]].

[Note: I had this problem (two step download of attachments, especially pdf, is confusing) and it took a lot of effort to find the &do=get solution, which is OK for me. There must be a better way to advertise this issue - I will put a question up pointing to here as a first step -- BoydBlackwell 2009-06-12 07:24:49]

Please improve HelpOnLinking e.g. add the example. A starting point is to read EditingOnMoinMaster. And finally remove the note from this page. -- ReimarBauer 2009-06-12 08:56:13

Reasons why &do=get can not be the default after we unified pages and attachments to items:

For a target item of type text/x-moin-wiki (wiki page), we obviously don't want to trigger the download of the page text, but get a rendered view of it as usual. Same thing applies for quite some other target mimetypes for which we have parsers (other markups, programming languages, diffs, CSV, etc.).

For some other target mimetypes, a direct download could make sense (like some word document for example, it is unlikely that we can ever render this). BUT: When rendering a page, we encounter quite some links (in navibar, in page content) and as we want to be rather fast, we have to be a bit careful about not needing to read too many target item metadata to do that.

Thus, if we would make the link target URL dependant on the link target mimetype, we would have to find out the mimetype of the target.

There are 2 ways to find out a target item mimetype:

use the file name extension. Fast, but has many problems like:
- some files don't have an extension: e.g. Makefile, or README or hosts or ...
  - you maybe don't want to add a .txt extension just to get the handling of these right
- some extensions have multiple mimetypes using this extension (like .dot is used by Graphviz files as well as by MS Word).
- on some systems, the system's mimetype map (e.g. /etc/mimes.type) is far from perfect. Can give headaches if there is no way for a wiki user to fix the mimetype if the magic goes wrong.
we just store the mimetype of the item in its metadata (code for doing this is already implemented in new storage backend)
- it can be either given on upload or autodetected once from extension if not given
- in can be changed in case autodetection chose the wrong type
- it is slow as we need to read item metadata to get this information
  - this doesn't matter for choosing the correct renderer for an item based on its mimetype [1 access and likely you need more/other metadata anyway]
  - but it does matter if you need it for rendering links to items [>20 accesses depending on page size])

Thus, we better avoid to need to know the target item mimetype for link rendering.

Thus, we can't do magic - all links need to be made in the same way.

Thus, this feature can't be implemented.

Not doable (user interface): direct links to raw attachment file as default

Assuming we get above "direct link / performance" stuff solved somehow, the next problem would be to decide what targets are linked to using the "show" url and what targets are linked to using "raw" url.

Problem: this depends on user agent's (browser's) capability and on what the user wants to do.

Note: it not necessarily depends on what wiki admin wants or page author wants.

Also, even if wiki admin or page author assumes that user just wants to download those file types, how do you rename, delete, revert a file?

If you just have a link that immediately starts a download, there is no way to show the user interface offering those actions for the target item.

Note: we likely won't have AttachFile-like managing code because attachments won't be contained in a page any more - thus the only "listing operation" we can do is "list all sub-items" (looking at their names). This operation will output "sub pages" as well as "sub files" (both are just sub items of different mimetypes).

Not wanted: directly serve files by the webserver

We had that already (until it was removed after years of deprecation in 1.7 or so).

It doesn't solve any problem, but creates some:

if wiki admins are not very careful, this can be used to upload exploit code (e.g. exploit.php). if files are directly served by apache and the wiki admin forgot to disallow php / allowed it accidentally, a user will be able to upload code and run it by just accessing that file.
if the webserver directly accesses files, there is no way for moin to implement ACLs
there is no way to implement non-filesystem storage (see our new storage backend)
there is no easy way to implement revisions
it makes the code more complex
it makes configuration more complex
I disagree that it doesn't solve any problems. It exactly solves the usability problems mentioned below. As for the problems it creates, I agree with most of these, but still feel it should be an option for admins to use if they are willing to accept the drawbacks. For example, some people know how to avoid uploaded code exploits, don't use ACLs and do use the host filesystem for storage. On one site I manage, I have people upload their files as MoinMoin attachments, then manually move them to a download area, and change the urls on the page to be direct URLs. Now that I'm thinking about it, maybe I should just automate this process and forego MoinMoin's attachment system. -- TimBird

??: Link box

Maybe the theme code can solve this dilemma by having a "Link box" that has links for every link target used on that page:

       vvvvvvvvvvv link text, linked to rendering of target item
+---------------------+
| [DL] [OtherPage]    |
| [DL] [/SubPage]     |
| [DL] [/foo.doc]     |
| ...                 |
+---------------------+
  ^^^^ download/raw icon, linked to raw download of target item

if you don't like it, just use another theme.
it does not make too much sense if target is an item people usually want to see rendered (wiki text, programming language text, maybe images, ...). But we don't want to look at target mimetype, thus we would have quite some icons with rather useless links.
minor problem: this won't easily work with interwiki links because we don't know the target wiki software so we can't know the raw/dl URL. That problem is a rather generic one and also true for non-ascii interwiki pages and other stuff.

??: subitems box

Similar to above, just listing all sub items.

would list all sub pages and sub files (just sorted by name)

??: Slightly modified inline link rendering

We currently decorate all non-intrawiki links with a link item (globe icon for http links, disk icon for ftp links, terminal icon for ssh, interwiki icon for interwiki links, etc.).

We do no use decoration for intra-wiki links, but we could:

               vvvvvvvvvv linked to rendering of target item
This is a [DL] [LinkText] example.
          ^^^^ download/raw icon, linked to raw download of target item

the decoration icons are currently rendered using css :before, there is no html for them. If we re-introduce doing it by extra html (we had that in old moins), it will cause some trouble to the gui editor converter (could be solved by not doing it when feeding gui editor), maybe line wrapping problems (icon at end of line n, link at beginning of line n+1), etc.
Not sure if that makes everybody happy, because of the icons interrupting flow of reading and some people like to remove the icons because of this.
Users would have to learn that it sometimes makes a difference whether they click on

User Expectations

Generally, when users see a link to something that looks like a file name, they expect to be able to use "standard" (ie, whatever their browser does) download functionality - click to download, right click and "save as" are common cases. The default behaviour proposed breaks this expectation (if I link to, say, a Python script, right click and "save as" won't download it, but rather a HTML page with a highlighted version).

Even if the default [[MyScript.py]] link points to the rendered page (I see the technical arguments in favour of this, but I'm not convinced yet that they outweigh the UI issues), then there needs to be a very simple and discoverable format for links which do go to the raw content ([[MyScript.py|MyScript.py|&do=get]] is neither intuitive nor discoverable). At the very least, a standard macro <<Download(MyScript.py)>> which renders exactly the same as [[MyScript.py|MyScript.py|&do=get]] would, should be provided. -- -- PaulMoore 2008-10-01 14:18:43

I like the idea of having a download macro. This macro could also automatically add the file size so that you get something like this FileXy.py(60.5kB) (At least some accessibility awards demand for this for some unknow reason). Sine page attachments are no longer possible (if I have understood it corretly). Why not have also have something like MacroMarket/AttachTable which mimic file attachments to a page (and when you click on it files a direclty downloaded)? Maybe even merge this with Arnica so that you get thumbnail images for attached images like this: But I haven't thought much about that... Note: The AttachTable-Macro/Parser is not an automatically created link box. It is added to the page / changed by the user and is this only a more complex download macro as PaulMoore suggested. Another question: Why not store file attachments in the metadata of the page? Metadata of a page could also be used to express an item's relations ("page") with other items ("attachments") [see also FeatureRequests/AssociatedPageFeature and FeatureRequests/RethinkingPageOrganization]. So metadata does not only store mimetype, revision number etc. but also semantic information and other stuff like relations to other pages. A metadata "attached files" could trigger another link generation with automatic download, a metadata "associated files" only a show. Don't know. The creation of the linkbox / AttachTable could also be automized based on these meta data, too. -- OliverSiemoneit 2008-10-01 21:23:32

One other usability issue is that the URL used for the link should end in the filename to be downloaded. When doing a "save-as" operation, most browsers will use that as the default filename for the save. On one site I manage, I have lots of files that I need to publish URLs to externally. The current URL syntax is awkward, and makes downloading files difficult. -- Tim

Comments

Just some notes about some ideas I got last night about how to solve that dilemma.

Currently we talk always about the rendered view. May be we can think about to change the link behaviour in edit mode. If it is rendered we can have a download link and in edit mode it changes to the view link. For the gui editor this is I think not too complicated for the text editor we will need some kind of other ui to manage new subitems and the already used subitems.

We can also have an action which changes all the links on the given page. Doing this it does not matter if we have on default get behaviour for the one who likes to download only. Someone who wants to change versions of the items has to enter an action beforehand. That action can be nicely integrated into the theme.

e.g. a stupid one like this

   1 # -*- coding: iso-8859-1 -*-
   2 """
   3     ChangeLink
   4     renames virtual [[attachment:file.ext||&do=get]] to [[attachment:file.ext||&do=view]]
   5 
   6     @copyright: 2008 MoinMoin:ReimarBauer
   7     @license: GNU GPL, see COPYING for details.
   8 """
   9 from MoinMoin import config, wikiutil
  10 from MoinMoin.Page import Page
  11 from MoinMoin.parser.text_moin_wiki import Parser as WikiParser
  12 
  13 def execute(pagename, request):
  14     _ = request.getText
  15     msg = None
  16 
  17     if not request.user.may.read(pagename):
  18         msg = _("You are not allowed to view this page!")
  19 
  20     if msg:
  21         request.theme.add_msg(msg, state)
  22         Page(request, pagename).send_page()
  23         return
  24 
  25     request.formatter.page = Page(request, pagename)
  26     raw = Page(request, pagename).get_raw_body()
  27     # of course we need some regex from the parser here to handle attachment markup
  28     # that is just to show the idea
  29     raw = raw.replace('&do=get', '&do=view')
  30 
  31     html = wikiutil.renderText(request, WikiParser, raw)
  32 
  33     mimetype = "text/html"
  34     request.emit_http_headers(["Content-Type: %s; charset=%s" % (mimetype, config.charset)])
  35 
  36     request.theme.send_title(pagename,
  37                              pagename=pagename,
  38                              )
  39 
  40     request.write(request.formatter.startContent("content"))
  41 
  42     request.write(request.formatter.rawHTML(html))
  43     request.write(request.formatter.endContent())
  44     request.write(request.theme.send_footer(pagename))
  45     request.theme.send_closing_html()

ChangeLink.py

We have to think also how to get the item pages if every attachment on a wiki gets added the non default &get attribute.

-- ReimarBauer 2008-10-01 20:21:13

A further thought - if the link to a non-existent page (item) always goes to the same place, does that mean that the current "Create New Page" and attachment management pages will be merged somehow? I'm not sure what such a merged page will look like, but that needs to be very carefully thought out if it's to look sensible to a naive user ("Why am I being asked to upload a file? I just wanted to create a new page"). -- PaulMoore 2008-10-02 14:01:19

Moin could display something like this "Create empty page, create page based on TemplateXY or create page by uploading an exisiting file?". So maybe there is a way to intergrate the load-action in there: You can upload a wikipage, this will create a wikipage. But you can also upload something else (an image). This will create a page displaying the image, offering a download link, displaying metadata (exif...) like it does already. Underneath the above question a box with similar, already existing items could be display. Clicking on "create empty page based on TemplateXY" takes you to another page, where you can choose the page template. May it is also possible to integrate some preview there how a template looks like. That could ease decision for the user. Maybe it is also possible to give templates metadata which describes them. This description could be also displayed. That's also better for choosing. -- OliverSiemoneit 2008-10-02 16:29:13

Paul: good comment! Yes, it will be about creating a new item then.

Because the item does not exist before, we don't know what it will be.

Currently (1.7) we offer to load the page editor with some template page. Looking at the generalization, that would mean for the future to load some template item somehow (not necessarily into some editor).

There has been some troubles/limitations with templates / the editor in the past:

you can't have ACLs on a template page loaded into the editor and save that if you don't have admin rights
if you start from nothing, the wiki editor does not know about what markup you are using, so it can't show correct quickhelp
you maybe could even start the gui editor with non-moin-wiki markup although that is unsupported

So maybe we can solve this by not putting the template content into the editor, but into the initial item revision.

Using template items in this generalized way could be an easy way to initialize content and metadata without demanding too much from the user (we will likely have much more metadata than we have now).

Thus for nonexisting items, we could show:

some item template names (radiobuttons?), would include the usual stuff:
- moin wiki page
- plain text
- binary file
- ... (how many would we have? selecting document type and preload content at once might give many templates)
"create" (submit button)

For advanced users (beginners just use those templates):

a item name input field for an item to clone (ad-hoc templating)
a mimetype input field to create an empty item of that type

Item creation could be decoupled from the first edit / the upload - because it is good to know beforehand what type of thing we want to edit before we try to invoke some editor. Once the item is created, its mimetype metadata will likely have some reasonable value. If we render that (possibly empty) item, the user interface will adapt to that mimetype and offer appropriate actions for that item (like calling text or gui editor for wiki pages, only the text editor for text/creole pages [gui editor does not work for them yet!), and not offer any editor at all for application/zip, but just a upload button).

-- ThomasWaldmann 2008-10-02 16:40:10

Seems like every user of wiki which I administer is mad about this additional page and that it's not possible to do Save As. That's because we store a lot of .exe and .doc files. Got an idea of the following work-around: store somewhere (on some page or in config) a list of file extentions that should be downloaded directly. Then check this list of exceptions when rendering a page and replace 'view' with 'get'. This should do the trick... at least for my use case. -- AlexanderAgibalov 2008-11-18 14:22:02

That doesn't solve the user interface problem (see above). -- ThomasWaldmann 2008-11-18 15:42:50
- Well, it's still possible to delete an attachment through the AttachFile action page. What are the chances that this action will be removed in future versions (and when)? While there's no other better (and easy to implement) solution, I think I will have to stick to this one in my local Wiki: we can sacrifice such features as file versioning and renaming in favor of having direct links. -- AlexanderAgibalov 2008-11-19 06:46:42

With the next storage api everything in moin becomes an item.

a page is an item
an attachment is an item
an user is an item.

items are stored dependent on their mimetype For further details see UnifyPagesAndAttachments

ReimarBauer: Thinks about browsers and items.

Browsers can't display every mimetype item. A user wants items to be displayed and not only downloaded. It is not expected that 3rd party browser plugins will solve this on any browser. The item needs to be displayed in the right context and not in an external tool. moin is a wiki software and not a file serving application. There are only a few builtin types for browsers e.g. some image types (png, gif, jpg) and markup (html) and plain (text) and since short time Scalable Vector Graphics (SVG). 3rd party browser plugins were written for various other mimetypes, If you look at /etc/mime-types "that" is a long list. And some of these are used also on the web.

ReimarBauer: Thinks about "What can we do?"

looks like we have to decide which type could be displayed in the context of the page. Only these can be used by transclusion syntax. All others can be only downloaded. Can we currently force users to choose the right mimetype if they want it used in the context of their wiki page without downloading it first? Currently we don't. I am not sure if we can at all. May be rather not. Only a few will get that we are between the devil and the deep blue sea. Sure we can, but did they like it?

ReimarBauer: "So what can we do?"

separate mimetypes into viewable (for translusion) and downloadable? converting downloadable mimetypes into viewable ones (snapshots) ? adding a placeholder (snapshot) into the page where the downloadable mimetype would be expected to be viewable ?

ReimarBauer: "Where can we look for converting?"

MoinMoin: UnifyPagesAndAttachments/Discussion