Make it possible to specify which pages have html_head_index in the robots meta tag. By default html_head_index is index, follow and is set only for page_front_page and html_head_index.
I am trying to create a page that lists the site's attachments so a webcrawler can index both page and attachment content. I modified the TitleIndex macro to create an AttachmentIndex macro by inserting the output of the AttachList macro for each page. Creating a new page that includes this macro, however, doesn't let the webcrawler index the attachments. If I'm reading the code correctly (wikiutil.py) only uses an index, follow tag for page_front_page and page_title_index. A page that lists the pages with the index, follow tag would make it easier to customize how a site was index. The workaround for now is just to use the TitleIndex page.
This feature would only be beneficial to those who were interested in controlling how their site is crawled.
- Check if you can't do this with a #pragma.
I have tried #pragma robots index,follow but that does not work. There are the following options. You can change html_head_normal to index,follow but this would change all pages to follow.
html_head_queries = '''<meta name="robots" content="noindex,nofollow">\n''' html_head_posts = '''<meta name="robots" content="noindex,nofollow">\n''' html_head_index = '''<meta name="robots" content="index,follow">\n''' html_head_normal = '''<meta name="robots" content="index,nofollow">\n'''-- PascalRosin 2009-05-06 10:14:11