FUDforum: FUDforum Suggestions » Including a default robots.txt file

Home » FUDforum » FUDforum Suggestions » Including a default robots.txt file

Show: Today's Messages :: Unread Messages :: Polls :: Message Navigator
| Subscribe to topic | Bookmark topic

Including a default robots.txt file [message #186582]

Fri, 03 October 2014 17:18

cpreston

Messages: 160
Registered: June 2012
Location: Oceanside

Karma: 6

Senior Member

remove from buddy list
ignore all messages by this user

It seems to me that if you want to be seo-friendly, you use the path_info_seo template set, and you use the robots.txt settings included in the FAQ.

If that's the case, what about including in the install a default robots.txt file like the one below that would work for anyone whether or not they turned on URL rewriting or not?

This is just like Joomla where they include an htacess.txt file. All you have to do is rename it to .htaccess and you're good to go.

#This following code can be appended to your site's robots.txt file if you are
#using a theme based on the PATH_INFO or #PATH_INFO_SEO template set.
#It will prevent search engine bots from finding multiple instances of your
#posts.  

#Note1: You need to change /forum/ to the subdirectory where you forum is located
#Note2: Your site's robots.txt file is in the site's root directory, not the subdirectory
#where you installed FUDforum

User-agent: *
Crawl-Delay: 2
#Default PATH_INFO and PATH_INFO_SEO URLs:
Disallow: /forum/index.php/a/
Disallow: /forum/index.php/bl/
Disallow: /forum/index.php/d/
Disallow: /forum/index.php/ef/
Disallow: /forum/index.php/gm/
Disallow: /forum/index.php/h/
Disallow: /forum/index.php/i/
Disallow: /forum/index.php/il/
Disallow: /forum/index.php/ip/
Disallow: /forum/index.php/l/
Disallow: /forum/index.php/lk/
Disallow: /forum/index.php/m/
Disallow: /forum/index.php/ma/
Disallow: /forum/index.php/mar/
Disallow: /forum/index.php/ml/
Disallow: /forum/index.php/mn/
Disallow: /forum/index.php/mv/
Disallow: /forum/index.php/ot/
Disallow: /forum/index.php/pdm/
Disallow: /forum/index.php/pl/
Disallow: /forum/index.php/pmm/
Disallow: /forum/index.php/r/
Disallow: /forum/index.php/re/
Disallow: /forum/index.php/rm/
Disallow: /forum/index.php/s/
Disallow: /forum/index.php/sel/
Disallow: /forum/index.php/st/
Disallow: /forum/index.php/tt/
Disallow: /forum/index.php/u/
Disallow: /forum/index.php/uc/
#PATH_INFO and PATH_INFO_SEO URLs if you remove the index.php from the URL
Disallow: /forum/a/
Disallow: /forum/bl/
Disallow: /forum/d/
Disallow: /forum/ef/
Disallow: /forum/gm/
Disallow: /forum/h/
Disallow: /forum/i/
Disallow: /forum/il/
Disallow: /forum/ip/
Disallow: /forum/l/
Disallow: /forum/lk/
Disallow: /forum/m/
Disallow: /forum/ma/
Disallow: /forum/mar/
Disallow: /forum/ml/
Disallow: /forum/mn/
Disallow: /forum/mv/
Disallow: /forum/ot/
Disallow: /forum/pdm/
Disallow: /forum/pl/
Disallow: /forum/pmm/
Disallow: /forum/r/
Disallow: /forum/re/
Disallow: /forum/rm/
Disallow: /forum/s/
Disallow: /forum/sel/
Disallow: /forum/st/
Disallow: /forum/tt/
Disallow: /forum/u/
Disallow: /forum/uc/

Report message to a moderator

Rate author: +1 -1

Re: Including a default robots.txt file [message #186617 is a reply to message #186582]

Sun, 05 October 2014 05:16

naudefj

Messages: 3771
Registered: December 2004

Karma: 27

Senior Member
Administrator
Core Developer

remove from buddy list
ignore all messages by this user

FUDforum ships with a .htaccess file.
See http://sourceforge.net/p/fudforum/code/HEAD/tree/trunk/install/www_root/

Do we add the above declarations to the same file (as comments), a separate file, or would it be sufficient to leave it on the wiki?

Report message to a moderator

Rate author: +1 -1

Re: Including a default robots.txt file [message #186620 is a reply to message #186617]

Sun, 05 October 2014 20:08

cpreston

Messages: 160
Registered: June 2012
Location: Oceanside

Karma: 6

Senior Member

remove from buddy list
ignore all messages by this user

The above code should go in a robots.txt file, not the .htaccess file. So I was suggesting putting a robots.txt file that they could move into their server's root directory to activate. In addition, I think it would be really nice if we added some comments to the read more or even some code that pops up so that if you use the path_info template set, you get told that you could/should probably use this robots.txt file. (I think it's important enough that it should be part of the install process is what I'm saying.)

(I referenced the .htaccess file as an example of how some products include default files that aren't actually used until you rename them or cut/paste their contents into another file.)

Report message to a moderator

Rate author: +1 -1