FUDforum: How To » robots.txt for path-info

Home » FUDforum » How To » robots.txt for path-info (

) 1 Vote

Show: Today's Messages :: Polls :: Message Navigator

Re: robots.txt for path-info [message #31535 is a reply to message #26997]

Thu, 04 May 2006 14:27

matthieu_phpmv

Messages: 44
Registered: November 2004

Karma:

Member

Here's mine on phpmyvisites.net/robots.txt

Quote:

User-agent: *
Disallow: /forums/pdf.php
Disallow: /forums/index.php/m
Disallow: /forums/index.php/sp/
Disallow: /forums/index.php/ef/
Disallow: /forums/index.php/mv/
Disallow: /forums/index.php/r/
Disallow: /forums/index.php/pmm/
Disallow: /forums/index.php/rm/
Disallow: /forums/index.php/sel/
Disallow: /forums/index.php/pv/
Disallow: /forums/index.php/ma/
Disallow: /forums/index.php/u/
Disallow: /forums/index.php/s/
Disallow: /forums/index.php/h/
Disallow: /forums/index.php/i/
Disallow: /forums/index.php/l/
Disallow: /forums/rdf.php

With this file bots will ONLY spider the messages, it is what I want. (no profile, no pdf messages, etc. only real and uniques message)

[Updated on: Fri, 05 May 2006 12:09]

Report message to a moderator

[Message index]

		robots.txt for path-info By: !alex on Wed, 24 August 2005 12:59
		Re: robots.txt for path-info By: matthieu_phpmv on Thu, 04 May 2006 14:27

Previous Topic:	Change Forum display style
Next Topic:	Understanding templates and themes better

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Sun Nov 24 22:37:10 GMT 2024

Total time taken to generate the page: 0.03856 seconds