FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum » FUDforum Suggestions » Detecting search engine bots
Show: Today's Messages :: Unread Messages :: Show Polls :: Message Navigator
| Subscribe to topic | Bookmark topic 
Switch to threaded view of this topic Create a new topic Submit Reply
Detecting search engine bots [message #12094] Mon, 28 July 2003 12:38 Go to next message
jrobbio is currently offline  jrobbio   United Kingdom
Messages: 32
Registered: June 2003
Karma: 0
Member
add to buddy list
ignore all messages by this user
search engine bots can be a real problem in that they appear as many users at the same time. What I suggest is to start associating the ip's that search engines like google, FAST, inktomi etc come from so that we can see where they appear. It would be cool to have some sort of tracker in the admin section so we can see which ones have been picked up and those that have been missed.

Rob
Re: Detecting search engine bots [message #12095 is a reply to message #12094] Mon, 28 July 2003 12:55 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Associating IPs is a bad practice because often, different people using the same proxy will appear as the same user. As for search engines there are sites that have ip ranges of the various spiders used by search engines and similar. If you don't want them to visit your site simply use the FUDforum's IP filter to deny them access.

FUDforum Core Developer
Re: Detecting search engine bots [message #12096 is a reply to message #12094] Mon, 28 July 2003 13:05 Go to previous messageGo to next message
jrobbio is currently offline  jrobbio   United Kingdom
Messages: 32
Registered: June 2003
Karma: 0
Member
add to buddy list
ignore all messages by this user
I don't want to deny them access, I just want them to appear in my userlist as Google, Inktomi etc etc.

The new IPB has some interesting additions where you can do this. Its also possible to remove the session ID for them so that indexing works properly.

Rob
Re: Detecting search engine bots [message #12099 is a reply to message #12096] Mon, 28 July 2003 13:12 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
IPB, what's that?

FUDforum's sessions are nonvolatile and do not prevent search engine parsing. You can also make forum work without using URL sessions. Plus I really see no harm as having a search engine appear as multiple anonymous users during. To spend time looking ips to group search engines would be too performance inhibitive and not bare any worth while advantages.


FUDforum Core Developer
Re: Detecting search engine bots [message #12102 is a reply to message #12094] Mon, 28 July 2003 13:42 Go to previous messageGo to next message
jrobbio is currently offline  jrobbio   United Kingdom
Messages: 32
Registered: June 2003
Karma: 0
Member
add to buddy list
ignore all messages by this user
IPB = Invision Power Board.

There's no harm in a search engine bot appearing as anonymous, but I know some people don't like them being accounted for in the users online section and others do.

Rob
Re: Detecting search engine bots [message #12103 is a reply to message #12102] Mon, 28 July 2003 13:56 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Like I said it is a minor annoyance if even that, but the cost of a solution which would affect every request is just too much.

FUDforum Core Developer
Re: Detecting search engine bots [message #12105 is a reply to message #12094] Mon, 28 July 2003 14:13 Go to previous messageGo to next message
JamesS is currently offline  JamesS   United States
Messages: 275
Registered: July 2002
Location: Atlanta, GA
Karma: 0
Senior Member
add to buddy list
ignore all messages by this user
You would probably be better off denying bots access to the forum directory any way. It causes undue stress and slows things down (even if marginal)

`echo "Disallow: /forum" >> /webroot/robots.txt`
Re: Detecting search engine bots [message #12106 is a reply to message #12105] Mon, 28 July 2003 14:20 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
That is true, even faster would be to block robots via iptables or firewall Smile.
But I think the goal here was not to block robots but merge them into a single user rather making it appear as many users.


FUDforum Core Developer
Re: Detecting search engine bots [message #12108 is a reply to message #12105] Mon, 28 July 2003 15:39 Go to previous messageGo to next message
jrobbio is currently offline  jrobbio   United Kingdom
Messages: 32
Registered: June 2003
Karma: 0
Member
add to buddy list
ignore all messages by this user
JamesS wrote on Mon, 28 July 2003 19:13

You would probably be better off denying bots access to the forum directory any way. It causes undue stress and slows things down (even if marginal)

`echo "Disallow: /forum" >> /webroot/robots.txt`



Your obviously not into people actually finding content on your forums then.

Rob
Re: Detecting search engine bots [message #12109 is a reply to message #12108] Mon, 28 July 2003 15:55 Go to previous messageGo to next message
Ilia is currently offline  Ilia   Canada
Messages: 13241
Registered: January 2002
Karma: 0
Senior Member
Administrator
Core Developer
add to buddy list
ignore all messages by this user
Actually even though your pages maybe indexed more often then not they are not included in search results or found near the very bottom of the search results. This is because search engine generally tend to ignore URLs with parameters, since those could be used to easily be used as a search engine trap.

FUDforum Core Developer
Re: Detecting search engine bots [message #12121 is a reply to message #12094] Mon, 28 July 2003 22:34 Go to previous message
jrobbio is currently offline  jrobbio   United Kingdom
Messages: 32
Registered: June 2003
Karma: 0
Member
add to buddy list
ignore all messages by this user
Which is why I wanted to use the path_info setting as on with session ID's as off. Since the path_info was buggy on my particular site I am looking forward to 2.51 so I can have it fully functioning.

As for forums at the bottom of the SERPS, you should tell that to my friend who has his Vbulletin forum at the top for loads of things because of the way he dealt with the search engine bots on the forum.

Rob
Quick Reply
Formatting Tools:   
  Switch to threaded view of this topic Create a new topic
Previous Topic: Members list
Next Topic: french community for FUDforum
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Mon Nov 18 12:31:40 EST 2019

Total time taken to generate the page: 0.00824 seconds