FUDforum
Fast Uncompromising Discussions. FUDforum will get your users talking.

Home » FUDforum » How To » Spiders, Bots and Crawlers - Problem
Show: Today's Messages :: Unread Messages :: Show Polls :: Message Navigator
| Subscribe to topic | Bookmark topic 
Switch to threaded view of this topic Create a new topic Submit Reply
Spiders, Bots and Crawlers - Problem [message #169333] Thu, 26 September 2013 06:54 Go to next message
danliker is currently offline  danliker   Switzerland
Messages: 57
Registered: January 2004
Location: Schweiz
Karma: 0
Member
add to buddy list
ignore all messages by this user
I have added Some Crawlers under "Spiders, Bots and Crawlers" also i have created
a group for them to only have rights to read / show. Then i selected auto-login.

But it looks like they get not logged in, are there some more options that needs to be changed to get it work?

Re: Spiders, Bots and Crawlers - Problem [message #169334 is a reply to message #169333] Thu, 26 September 2013 15:47 Go to previous messageGo to next message
naudefj is currently offline  naudefj   
Messages: 3624
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
remove from buddy list
ignore all messages by this user
The feature is working as you can see on this forum.
The user agent string must match within the first 32 characters (will be 64 chars in 3.0.5).
Aw: Spiders, Bots and Crawlers - Problem [message #169336 is a reply to message #169333] Fri, 27 September 2013 04:30 Go to previous messageGo to next message
danliker is currently offline  danliker   Switzerland
Messages: 57
Registered: January 2004
Location: Schweiz
Karma: 0
Member
add to buddy list
ignore all messages by this user
ok, i need to wait for 3.0.5...
Aw: Spiders, Bots and Crawlers - Problem [message #169337 is a reply to message #169336] Fri, 27 September 2013 05:03 Go to previous messageGo to next message
danliker is currently offline  danliker   Switzerland
Messages: 57
Registered: January 2004
Location: Schweiz
Karma: 0
Member
add to buddy list
ignore all messages by this user
sorry but the only user agent string that is too long is the Google Bot, any hints how to troubleshoot my problem with other Bots that are correct in the DB?

[Updated on: Fri, 27 September 2013 05:04]

Report message to a moderator

Re: Aw: Spiders, Bots and Crawlers - Problem [message #169338 is a reply to message #169337] Fri, 27 September 2013 12:13 Go to previous messageGo to next message
naudefj is currently offline  naudefj   
Messages: 3624
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
remove from buddy list
ignore all messages by this user
Case should match as well. Besides that, I cannot think of anything else. To continue, maybe post your useragent mask as well as the "User-Agent" header of a bot that's not working.
Re : Re: Aw: Spiders, Bots and Crawlers - Problem [message #169339 is a reply to message #169338] Fri, 27 September 2013 13:07 Go to previous messageGo to next message
TheDude is currently offline  TheDude
Messages: 120
Registered: November 2012
Location: France
Karma: 2
Senior Member
add to buddy list
ignore all messages by this user
Stupid question perhaps, but what's the point of this, adding bots... ?

Re: Re : Re: Aw: Spiders, Bots and Crawlers - Problem [message #169340 is a reply to message #169339] Fri, 27 September 2013 13:55 Go to previous messageGo to next message
naudefj is currently offline  naudefj   
Messages: 3624
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
remove from buddy list
ignore all messages by this user
You can block the bad bots and keep track of the good bots visiting your forum.

EDIT: Something else: you can present a bot with a certain view of the site. For example, ensure Google index the French version, but Yahoo only see the German side. Silly example, but you get the idea.
Aw: Spiders, Bots and Crawlers - Problem [message #172197 is a reply to message #169333] Mon, 30 September 2013 06:24 Go to previous messageGo to next message
danliker is currently offline  danliker   Switzerland
Messages: 57
Registered: January 2004
Location: Schweiz
Karma: 0
Member
add to buddy list
ignore all messages by this user
ok, maybe i have some other problem.
my goal was to get correct amount of online users, as this is completely wrong in our forum.

as you can see on screen2 there is the same ip two times and this is the baiduspider ....

123.125.71.45 - - [30/Sep/2013:09:53:45 +0200] "GET /index.php/m/1223/efe26d61487b9e6e894945accc51efd1/ HTTP/1.1" 200 11899 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:10:07:57 +0200] "GET /index.php/m/1204/3bfc5723f883614c32e31a9ecd4aef58/ HTTP/1.1" 200 8065 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:10:43:38 +0200] "GET /index.php/e/Baiduspider/ HTTP/1.1" 200 1813 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:11:34:55 +0200] "GET /index.php/e/Baiduspider/ HTTP/1.1" 200 1812 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:11:41:58 +0200] "GET /index.php/m/1103/0f13c41485b761ea963f4e1bc61dc6a0/ HTTP/1.1" 200 7042 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:12:19:49 +0200] "GET /index.php/f/81/867941afb0ef4243549bfcba62da83d6/ HTTP/1.1" 302 505 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
123.125.71.45 - - [30/Sep/2013:12:19:59 +0200] "GET /index.php/m/1229/f46027036e10bdc9581b0bebf17c2411/ HTTP/1.1" 200 5775 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
  • Attachment: screen1.jpg
    (Size: 142.52KB, Downloaded 161 times)
  • Attachment: screen2.jpg
    (Size: 94.83KB, Downloaded 160 times)
Aw: Spiders, Bots and Crawlers - Problem [message #172198 is a reply to message #172197] Mon, 30 September 2013 06:26 Go to previous messageGo to next message
danliker is currently offline  danliker   Switzerland
Messages: 57
Registered: January 2004
Location: Schweiz
Karma: 0
Member
add to buddy list
ignore all messages by this user
also 220.181.108.163 is Baiduspider:

220.181.108.163 - - [30/Sep/2013:10:22:15 +0200] "GET /index.php/m/1195/15a8285e22b72251933bdb05f0e95e28/ HTTP/1.1" 200 7191 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:10:29:34 +0200] "GET /index.php/m/1189/f6f756c96cbe97b8bab3ba1749e8fd2c/ HTTP/1.1" 200 7235 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:10:29:40 +0200] "GET /index.php/m/1189/e320b79752a4308f46d2e856150bd843/ HTTP/1.1" 200 7235 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:11:06:21 +0200] "GET /index.php/e/Baiduspider/ HTTP/1.1" 200 1813 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:11:20:22 +0200] "GET /index.php/m/1139/3706e6fa5c19ee48313ba4f31b5fc044/ HTTP/1.1" 302 523 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:12:19:44 +0200] "GET /index.php/f/106/ed1e5b660769bbe16cdf924b4de0b9fa/ HTTP/1.1" 302 505 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
220.181.108.163 - - [30/Sep/2013:12:24:27 +0200] "GET /index.php/e/Baiduspider/ HTTP/1.1" 200 1813 "-" "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)"
Aw: Spiders, Bots and Crawlers - Problem [message #185768 is a reply to message #172198] Wed, 07 May 2014 07:13 Go to previous messageGo to next message
danliker is currently offline  danliker   Switzerland
Messages: 57
Registered: January 2004
Location: Schweiz
Karma: 0
Member
add to buddy list
ignore all messages by this user
is there no one that can help us here?

we have updated to 3.0.5 and actually google bot still does not get logged in

[Updated on: Wed, 07 May 2014 07:14]

Report message to a moderator

Re: Aw: Spiders, Bots and Crawlers - Problem [message #185769 is a reply to message #185768] Wed, 07 May 2014 10:42 Go to previous messageGo to next message
naudefj is currently offline  naudefj   
Messages: 3624
Registered: December 2004
Karma: 17
Senior Member
Administrator
Core Developer
remove from buddy list
ignore all messages by this user
It work just fine on this forum.
See http://fudforum.org/forum/index.php?t=usrinfo&id=9457&
Aw: Re: Aw: Spiders, Bots and Crawlers - Problem [message #185774 is a reply to message #185769] Wed, 07 May 2014 13:40 Go to previous message
danliker is currently offline  danliker   Switzerland
Messages: 57
Registered: January 2004
Location: Schweiz
Karma: 0
Member
add to buddy list
ignore all messages by this user
how can a troubleshot this? any hints?
Quick Reply
Formatting Tools:   
  Switch to threaded view of this topic Create a new topic
Previous Topic: Lightbox plugin help getting started
Next Topic: Announsment is not working
Goto Forum:
  

-=] Back to Top [=-
[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Wed Oct 18 05:22:22 EDT 2017

Total time taken to generate the page: 0.00775 seconds