Your Webmaster Resources Webmaster Webmaster Your Webmaster Resources

Www Yourwebmasterresources R Your Webmaster Resources Webmaster Szh Boards Food Asp Your Webmaster Resources A Close to perfect .htaccess ban list - Part 3 Apache Web Server forum at WebmasterWorld

Www Yourwebmasterresources R Your Webmaster Resources Webmaster Szh Boards Food Asp Your Webmaster Resources

marke

Asp ssearch: Resources 5 Szh 6 Your 6search 9:48 am on Nov 30, 2003 (gmt 0)
Hi all

What a great resource for ideas and information.
I have been plying with .htaccess to try to block email spiders.
The info posted here has been of great help, but it only blocks those that advertise their presence.
I have downloaded a couple of email spiders, (first that came up on a search) and both of these still work through my site quite happily. They show up in the logs as Exlorer 6. I guess there is no way around this other than to have no emails on the web site?

Best regards,
Mark Empson
<snip>

[edited by: jdMorgan at 4:56 pm (utc) on Nov. 30, 2003]
[edit reason] No sigs or URLs, please [/edit]
jdMorgan

msg:1506467 5:01 pm on Nov 30, 2003 (gmt 0)
Mark,

Welcome to WebmasterWorld [webmasterworld.com]!

Take a look at this thread [webmasterworld.com] for an additional technique to stop site exploits. User-agent screening has the advantage of efficiency, in that you catch many intruders with one test. However, the ones you describe must be blocked by IP address, and further, a few must be blocked by forwarded IP address if they come through proxies.

Jim
jackson

msg:1506468 4:27 am on Dec 5, 2003 (gmt 0)
Just wondering if its a good to continue with this thread or start a new one ...

Picked this up today:

61.173.105.6 - - [04/Dec/2003:09:08:05 -0600] "GET /phpinfo.php HTTP/1.1" 200 37 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:0.9.4) Gecko/20011128 Netscape6/6.2.1"

The first thing that came to mind was, what's this preson doing requesting phpinfo.php? The next item that comes to mind is block access. And then, in doing that, that would mean including myself if ever I want to check my own details. This is for a hosted web site.

The question here is what to do with something like this?
jdMorgan

msg:1506469 5:07 am on Dec 5, 2003 (gmt 0)
Block it, then use mod_rewrite to give phpinfo an "alias" URL so you can still get to it.

Your post is on-topic as a potential addition to the list of troublesome user-agents. Any further discussion of *how* to deal with this specific problem probably does need its own thread, though.

Jim
jackson

msg:1506470 12:39 am on Dec 6, 2003 (gmt 0)
Jim,

Thanks for the follow up and the suggestion.

We'll chug on in what seems to be an ever-changing landscape.
Wizcrafts

msg:1506471 8:01 pm on Dec 27, 2003 (gmt 0)
Happy Holidays everyone!

I have a question concerning a RedirectMatch issue in my .htaccess file.

I have a hidden link to a non-existant file, which we will call example.html, embedded in a section containing all of the site links. Here is what the link resembles:
<a ngentmemekent="example.html" onclick="return false"> </a>. The onclick false action is a safety net for visual readers so they don't accidently trigger the banning redirect. Because there is no text for the link it is invisible on the displayed web page.

The link is redirected by .htaccess to my banning script, which we will call ban.pl, for this example. The url cleansed code appears below:
RedirectMatch example\.html /cgi-bin/ban.pl
Now, whenever a scooper-bot, or html-only downloader visits and scrapes for links, they follow the link to example.html and get a 302 redirect, according to my web log, but they do not hit the Perl script! When I tested this in Wannabrowser I was sent to the Perl script and banned, as designed. Here is my latest log of this mis-event:
"GET /example.html HTTP/1.0" 302 219 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT)"
The ban script is definitely larger than 219 bytes! This leach also took many more html only pages before leaving my server. Thus, he was not self banned, and never triggered the script to which he was supposed to be redirected.