It’s been a few days since I last posted, so I figured I would make a quick post about why I haven’t posted in a few days.
I have been trying to track down a few problems that are preventing this site and others I have on the same server from being crawled by Googlebot. About a week ago my daily crawl averages dropped straight down to zero on about 6 sites that I have on the same server, which just happens to be a shared hosting account. I also have a bunch of other sites that are on a different host, and some that are on dedicated servers that are being crawled just fine, so this led me to check if perhaps my hosting provider was blocking Googlebot. By all indications, this is indeed what is happening but they are reluctant to admit or even check into it because of the “work” involved in this process.
Servers have several security features typically that are designed to make for a safe, secure environment. The practices they use to provide this protection can sometimes target the wrong entity… Googlebot can be an aggressive lil bugger, crawling many pages at once from multiple IP addresses and when you are on a shared server, the same thing may be happening to many sites on the same server at the same time, this making it look like someone at Google HQ is going after that server hardcore which can cause the firewall to block the “offending” IP addresses and in effect, keeping Googlebot or any other similar crawler that has been banned from even accessing your site.
The dreadfully painful part about this is that the firewall will block based upon IP address, not a user agent that is easily searched for in log files etc, and Google owns a ton of IP addresses, so getting to the bottom of this can be quite the task. If something isn’t sorted out very soon I will just do the quick fix for me, which will be to change servers and/or my web hosting provider for the affected sites. I will give them a day or two more to sort it, but I am just about out of patience with this, so they better hurry up and get on it lol.
Also, the contest is still going strong, if you haven’t signed up for my newsletter and commented on a post, remember to do so before the end of the month to get yourself in on the contest and try to win yourself a few bucks.