It’s been a few days since I last posted, so I figured I would make a quick post about why I haven’t posted in a few days.
I have been trying to track down a few problems that are preventing this site and others I have on the same server from being crawled by Googlebot. About a week ago my daily crawl averages dropped straight down to zero on about 6 sites that I have on the same server, which just happens to be a shared hosting account. I also have a bunch of other sites that are on a different host, and some that are on dedicated servers that are being crawled just fine, so this led me to check if perhaps my hosting provider was blocking Googlebot. By all indications, this is indeed what is happening but they are reluctant to admit or even check into it because of the “work” involved in this process.
Servers have several security features typically that are designed to make for a safe, secure environment. The practices they use to provide this protection can sometimes target the wrong entity… Googlebot can be an aggressive lil bugger, crawling many pages at once from multiple IP addresses and when you are on a shared server, the same thing may be happening to many sites on the same server at the same time, this making it look like someone at Google HQ is going after that server hardcore which can cause the firewall to block the “offending” IP addresses and in effect, keeping Googlebot or any other similar crawler that has been banned from even accessing your site.
The dreadfully painful part about this is that the firewall will block based upon IP address, not a user agent that is easily searched for in log files etc, and Google owns a ton of IP addresses, so getting to the bottom of this can be quite the task. If something isn’t sorted out very soon I will just do the quick fix for me, which will be to change servers and/or my web hosting provider for the affected sites. I will give them a day or two more to sort it, but I am just about out of patience with this, so they better hurry up and get on it lol.
Also, the contest is still going strong, if you haven’t signed up for my newsletter and commented on a post, remember to do so before the end of the month to get yourself in on the contest and try to win yourself a few bucks.
Whoa! I didn’t think that shared hosting could harm in my site in some way.
Does this happens on all shared hosting?
No, this won’t happen a lot.. It’s just something to keep your eyes open for, if you monitor your site (I highly recommend Google analytics and webmaster tools) you will be able to pick up on any strangeness that occurs.. Didn’t mean to scare anyone, just wanted to make it known that this can happen.
Jesse
Firstly – bad luck. I’ve had an awful time with hosting really, I know how bad it can be. You spent all your time battling other sites, SERPs and Google; the last thing you want is to be battling your hosting.
P.s. I never got this article through my email subscription – I think it’s broke because I’m definately subscribed.
Chris’s last blog post..Adsblacklist and adsense competitive ad filter, does it work?
Thanks for the comment, it sure is a painful process when things don’t go as planned. I have also checked and you are shown as active in my subscribers list, so I am not sure what could have gone wrong.. Constant Contact typically has a very high delivery rate. Have you checked your spam folder? if its in there let me know and I will inform them so they can look into it.