Google Nonsense: So now you need a robots.txt file before Google will index your site?

Submitted by Ryan Caldwell on January 10, 2008 - 1:22pm in
Google

Here's the most bizarre thing I've ever seen from Google. Literally.

In early December, Performancing launched Performancing Services ... a social media marketing and authority building service.

Embarassingly, Google decided not to index Performancing Services. Literally.

So a month later, I added Performancing Services to Webmaster Tools just to see what the hell Google was up to.

Here's what I found:

URL unreachable /robots.txt unreachable

Before we crawled the pages of your site, we tried to check your robots.txt file to ensure we didn't crawl any pages that you had roboted out. However, your robots.txt file was unreachable. To make sure we didn't crawl any pages listed in that file, we postponed our crawl. When this happens, we return to your site later and crawl it once we can reach your robots.txt file. Note that this is different from a 404 response when looking for a robots.txt file. If we receive a 404, we assume that a robots.txt file does not exist and we continue the crawl.

Now that is complete and utter nonsense. Since when do you need to upload a robots.txt file to get your site indexed?

Ryan Caldwell's blog

Ryan, it's because "your

Submitted by itrends on January 10, 2008 - 1:47pm.

Ryan, it's because "your robots.txt file was unreachable.". If it had returned a 404 message then they would have gone ahead as they would have known you have not supplied one.

I would suggest that either
a) your server was returning the wrong header code for the missing file
b) your server was unreachable / not responding during the time they tried to call it up

As a matter of practice you should always upload a blank robots.txt file to any website you build really to be on the safe side.

Either way, their message makes sense, it could be a little clearer, but I agree with what they are saying. I would do some checks on your side ;)

Just did a check and indeed you are sending back the wrong header information if a file is not there.

Go to http://www.seoconsultants.com/tools/headers.asp#results and run a check on a page such as http://services.performancing.com/testing.txt for a file that does not exist. It spits back an error 500 rather than an error 404. Bad server set up on your part I am affraid ;)