Have you ever thought about your robots.txt file, beyond how the various crawlers interact with it? Chances are that if you have one, you probably haven’t looked at it in since the day you created it. Well, it is time you take a fresh look at it and see how it looks not just to a bot’s eyes, but look at it through the eyes of a competitor.
You would be surprised at the number of sites and companies who use their robots.txt file as a way to keep bots out of certain directories on their site, but not considering the fact they have just pretty much handed the keys to those private areas over to their competitors. How? Because many people create their robots.txt file thinking that if the bots aren’t indexing those pages, no one will find it… but when you include those directories in your robots.txt file, you are telling real people exactly where those directories are. And surprisingly, many of those “secret” directories allow competitors to access it without requiring any kind of authentication or password.
Another thing people often give away in their robots.txt file is what they are working on adding to the site, even if they haven’t officially announced it yet. And of course, since webmasters are thinking future SEO value even in pre-launch, the directory names are always very telling because they are often keyword rich. One client was able to launch an entire section to their site because they noticed their competitor was doing something very similar when the directory popped up in the competitor’s robots.txt file but before they had either announced it or officially added it to the site. And like any good businessperson, my client beat them to the punch and launched it first. If only that competitor hadn’t jumped the gun by robots.txt-ing it, my client would have been none the wiser until launch day.
When you are working on something that you plan to launch in the future - especially launch with a splash to get publicity on it, don’t put it in your robots.txt file… all you need is a competitor to come along and see your robots.txt entry to discover what you are working on (and worse yet, if you end up having that directory open so they can spy on all your work-in-progress) and then launch their own copy of it first. Ideally, it should be completely password protected or done on a test domain that isn’t connected to your site.
Don’t want to worry about what your competitors are seeing in your robots.txt file? Carefully look at it and see what it is revealing. Do you have a super secret directory listed on there? Remove it from the robots.txt file, password protect those pages and add the no robots meta tag on those pages for good measure.
Are you working on something new for the site but haven’t announced or launched it yet? Again, password protect it or move it to a completely different unassociated domain you can lock down with robots.txt (since hopefully your competitor won’t know about that site). Or the best solution is to simply leave it offline except for brief testing periods pre-launch. This means if you want to check how it is working online, you upload it for only the amount of time you need to see it, and then delete it off the server.
Play it smart with your robots.txt file so you don’t inadvertantly hand your competitors the keys to your site or give them the edge up by alerting them to what you are working on. And for fun, check out some of your competitor’s robots.txt files… you can usually find something interesting on at least one or two of them.
Source: http://www.jenniferslegg.com/2008/02/12/what-is-your-robotstxt-file-telling-your-competitors-about-you/#more-56
No comments:
Post a Comment