Jill Kocher suggest a few ways to help you get an idea of how a search engine spider looks at your site.. fun? I’m not so sure, but valuable.. oh yes.
The key thing to remember is that search engines crawlers don’t have JavaScript, CSS or cookies enabled. They can’t “see” content embedded in media files such as images, Flash, video or audio. You take all this away, and what are you left with? Plain HTML, text and links. In many cases, the entire focus of the page is rendered indecipherable, and in some cases the site isn’t even navigable. It’s an incredible illustration of the structural challenges bots (i.e. search engine robots) face when they’re crawling a site.
Emulate a Search Engine Spider
Firefox’s Web Developer Toolbar by Chris Pederick will make surfing like a search spider much easier. For this exercise, focus on the yellow highlighted sections of the toolbar and select the following features:
Disable > Disable JavaScript > All JavaScript
Cookies > Disable Cookies > All Cookies
CSS > Disable Styles > All Styles
Images > Disable Images > All Images
Visit any page and you’re surfing approximately like a bot. If you’re already at the page you want to test, reload the page to get the full experience.
Another way to confirm what Googlebot is seeing is to visit Google’s text-only cache with JavaScript, cookies and CSS disabled. It should look the same as the live page with all those things disabled. If it doesn’t, it’s possible there is some form of user agent or bot detection in play on the site to deliver a different experience to search engine crawlers.
Wow! Thank you! I always wanted to write in my site something like that. Can I take part of your post to my blog?