Tuesday, May 8, 2007

Condemned To Google Hell


Beginning in September 2006, Skyfacet no longer showed up on the first few pages of Google's results when users typed in search terms like "diamonds" and "engagement ring." The site's traffic vanished, and Sanar says his sales dropped $500,000 in three months.

What happened? Sanar isn't completely sure. But he does know that his site has been condemned to the supplemental index, a dreaded backwater region of Google search results that goes by another name in online marketing circles: Google Hell.

Google Hell is the worst fear of the untold numbers of companies that depend on search results to keep their business visible online. Getting stuck there means most users will never see the site, or at least many of the site's pages, when they enter certain keywords. And getting out can be next to impossible--because site operators often don't know what they did to get placed there.

Google's programmers appear to have created the supplemental index with the best intentions. It's designed to lighten the workload of Google's "spider," the algorithm that constantly combs and categorizes the Web's pages. Google uses the index as a holding pen for pages it deems to be of low quality or designed to appear artificially high in search results.

Those pages are scanned far less frequently than those in the main index, meaning that once a page is marked for Google Hell, it can languish there for as long as a year before Google even deigns it worthy of a reappraisal. And as Google tries to manage an explosively growing Web, more and more sites are finding themselves thrown into the search engine's digital dungeon.

If that makes the world's leading Web-crawler sound judgmental, consider Google's difficult position. The search juggernaut is faced with the endless task of reading and ranking the ever-expanding Web's billions of pages, the equivalent of putting the Earth's population in order from tallest to shortest every few minutes. Meanwhile there are growing numbers of pages filled only with junk text and advertising, designed solely to fool the engine. It's Google's task to sort out the trash from the worthwhile, and to do it better and faster than competitors like Yahoo! (nasdaq: YHOO - news - people ), Microsoft (nasdaq: MSFT - news - people ), or InterActiveCorp's (nasdaq: IACI - news - people ) Ask.com