CWI Media, LLC

By Gary Cope

What a difference being indexed makes! I’m speaking, of course, about a Web site being indexed by Google, Yahoo, Ask and MSN, the big four search engines. I manage several Web sites, but the biggest one in terms of traffic was not indexed and was actually blocking search engine robots from indexing the site via the robots.txt.

I’m not entirely sure why someone would do that, but my theory is that they developed the site on a live server and set up the robots.txt file (the file that tells the robots what they can and cannot look at on the site) to block the search engines because the site wasn’t ready. Then, when the site was ready to launch, they copied the site to another folder and forgot to ammend the robots.txt file.

Anyhow, since I “unlocked” the site and allowed the robots to crawl and index the site, (I also submitted an xml sitemap to Google) search engine traffic has increased nearly 500 percent! While that sounds really good (and it is relatively speaking), you have to put it in perspective. The site was only getting less than 2 percent of its traffic from search engines prior to my modifications and that’s because only the home page was visible to the search engines. Now the site is getting about 12 percent of its traffic from search engines and my goal is to have it reach 30 percent by the fall.

Fingers crossed and I’ll keep you posted!