"We expected that robots.txt files would treat all search engines equally, or maybe disfavor certain obnoxious bots, so we were surprised to discover a strong correlation between the robots favored and the search engines' market share," said C. Lee Giles, the David Reese Professor of Information Sciences and Technology at Penn State who led the research team that developed BotSeer, in a statement. "Robots.txt files are written by Web policy-makers and administrators who have to intentionally specify Google as the favored search engine" Giles said. Yahoo and MSN also were given greater-than-average access to Web sites, but Google showed up in robots.txt files more than twice as often as either of these competing search sites. Full Article at PC World