The researchers didn’t know what they would find when they set BotSeer on the loose to look at and index the content of the robots.txt files of the Web sites which spanned several market segments including government, newspaper, university and Fortune 1000 companies. "Our intent was exploratory—to see if there was anything interesting" Councill said. Consumers with a soft spot for Google aren’t affected by the bias.
Full Article at EurekAlert