Googlebot Is Getting Caught in robots.txt Spider Traps

I saw today that Googlebot got caught in a spider trap that it shouldn’t have as that dir is blocked via robots.txt I know of at least one other person recently who this has also happened to. Why is GB ignoring robots? …Perhaps we need a list from Google explaining EXACTLY under which circumstances they would/will ignore robots.txt, since they are essentially acting like a spam bot or scraper.

This entry was posted in Uncategorized and tagged , , , , , , , , , . Bookmark the permalink.