I saw today that Googlebot got caught in a spider trap that it shouldn’t have as that dir is blocked via robots.txt I know of at least one other person recently who this has also happened to. Why is GB ignoring robots? …Perhaps we need a list from Google explaining EXACTLY under which circumstances they would/will ignore robots.txt, since they are essentially acting like a spam bot or scraper.
Googlebot Is Getting Caught in robots.txt Spider Traps
This entry was posted in Uncategorized and tagged canada, domain, host, hosting, seo, server, web and hosting, web hosting, web server, website. Bookmark the permalink.