I use several crawling tools, and often crawl tests come back saying they were only able to crawl one page, after much trial and error I discovered it was down to my robots.txt file. This is how it looked like: I attempted to validate my robots.txt file with this website http://tool.motoricerca.info/robots-checker.phtml. I had some initial [...]