Fun with robots.txt
Posted: Fri Nov 25, 2016 2:31 amColumnist Patrick Stox provides some dos and don'ts for creating your robots.txt file along with examples of companies who have gotten creative with their files.
One of the most boring topics in technical SEO is robots.txt. Rarely is there an interesting problem needing to be solved in the file, and most errors come from not understanding the directives or from typos. The general purpose of a robots.txt file is simply to suggest to crawlers where they can and cannot go.
Read more : http://searchengineland.com/fun-robots-txt-263796
One of the most boring topics in technical SEO is robots.txt. Rarely is there an interesting problem needing to be solved in the file, and most errors come from not understanding the directives or from typos. The general purpose of a robots.txt file is simply to suggest to crawlers where they can and cannot go.
Read more : http://searchengineland.com/fun-robots-txt-263796