Robots.txt & Robots Tag in SEO

A robots.txt file helps in telling a search engine crawler what parts of your site you would not like crawled. A server level file, that excludes certain domains, directories, or files from being indexed by the search engines. The sitemap xml feed should be...