A robots.txt file helps in telling a search engine crawler what parts of your site you would not like crawled. A server level file, that excludes certain domains, directories, or files from being indexed by the search engines. The sitemap xml feed should be listed in the robots.txt file.
Secondarily, the best practice is to not index search pages. Search pages are blocked from robots.txt files so no SEO is required for search pages.
Do not confuse the robots.txt file with the Robots Meta Tag.
A robots META tag can tell search engines to not index the page & to follow the page links or not.
- Follow/Index tags are no longer necessary
- By default, if a robots tag is absent, a crawler will index the page & follow on page links
- ONLY apply a robots tag, if you DO NOT want your page indexed