What's the difference between robots.txt and Meta Tags?
There are several types of Meta Tags, including Title, Description, Keywords. You place Meta Tags in the "head" section of your Web pages HTML to provide information that helps control robots and crawlers searching your website. The information in Meta Tags is not viewable by site visitors unless they view the page's source.
A robots.txt file specifies which parts of your Web page robots or crawlers can access. While some can ignore your robots.txt file, many search engines will find it and follow the specified protocol. You create and place a robots.txt file in the top-level directory of your Web server.
While both a robots.txt file and a Meta Tag communicate the preferences of your website to search engines attempting to crawl and collect information, using a robots.txt file is recommended. The robots.txt file allows for more flexibility and control over what gets searched. It should be uploaded to the hosted site's root directory.
