Robots.txt in WordPress is a text file which helps in communicating with the web crawling bots. In this text file, we share all the details regarding what to display in the search engine and what not to display. No matter what we command to this file, it directly transfers it to the search engine.
We all are aware of the fact that search engine like Google index all the available content on our site by crawling it. However, sometimes we don’t prefer to display some content of our site on the SERP (search engine result page) so in such cases, Robot.txt comes at your rescue.
Robot.txt in WordPress is a file which tells the crawling bots which pages they aren’t supposed to visit. The search engines will probably follow the commands and will avoid crawling those particular pages. It is always advisable to avoid keeping sensitive data in the site and thinking of keeping it secure from crawling with a robots.txt file as following robots.txt isn’t mandatory for the search engines.
To implement Robots.txt in WordPress site, first, you need to be aware of the fact that:
Location of Robots.txt plays a significant role and the file should be placed in the main directory of the site otherwise the search engines won’t be able to find the robot.txt file. The search engine will look for the file here: http://mydomain.com/robots.txt and if there is no file there and it will index the whole site.
Structure of a Robots.txt in WordPress:
Its structure is very simple, it is a list comprises disallowed files, directories, and user agents.
User agents are generally the crawlers of the search engines and in disallowed files, one needs to mention the whole list of pages, posts or directories which you don’t want the search engine to index.