Robots.txt file is one of several text files. Website owners develop this to instruct Google and other search engines about how they will crawl on their website pages. This file tells the search engine not where to and where to not go on a website. Google describes robots.txt as being primarily used to manage crawler traffic into a website and keep a website page away from Google, although this will depend on the type of file that it is.