You may add these pages to your file to get explicitly disregarded. Robots.txt files use a thing known as the Robots Exclusion Protocol. This website will easily crank out the file for you with inputs of pages to get excluded. A robots.txt file controls how search engines crawl and index https://seotoolstube.com/