Ten. Gzip Compression: Help Gzip compression on your own server to lessen the measurement of files transferred concerning the server and the consumer’s browser. txt file is then parsed and may instruct the robot concerning which web pages usually are not to get crawled. To be a search engine crawler https://www.youtube.com/watch?v=0eLlfoHTk_E