When you make a new website, usually you need to have a robots.txt file to dictate how robots or spiders will crawl your website. If you do not have a robots.txt file, then search engines will crawl all pages and directories on your website.Read More