In this tutorial we’ll discuss robots.txt and How to prevent Search Engines from creeping your site’s folder or files.Suppose you have stored some private records on server and you would prefer not to file those documents on web crawler then you should utilize taking after User-operator administers in your robots.txt file.And robots.txt likewise help you to piece awful water crafts which are creeping your site and murdering your server assets. You can train search engines on how they ought to crawl a site, by utilizing a robots.txt file. At the point when a search engine crawls, it asks for the robots.txt file first and afterward takes after the tenets with-in.
block search engines robots txt, how to stop bots from crawling my site, how to stop search engines from crawling your website, prevent google from indexing site
Fist create robots.txt file on your server root folder should be accessible like this.
You ought to utilize User-agent run in your robots.txt file characterize web index how to crawl your website.Search engine crawlers utilize a User-agent to distinguish themselves when crawling, see cases:
Allow all search engines to crawl website –
By default search engine crawl all pages of your website if you haven’t define rules in your robots.txt file.
Disallow all search engines from crawling website-
You can easily disable search engine do not craw and index any page and files of website.
Disallow all search engines from particular folders or files –
If you want to disallow selected folder or files should not crawl by any search engine then use following rules in robots.txt file, Here i am disallowing my private-image folder and private.doc file.
Disallow bad boats to crawl your website –
You can set by default don’t crawl my website and after that allow trusted boats to crawl your website, which help you get away from bad boats and reduce server kill time.
Where Disallow: and Allow: / both are the same please don’t get confuse.