Robots.txt Generator

Default - All Robots are:	Allowed Refused
Crawl-Delay:	Default - No Delay 5 Seconds 10 Seconds 20 Seconds 60 seconds 120 Seconds
Sitemap: (leave blank if you don't have)
Search Robots:	Google	Same as Default Allowed Refused
	Google Image	Same as Default Allowed Refused
	Google Mobile	Same as Default Allowed Refused
	MSN Search	Same as Default Allowed Refused
	Yahoo	Same as Default Allowed Refused
	Yahoo MM	Same as Default Allowed Refused
	Yahoo Blogs	Same as Default Allowed Refused
	Ask/Teoma	Same as Default Allowed Refused
	GigaBlast	Same as Default Allowed Refused
	DMOZ Checker	Same as Default Allowed Refused
	Nutch	Same as Default Allowed Refused
	Alexa/Wayback	Same as Default Allowed Refused
	Baidu	Same as Default Allowed Refused
	Naver	Same as Default Allowed Refused
	MSN PicSearch	Same as Default Allowed Refused
Restricted Directories:	The path is relative to root and must contain a trailing slash "/"

Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.

About Robots.txt Generator

How do I create a robots txt file?
Follow these simple steps:
Open Notepad, Microsoft Word or any text editor and save the file as 'robots,' all lowercase, making sure to choose . txt as the file type extension (in Word, choose 'Plain Text' ).
Next, add the following two lines of text to your file:

How to use  Robots.txt Generator from Free-SEOTool.com ?

Open this Link to Generate Robots.txt file http://silebu.com/robots-txt-generator

Enter your Domain / Sitemap of your Domain 

Example : https://silebu.com/sitemap.xml

Then scroll down, you will find CREATE ROBOTS.TX button and press it.

It will ask for confirmation, press OK to continue or Cancel to stop the process. 

Thats it, you are done, you can download / save the file and upload to your domain.  

 What if there is no robots txt?
robots. txt is completely optional. If you have one, standards-compliant crawlers will respect it, if you have none, everything not disallowed in HTML-META elements   is crawlable. Site will be indexed without limitations.
 
Is robot txt necessary?
Most websites don't need a robots. txt file. That's because Google can usually find and index all of the important pages on your site. And they'll automatically NOT index pages that aren't important or duplicate versions of other pages.
 
What should be in a robots txt file?
txt file contains information about how the search engine should crawl, the information found there will instruct further crawler action on this particular site. If the robots. txt file does not contain any directives that disallow a user-agent's activity (or if the site doesn't have a robots.

What does a robots txt file do?
A robots. txt file tells search engine crawlers which pages or files the crawler can or can't request from your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.