SEO| SMO| Instant Approval Articles | Tips

Helpful Information about Search Engine Optimization and Social Media Optimization

How do you create a Robots.txt file?

Posted by selfsmo on September 29, 2009

A Robots.txt file can be created in a simple text editor such as notepad. An example of a Robots.txt file is shown below:

User-agent:Name of the robot comes here for eg. Googlebot

Disallow:The name of the file or directory comes here (This instruction disallows the files/directory from being indexed). Here are some examples:

  • Below is an example of a Robots.txt file that disallows all webpages from being indexed:

User-agent:*

Disallow:/

  •  Below is an example of a Robots.txt file that allows all webpages to be indexed :

User-agent:*

Disallow:

  • Below is an example of a Robots.txt file that disallows the Altavista Robot called “scooter” accessing the Admin directory and personal directory:

User-agent:scooter

Disallow:/admin/

Disallow:/personal/

  • Below is an example of a Robots.txt file that instructs bots not to crawl any file ending in .PDF

User-agent:*

Disallow:/*.pdf

The Robots.txt file can also have multiple sets of instructions for more than one bot. Each set of instructions should be seperated by a blank line. There is only one Robots.txt file for a website.

  • Below is an example of a Robots.txt file that disallows Google from crawling any of the dynamically generated pages and allows the altavista scooter bot to access every page.

User-agent:Googlebot

Disallow:/*?

User-agent:Scooter

Disallow:

Advertisements

6 Responses to “How do you create a Robots.txt file?”

  1. […] more here: How do you create a Robots.txt file? Tagged as: a-keyword-rich, a-scooter-bot, a-simple-text, carry-out-your, directory-and, […]

  2. […] Read this article: How do you create a Robots.txt file? « SEO| SMO| Instant Approval … […]

  3. Tnelson said

    Hey, I found your blog in a new directory of blogs. I dont know how your blog came up, must have been a typo, anyway cool blog, I bookmarked you. 🙂

  4. RobD said

    Hey, I found your blog while searching on Google your post looks very interesting for me. I will add a backlink and bookmark your site. Keep up the good work! 🙂

  5. Savannah said

    Awesome blog!

    I thought about starting my own blog too but I’m just too lazy so, I guess Ill just have to keep checking yours out.
    LOL,

  6. Alex said

    Some seo experts say that PR is important in seo and some seo expert say that there is no mean of PR only your keywords get good ranking in search engine but when I read this post now I can describe to in front of others that how PR is important and what does it means?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: