Matt Fiddles

Life's so vast, there's just so much to do...

User Tools

Site Tools


Sidebar

"I find your lack of faith disturbing."

- Darth Vader



Where will you go today?

"`What's been happening here?' he demanded.
`Oh just the nicest things, sir, just the nicest things.
can I sit on your lap please?'"
"`Colin, I am going to abandon you to your fate.'
`I'm so happy.'"
"`It will be very, very nasty for you, and that's just too bad. Got it?'
`I gurgle with pleasure.'"

- Ford and Colin the robot.
computers:websites:robots.txt

Robots.txt Hints

Checkers

Docs

Wordpress

Basic one:

robots.txt
User-agent: Googlebot-Image
Disallow:
 
User-agent: Mediapartners-Google
Disallow:
 
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
# Disallow: /tag/ # uncomment if you are not using tags
# Disallow: /category/ # uncomment if you are not using categories
# Disallow: /author/ # uncomment for single user blogs
Disallow: /feed/
Disallow: /trackback/
# Disallow: /print/ # wp-print block
 
Sitemap: http://www.example.com/sitemap.xml

More advanced one, from: http://www.wpwebhost.com/how-to-create-a-wordpress-friendly-robots-txt-file/

robots.txt
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
# Disallow: /tag/ # uncomment if you’re not using tags
# Disallow: /category/ # uncomment if you’re not using categories
# Disallow: /author/ # uncomment for single user blogs
Disallow: /feed/
Disallow: /trackback/
# Disallow: /print/ # wp-print block
Disallow: /2009/ # the year your blog was born
Disallow: /2010/
Disallow: /2011/
Disallow: /2012/
Disallow: /2013/
Disallow: /2014/
Disallow: /2015/
Disallow: /2016/
Disallow: /2017/
Disallow: /2018/ # and so on
Disallow: /index.php # separate directive for the main script file of WP
Disallow: /*? # search results
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: */trackback/
# Disallow: */print/
User-agent: Googlebot-Image
Disallow:
Allow: /
User-agent: Mediapartners-Google
Disallow:
Allow: /
Sitemap: http://yourdomain.com/sitemap.xml

Be sure to test it against Google Webmaster.

computers/websites/robots.txt.txt · Last modified: May 29, 2014 (4 years ago) by Matt Bagley