DIYZONE

Bored just search google and found that it contains some do not. Useless connections.

However, we can use robots.txt to limit the search engine to crawl part of the directories and files.

What is a robots.txt? In fact, it is only one on the website root directory, and write the text file of some text.

Those things is to tell the search engine can be included, those who can not.

Doing so will increase the friendliness of the web page, seo, one of the methods.

Otherwise included several URLs, but the same page. This is not very good.

In google to find the corresponding wordpress robots.txt to modify some.

As follows

User-agent: *
Disallow: / cgi-bin
Disallow: / wp - admin
Disallow: / wp-includes
Disallow: / wp-content/plugins
Disallow: / wp-content/cache
Disallow: / your wp-content/themes
Disallow: / wp-content/upgrade
Disallow: / wpau-backup
Disallow: / trackback
Disallow: / comments
Disallow: * / trackback
Disallow: * / comments
Disallow: / wp-login.php

# Disallow all files with? In. the url
Disallow: / *? *
Disallow: / *?

Allow: / wp-content/uploads

# The allow the google image the bot to search All images
User-agent: Googlebot-Image
Disallow:
Allow: / *

# In Google AdSense
User-agent: Mediapartners-Google *
Disallow:
Allow: / *

User-agent: *
# Disallow all files ending with these extensions
Disallow: / *. Php $
Disallow: / * js $
Disallow: / * inc $
Disallow: / * css $
Disallow: / *. Cgi $
Disallow: / *. Xhtml

# The digg mirror
User-agent: duggmirror
Disallow: /

# Allow adsense the bot on entire site
User-agent: Mediapartners-Google *
Disallow:
Allow: / *

Sitemaps: http://diy.001zone.com/sitemap.xml.gz

The above code is saved as robots.txt, all lowercase.
And then uploaded to the site root directory.
Such as

http://diy.001zone.com

robots.txt would be on

http://diy.001zone.com/robots.txt

Related posts:

1. Ultra-hanging the google
2. Google launched a new service: Google Public DNS