Magento SEO 总结 — Magento Robots.txt

下面是典型的一个magento站点的robots.txt具体设置, 有需要的同学可以根据实际情况做适当的调整…

# Website Sitemap

Sitemap: http://www.mydomain.com/sitemap.xml

# Crawlers Setup

User-agent: *

Crawl-delay: 10

# Allowable Index

Allow: /*?p=

Allow: /index.php/blog/

Allow: /catalog/seo_sitemap/category/

Allow:/catalogsearch/result/

# Directories

Disallow: /404/

Disallow: /app/

Disallow: /cgi-bin/

Disallow: /downloader/

Disallow: /includes/

Disallow: /js/

Disallow: /lib/

Disallow: /magento/

Disallow: /media/

Disallow: /pkginfo/

Disallow: /report/

Disallow: /skin/

Disallow: /stats/

Disallow: /var/

# Paths (clean URLs)

Disallow: /index.php/

Disallow: /catalog/product_compare/

Disallow: /catalog/category/view/

Disallow: /catalog/product/view/

Disallow: /catalogsearch/

Disallow: /checkout/

Disallow: /control/

Disallow: /contacts/

Disallow: /customer/

Disallow: /customize/

Disallow: /newsletter/

Disallow: /poll/

Disallow: /review/

Disallow: /sendfriend/

Disallow: /tag/

Disallow: /wishlist/

# Files

Disallow: /cron.php

Disallow: /cron.sh

Disallow: /error_log

Disallow: /install.php

Disallow: /LICENSE.html

Disallow: /LICENSE.txt

Disallow: /LICENSE_AFL.txt

Disallow: /STATUS.txt

# Paths (no clean URLs)

Disallow: /*.js$

Disallow: /*.css$

Disallow: /*.php$

Disallow: /*?p=*&

Disallow: /*?SID=