抵制百度要从屏蔽它的爬虫做起

首先我们来观摸一下淘宝网的做法：http://www.taobao.com/robots.txt

依葫芦画瓢，这是我的写法：
[text]
User-agent: Baiduspider
Disallow: /

User-agent: baiduspider
Disallow: /

User-agent: *
Disallow: /down/
#更多要屏蔽的目录罗列在这里
[/text]

参考内容：在Google里搜索“robots 屏蔽 百度”关键词组即可找到相关内容。

Tags: ,

This entry was posted on 星期三, 27 4 月, 2011 at 17:33 and is filed under 东东博客. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

Leave a Reply