seo - Robots.txt 中的多个用户代理

Question

在 robots.txt 文件中，我有以下部分

User-Agent: Bot1
Disallow: /A

User-Agent: Bot2
Disallow: /B

User-Agent: *
Disallow: /C

语句 Disallow:c 对 Bot1 和 Bot2 可见吗？

Answer 1

tl;dr:不，Bot1 和 Bot2 会愉快地抓取以 C 开头的路径。

每个机器人最多只能遵守 a single record (block) .

原始规范

在original specification它说:

If the value is '*', the record describes the default access policy for any robot that has not matched any of the other records.

过期的 RFC 草案

原始规范，包括一些补充(如 Allow)成为 RFC 的草案，但从未被接受/发布。在 3.2.1 The User-agent line它说:

The robot must obey the first record in /robots.txt that contains a User-Agent line whose value contains the name token of the robot as a substring. The name comparisons are case-insensitive. If no such record exists, it should obey the first record with a User-agent line with a "*" value, if present. If no record satisfied either condition, or no records are present at all, access is unlimited.

因此它证实了原始规范的解释。

实现

谷歌，例如，gives an example这似乎符合规范:

Each section in the robots.txt file is separate and does not build upon previous sections. For example:

User-agent: *
Disallow: /folder1/

User-Agent: Googlebot
Disallow: /folder2/

In this example only the URLs matching /folder2/ would be disallowed for Googlebot.