robots.txt - robots.txt 中的顺序重要吗？

Question

这个问题在这里已经有了答案:





order of directives in robots.txt, do they overwrite each other or complement each other?

(1 个回答)


3年前关闭。




我的 robots.txt 如下:

User-agent: *
Disallow: /user/*
Disallow: /invitations/*
Disallow: /api/*

#Adsense crawler
User-agent: Mediapartners-Google
Disallow:

“Mediapartners-Google”爬虫真的可以扫描所有页面吗？
还是我robots.txt第一行 User-agent: *即使文件末尾有以下几行，也会阻止所有爬虫(包括“Mediapartners-Google”)访问上面提到的根:

User-agent: Mediapartners-Google
Disallow:

换句话说 - robots.txt 中的规则顺序是否重要 - 就像我的情况一样？

Answer 1

看来我找到了答案。
答案是 - 顺序并不重要，因为:

In a robots.txt file with multiple user-agent directives, each disallow or allow rule only applies to the useragent(s) specified in that particular line break-separated set. If the file contains a rule that applies to more than one user-agent, a crawler will only pay attention to (and follow the directives in) the most specific group of instructions.

来源: https://moz.com/learn/seo/robotstxt