robots.txt - "User-agent: * Disallow:/"是什么意思？

Question

关闭。这个问题不满足Stack Overflow guidelines .它目前不接受答案。












想改善这个问题吗？更新问题，使其成为 on-topic对于堆栈溢出。

7年前关闭。




Improve this question




在/robots.txt 页面上，这是什么意思？

User-agent: * 
Disallow: / 

这是否意味着您无法在搜索引擎上搜索并获得该网站的结果？例如它会阻止谷歌吗？

Answer 1

它阻止(好的)机器人(例如 Googlebot)索引任何页面。

来自 this page :

The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.

There are two important considerations when using /robots.txt:

• robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
• the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.