robots.txt:用户代理:Googlebot 不允许:/Google 仍在编制索引

Question

查看本站的robots.txt:

fr2.dk/robots.txt

内容是:

User-Agent: Googlebot
Disallow: /

那应该告诉谷歌不要索引该网站，不是吗？

如果为真，为什么该网站会出现在谷歌搜索中？

Answer 1

除了必须等待，因为 Google 的索引更新需要一些时间，还要注意，如果您有其他网站链接到您的网站，仅 robots.txt 不足以删除您的网站。

引用 Google 的支持页面 "Remove a page or site from Google's search results" :

If the page still exists but you don't want it to appear in search results, use robots.txt to prevent Google from crawling it. Note that in general, even if a URL is disallowed by robots.txt we may still index the page if we find its URL on another site. However, Google won't index the page if it's blocked in robots.txt and there's an active removal request for the page.

上述文件中还提到了一种可能的替代解决方案:

Alternatively, you can use a noindex meta tag. When we see this tag on a page, Google will completely drop the page from our search results, even if other pages link to it. This is a good solution if you don't have direct access to the site server. (You will need to be able to edit the HTML source of the page).