asp.net - 子目录中的 robots.txt

Question

我有一个位于主域下的文件夹中的项目，但我无权访问域本身的根目录。

http://mydomain.com/myproject/

我想禁止对子文件夹“forbidden”建立索引

http://mydomain.com/myproject/forbidden/

我可以简单地在 myproject 文件夹中放一个 robots.txt 吗？即使根目录中没有 robots.txt，它也会被读取吗？

禁止使用禁止文件夹的正确语法是什么？

User-agent: *
Disallow: /forbidden/

或

User-agent: *
Disallow: forbidden/

Answer 1

来自 robotstxt.org :

Where to put it

The short answer: in the top-level directory of your web server.

The longer answer:

When a robot looks for the "/robots.txt" file for URL, it strips the path component from the URL (everything from the first single slash), and puts "/robots.txt" in its place.

For example, for "http://www.example.com/shop/index.html, it will remove the "/shop/index.html", and replace it with "/robots.txt", and will end up with "http://www.example.com/robots.txt".

So, as a web site owner you need to put it in the right place on your web server for that resulting URL to work. Usually that is the same place where you put your web site's main "index.html" welcome page. Where exactly that is, and how to put the file there, depends on your web server software.

Remember to use all lower case for the filename: "robots.txt", not "Robots.TXT.

所以恐怕答案是您必须将它放在根文件夹中:-(

关于您的第二个问题，我认为正确的语法是以正斜杠开头的语法(例如 /forbidden/)。