apache - 如何使用 Apache Nutch 抓取 .pdf 链接

Question

我有一个要抓取的网站，其中包含一些指向 pdf 文件的链接。我想让 nutch 抓取该链接并将它们转储为 .pdf 文件。我正在使用 Apache Nutch1.6，我也在 java 中将其作为

ToolRunner.run(NutchConfiguration.create(), new Crawl(),
                                 tokenize(crawlArg));
 SegmentReader.main(tokenize(dumpArg));

谁能帮我解决这个问题

Answer 1

如果你想让 Nutch 抓取你的 pdf 文档并建立索引，你必须启用文档抓取和 Tika 插件:

1. 文档抓取

   1.1 编辑 regex-urlfilter.txt 并删除任何出现的“pdf”

   # skip image and other suffixes we can't yet parse
   # for a more extensive coverage use the urlfilter-suffix plugin
   -\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|xls|XLS|gz|GZ|rpm|RPM|tgz|TGZ|mov|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS)$
   

   1.2 编辑 suffix-urlfilter.txt 并删除所有出现的“pdf”

   1.3 编辑nutch-site.xml，在plugin.includes部分添加“parse-tika”和“parse-html”

   <property>
     <name>plugin.includes</name>
     <value>protocol-http|urlfilter-regex|parse-(html|tika|text)|index-(basic|anchor)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
     <description>Regular expression naming plugin directory names to
     include.  Any plugin not matching this expression is excluded.
     In any case you need at least include the nutch-extensionpoints plugin. By
     default Nutch includes crawling just HTML and plain text via HTTP,
     and basic indexing and search plugins. In order to use HTTPS please enable 
     protocol-httpclient, but be aware of possible intermittent problems with the 
     underlying commons-httpclient library.
     </description>
   </property>
   

2. 如果你真正想要的是从一个页面下载所有的 pdf 文件，你可以使用像 Teleport in Windows 这样的东西。或 *nix 中的 Wget。