java - Nutch 不会解析整个网站，只会解析第一个 URL

Question

我尝试使用 Nutch Fetcher 来获取整个网站，但它只加载第一个 URL:

import org.apache.nutch.fetcher.Fetcher;
new Fetcher(conf).fetch(segment, 1);

这是我在日志中看到的内容:

[INFO] org.apache.nutch.fetcher.Fetcher: Fetcher: starting at 2019-03-29 00:11:47
[INFO] org.apache.nutch.fetcher.Fetcher: Fetcher: segment: /var/folders/vl/633jwjvn2jvbj9zfg1sgglhw0000gp/T/1198814103175176756/segments/20190329001146
[WARN] org.apache.hadoop.mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
[INFO] org.apache.nutch.fetcher.FetchItemQueues: Using queue mode : byHost
[INFO] org.apache.nutch.fetcher.Fetcher: Fetcher: threads: 1
[INFO] org.apache.nutch.fetcher.Fetcher: Fetcher: time-out divisor: 2
[INFO] org.apache.nutch.fetcher.QueueFeeder: QueueFeeder finished: total 1 records hit by time limit : 0
[INFO] org.apache.nutch.net.URLExemptionFilters: Found 0 extensions at point:'org.apache.nutch.net.URLExemptionFilter'
[INFO] org.apache.nutch.fetcher.FetcherThread: FetcherThread 129 Using queue mode : byHost
[INFO] org.apache.nutch.fetcher.Fetcher: Fetcher: throughput threshold: -1
[INFO] org.apache.nutch.fetcher.Fetcher: Fetcher: throughput threshold retries: 5
[INFO] org.apache.nutch.fetcher.FetcherThread: FetcherThread 133 fetching http://www.zerocracy.com/ (queue crawl delay=5000ms)
[INFO] org.apache.nutch.protocol.RobotRulesParser: robots.txt whitelist not configured.
[INFO] org.apache.nutch.protocol.http.Http: http.proxy.host = null
[INFO] org.apache.nutch.protocol.http.Http: http.proxy.port = 8080
[INFO] org.apache.nutch.protocol.http.Http: http.proxy.exception.list = false
[INFO] org.apache.nutch.protocol.http.Http: http.timeout = 10000
[INFO] org.apache.nutch.protocol.http.Http: http.content.limit = 65536
[INFO] org.apache.nutch.protocol.http.Http: http.agent = yc/Nutch-1.15
[INFO] org.apache.nutch.protocol.http.Http: http.accept.language = en-us,en-gb,en;q=0.7,*;q=0.3
[INFO] org.apache.nutch.protocol.http.Http: http.accept = text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
[INFO] org.apache.nutch.protocol.http.Http: http.enable.cookie.header = true
[INFO] org.apache.nutch.fetcher.FetcherThread: FetcherThread 133 has no more work available
[INFO] org.apache.nutch.fetcher.FetcherThread: FetcherThread 133 -finishing thread FetcherThread, activeThreads=0
[INFO] org.apache.nutch.fetcher.Fetcher: -activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=0
[INFO] org.apache.nutch.fetcher.Fetcher: -activeThreads=0
[INFO] org.apache.nutch.fetcher.Fetcher: Fetcher: finished at 2019-03-29 00:11:49, elapsed: 00:00:02

我错过了什么？这是 Nutch 1.15。

Answer 1

Fetcher 类仅负责使用配置的线程数来获取/下载段中存在的 URL。这意味着 fetcher 不会从所获取的内容中解析或提取 URL。 fetch 方法仅下载内容，仅此而已。对于您的用例，您需要自行解析 HTML 内容(或使用 org/apache/nutch/parse 工具)并生成一个新段来获取新发现的链接。

这就是 Nutch 通常的工作方式，您提供一个或多个种子 URL。获取/解析此 URL，并存储新发现的链接以供下一次迭代使用。