hadoop - 使用Hbase运行Nutch crawler 2.2时获取空指针异常

Question

当我运行Nutch命令时:〜/ nutch / runtime / deploy $ bin / nutch抓取urls -dir / user / dlequoc / urls -depth 2 -topN 5，出现以下异常:

================================================== =====

13/08/26 16:30:15 INFO mapred.JobClient: map 100% reduce 0% 13/08/26 16:30:29 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000000_0, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:32 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000001_0, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:32 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000005_0, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:32 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000004_0, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:32 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000002_0, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:32 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000003_0, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:44 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000001_1, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:47 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000000_1, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:47 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000005_1, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:47 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000002_1, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:47 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000004_1, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:47 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000003_1, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:59 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000000_2, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:59 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000002_2, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:30:59 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000001_2, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:31:02 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000005_2, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:31:05 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000003_2, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:31:11 INFO mapred.JobClient: Task Id : attempt_201308261546_0004_r_000004_2, Status : FAILED java.lang.NullPointerException at org.apache.avro.util.Utf8.(Utf8.java:37) at org.apache.nutch.crawl.GeneratorReducer.setup(GeneratorReducer.java:100) at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:174) at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:649) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:417) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) at org.apache.hadoop.mapred.Child.main(Child.java:249) 13/08/26 16:31:20 INFO mapred.JobClient: Job complete: job_201308261546_0004 13/08/26 16:31:20 INFO mapred.JobClient: Counters: 24 13/08/26 16:31:20 INFO mapred.JobClient: Job Counters 13/08/26 16:31:20 INFO mapred.JobClient: Launched reduce tasks=23 13/08/26 16:31:20 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=113452 13/08/26 16:31:20 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/08/26 16:31:20 INFO mapred.JobClient:
Total time spent by all maps waiting after reserving slots (ms)=0 13/08/26 16:31:20 INFO mapred.JobClient: Rack-local map tasks=1 13/08/26 16:31:20 INFO mapred.JobClient: Launched map tasks=1 13/08/26 16:31:20 INFO mapred.JobClient: Failed reduce tasks=1 13/08/26 16:31:20 INFO mapred.JobClient:
SLOTS_MILLIS_REDUCES=268210 13/08/26 16:31:20 INFO mapred.JobClient:
FileSystemCounters 13/08/26 16:31:20 INFO mapred.JobClient:
FILE_BYTES_READ=25743276 13/08/26 16:31:20 INFO mapred.JobClient:
HDFS_BYTES_READ=704 13/08/26 16:31:20 INFO mapred.JobClient:
FILE_BYTES_WRITTEN=51473783 13/08/26 16:31:20 INFO mapred.JobClient:
File Input Format Counters 13/08/26 16:31:20 INFO mapred.JobClient:
Bytes Read=0 13/08/26 16:31:20 INFO mapred.JobClient: Map-Reduce Framework 13/08/26 16:31:20 INFO mapred.JobClient: Map output materialized bytes=25720344 13/08/26 16:31:20 INFO mapred.JobClient:
Combine output records=0 13/08/26 16:31:20 INFO mapred.JobClient:
Map input records=333988 13/08/26 16:31:20 INFO mapred.JobClient:
Physical memory (bytes) snapshot=449036288 13/08/26 16:31:20 INFO mapred.JobClient: Spilled Records=667976 13/08/26 16:31:20 INFO mapred.JobClient: Map output bytes=25052332 13/08/26 16:31:20 INFO mapred.JobClient: CPU time spent (ms)=81870 13/08/26 16:31:20 INFO mapred.JobClient: Total committed heap usage (bytes)=208011264 13/08/26 16:31:20 INFO mapred.JobClient: Virtual memory (bytes) snapshot=740638720 13/08/26 16:31:20 INFO mapred.JobClient:
Combine input records=0 13/08/26 16:31:20 INFO mapred.JobClient:
Map output records=333988 13/08/26 16:31:20 INFO mapred.JobClient:
SPLIT_RAW_BYTES=704 Exception in thread "main" java.lang.RuntimeException: job failed: name=generate: null, jobid=job_201308261546_0004 at org.apache.nutch.util.NutchJob.waitForCompletion(NutchJob.java:54) at org.apache.nutch.crawl.GeneratorJob.run(GeneratorJob.java:199) at org.apache.nutch.crawl.Crawler.runTool(Crawler.java:68) at org.apache.nutch.crawl.Crawler.run(Crawler.java:152) at org.apache.nutch.crawl.Crawler.run(Crawler.java:250) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.nutch.crawl.Crawler.main(Crawler.java:257) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

能否请你帮忙？
谢谢!

Answer 1

在Nutch 2.x中不推荐使用/bin/nutch crawl命令。请改用/bin/crawl。