java - Mac 上的 Snappy 在 Hadoop 上执行 Avro 作业时出现问题

Question

我在 Mac OSX 上工作。我在尝试运行在 hadoop 集群上使用 Avro 文件的 hadoop 作业时遇到以下异常。我使用 Java 1.6，我使用的 Snappy 版本是 1.0.4.1，这不是最后一个，但我无法更改它，因为其他部分可能会发生其他问题。

堆栈跟踪如下:

java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.xerial.snappy.SnappyLoader.loadNativeLibrary(SnappyLoader.java:317)
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:219)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
    at org.apache.avro.file.SnappyCodec.decompress(SnappyCodec.java:58)
    at org.apache.avro.file.DataFileStream$DataBlock.decompressUsing(DataFileStream.java:343)
    at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:199)
    at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:67)
    at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:34)
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210)
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:263)
Caused by: java.lang.UnsatisfiedLinkError: no snappyjava in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1758)
    at java.lang.Runtime.loadLibrary0(Runtime.java:823)
    at java.lang.System.loadLibrary(System.java:1045)
    at org.xerial.snappy.SnappyNativeLoader.loadLibrary(SnappyNativeLoader.java:52)
    ... 18 more
13/06/28 16:26:25 WARN mapred.LocalJobRunner: job_local_0001
org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
    at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:229)
    at org.xerial.snappy.Snappy.<clinit>(Snappy.java:44)
    at org.apache.avro.file.SnappyCodec.decompress(SnappyCodec.java:58)
    at org.apache.avro.file.DataFileStream$DataBlock.decompressUsing(DataFileStream.java:343)
    at org.apache.avro.file.DataFileStream.hasNext(DataFileStream.java:199)
    at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:67)
    at org.apache.avro.mapred.AvroRecordReader.next(AvroRecordReader.java:34)
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:210)
    at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:195)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:393)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:327)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:263)
13/06/28 16:26:25 INFO mapred.JobClient:  map 0% reduce 0%
13/06/28 16:26:25 INFO mapred.JobClient: Job complete: job_local_0001
13/06/28 16:26:25 INFO mapred.JobClient: Counters: 0
13/06/28 16:26:25 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323)
    at net.reunify.socialmedia.NumberOfTweetsPerUser.run(NumberOfTweetsPerUser.java:105)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at net.reunify.socialmedia.NumberOfTweetsPerUser.main(NumberOfTweetsPerUser.java:120)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Answer 1

如果可能的话，将 snappy 从 1.0.4 更改为 1.0.5。希望这能解决这个问题。