mongodb - Mongo-Hadoop 流式传输

Question

我是 Mongodb 和 Hadoop 的新手。我正在尝试访问 mongodb 数据作为 hadoop mapreduce 作业的输入。我不太清楚如何指定使用哪个集合从中获取数据。这是我试过的:

hadoop jar/usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar 
-input user/test/input/
-output user/test/output/
-inputformat com.mongodb.hadoop.mapred.MongoInputFormat
-outputformat com.mongodb.hadoop.mapred.MongoOutputFormat
-io mongodb
-D mongo.input.uri=mongodb://localhost/my_dbs.collectionName 
-D stream.io.identifier.resolver.class=com.mongodb.hadoop.streaming.io.MongoIdentifierResolver 
-mapper /Users/wordcountMapper.py 
-reducer /Users/wordcountReducer.py 
-libjars /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/mongo-hadoop-streaming.jar

但是我得到以下错误:

ERROR streaming.StreamJob: Unrecognized option: -D
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]

当我尝试这个时，我得到了另一个错误:

 hadoop jar /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/hadoop-streaming-2.6.0.jar 
-input user/input/ 
-output user/test/output 
-inputformat com.mongodb.hadoop.mapred.MongoInputFormat 
-outputformat com.mongodb.hadoop.mapred.MongoOutputFormat 
-io mongodb -jobconf mongo.input.uri=mongodb://localhost/my_dbs.collectionName 
-jobconf stream.io.identifier.resolver.class=com.mongodb.hadoop.streaming.io.MongoIdentifierResolver 
-mapper /Users/wordcountMapper.py 
-reducer /Users/wordcountReducer.py 
-libjars /usr/local/Cellar/hadoop/2.6.0/libexec/share/hadoop/tools/lib/mongo-hadoop-streaming.jar

`ERROR streaming.StreamJob: Unrecognized option: -libjars
Usage: $HADOOP_PREFIX/bin/hadoop jar hadoop-streaming.jar [options]`

请帮忙。

Answer 1

请查看this有关如何将 MongoDB 连接到 Hadoop 的更好想法的链接。

编辑:

或者，

你可以直接在你的驱动程序中写成:

args.add("-libjars");
args.add("/some/path/to/your/jar");