mongodb - 将 mongoDB 的所有集合作为输入传递给 mapreduce hadoop

Question

我需要将 MongoDB 中数据库的所有集合作为输入传递给 Hadoop MR 作业。有一种方法允许多路输入:

MultiCollectionSplitBuilder mcsb = new MultiCollectionSplitBuilder();
mcsb.add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
        (MongoURI)null, // authuri
        true, // notimeout
        (DBObject)null, // fields
        (DBObject)null, // sort
        (DBObject)null, // query
        false,
        MultiMongoCollectionSplitter.class)
.add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
        (MongoURI)null, // authuri
        true, // notimeout
        (DBObject)null, // fields
        (DBObject)null, // sort
        new BasicDBObject("_id", new BasicDBObject("$gt", new Date(883440000000L))),
        false, // range query
        MultiMongoCollectionSplitter.class);

但我的数据库中有大约 10 个集合。上述方法仅允许 2 个集合参数。我需要做的就是单独获取映射器方法中的所有集合。我的 Reducer 对它们都是一样的。

感谢任何帮助。

Answer 1

可以继续添加到MultiCollectionSplitBuilder

    MultiCollectionSplitBuilder mcsb = new MultiCollectionSplitBuilder();
    mcsb
            .add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
                    (MongoURI) null, // authuri
                    true, // notimeout
                    (DBObject) null, // fields
                    (DBObject) null, // sort
                    (DBObject) null, // query
                    false,
                    MultiMongoCollectionSplitter.class
            )
            .add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
                    (MongoURI) null, // authuri
                    true, // notimeout
                    (DBObject) null, // fields
                    (DBObject) null, // sort
                    new BasicDBObject("_id", new BasicDBObject("$gt", new Date(883440000000L))),
                    false, // range query
                    MultiMongoCollectionSplitter.class
            )
            .add(new MongoURI("mongodb://localhost:27017/mongo_hadoop.yield_historical.in"),
                    (MongoURI) null, // authuri
                    true, // notimeout
                    (DBObject) null, // fields
                    (DBObject) null, // sort
                    new BasicDBObject("_id", new BasicDBObject("$gt", new Date(883440000000L))),
                    false, // range query
                    MultiMongoCollectionSplitter.class
            )
    ;