java - 如何使用 Java 在 mongodb 中上传 json 文件？

Question

我正在尝试使用 Java 将一个大的 JSON 文件 (newclicklogs.json) 上传到 mongodb。这是我的 JSON 文件的样子:

{"preview":false,"result":{"search_term":"rania","request_time":"Sat Apr 01 12:47:04 -0400 2017","request_ip":"127.0.0.1","stats_type":"stats","upi":"355658761","unit":"DR","job_title":"Communications Officer","vpu":"INP","organization":"73","city":"Wash","country":"DC","title":"Tom","url":"www.demo.com","tab_name":"People-Tab","page_name":"PEOPLE","result_number":"5","page_num":"0","session_id":"df234f468cb3fe8be","total_results":"5","filter":"qterm=rina","_time":"2017-04-01T12:47:04.000-0400"}}
{"preview"......}
{"preview"......}
....

这是我的 Java 代码:

import java.io.File;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.io.FileUtils;
import org.bson.Document;
import com.mongodb.DBObject;
import com.mongodb.MongoClient;

public class Main {

    public static void main(String[] args) throws IOException {

        String jsonString = FileUtils.readFileToString(new File("data/newclicklogs.json"), "UTF-8");

        Document doc = Document.parse(jsonString);
        List<Document> list = new ArrayList<>();
        list.add(doc);

        new MongoClient().getDatabase("test2").getCollection("collection1").insertMany(list);

    }
}

当我查询我的 mongodb 集合时，只添加了一个文档。如何将文件中的所有文档添加到 mongodb 集合中。我是 mongodb 的新手。任何帮助表示赞赏。

Answer 1

您应该尝试使用缓冲读取器进行批量写入。

下面的代码会从文件中读取json数据，每次一行(document)，将json解析为Document并在写入数据库之前批量请求。

MongoClient client = new MongoClient("localhost", 27017);
MongoDatabase database = client.getDatabase("test2");
MongoCollection<Document> collection = database.getCollection("collection1");

int count = 0;
int batch = 100;

List<InsertOneModel<Document>> docs = new ArrayList<>();

try (BufferedReader br = new BufferedReader(new FileReader("data/newclicklogs.json"))) {
      String line;
      while ((line = br.readLine()) != null) {
         docs.add(new InsertOneModel<>(Document.parse(line)));
         count++;
         if (count == batch) {
           collection.bulkWrite(docs, new BulkWriteOptions().ordered(false));
           docs.clear();
           count = 0;
        }
    }
}

if (count > 0) {
   collection.bulkWrite(docs, new BulkWriteOptions().ordered(false));
}

当您在整个 json 上运行 Document.parse 时，您实质上是通过覆盖所有以前的文档来将文档减少到最后一个文档。

更多内容

http://mongodb.github.io/mongo-java-driver/3.4/driver/tutorials/bulk-writes/