java - 编写大 JSON 文件避免内存不足问题的最佳方法

Question

首先，请注意今天是我使用 GSON 的第一天。我正在尝试使用 GSON 库编写 Json 文件。我在 ArrayList 中有数千个 JsonObjects。当写入 Json 文件时，它应该看起来与此类似。

[
    {
        "hash_index": "00102x05h06l0aj0dw",
        "body": "Who's signing up for Obamacare?",
        "_type": "ArticleItem",
        "title": "Who's signing up for Obamacare? - Jan. 13, 2014",
        "source": "money.cnn.com",
        "primary_key": 0,
        "last_crawl_date": "2014-01-14",
        "url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
    },
    {
        "hash_index": "00102x05h06l0aj0dw0iz0kn0l@0t#0",
        "body": "Who's signing up for Obamacare?",
        "_type": "ArticleItem",
        "title": "Who's signing up for Obamacare? - Jan. 13, 2014",
        "source": "money.cnn.com",
        "primary_key": 1,
        "last_crawl_date": "2014-01-14",
        "url": "http://money.cnn.com/2014/01/13/news/economy/obamacare-enrollment/index.html"
    }
]

现在，我使用以下代码编写 JSOn。

 private void writeNewJsonFile() throws IOException
    {
        System.out.println("Starting to write the JSON File");
        //Add everything into a JSONArray
        JsonArray jsonArrayNew = new JsonArray();

        for(int i=0;i<jsonObjectHolder.size();i++)
        {
            System.out.println("inside array");
            jsonArrayNew.add(jsonObjectHolder.get(i));
        }


        //Write it to the File
    /*  File file= new File("items_Articles_4_1.json");

        FileWriter fw = new FileWriter(file);;
        fw.write(jsonArrayNew.toString());
        fw.flush();
        fw.close();*/

        System.out.println("outside array");

        ByteArrayInputStream input = new ByteArrayInputStream(jsonArrayNew.toString().getBytes());

        Long contentLength = Long.valueOf(jsonArrayNew.toString().getBytes().length);

        ObjectMetadata metaData = new ObjectMetadata();
        metaData.setContentLength(contentLength);

        s3.putObject(outputBucket,outputFile,input,metaData);


    }

这里我将 JsonArray 转换为 String 并进行写入。我担心这很快就会因 Big Json 数组而崩溃并给出 OutOfMemoryException。就像我使用 GSON 逐段读取 Json 文件一样，有什么方法可以逐段写入 Json 文件或其他东西，从而避免 OutOfMemoryException 问题？

Answer 1

我正在使用下一个代码:

WriteJsonArrayByParts<Cache> write = new WriteJsonArrayByParts<Cache>(fileNameTest, " ");
write.writeStart();
for(Cache cache : listOfObjects()) {
    write.writeObject(cache, Cache.class);
}
write.writeEnd();
write.close();

...

public static class WriteJsonArrayByParts<T> {
    Gson gson = new Gson();
    JsonWriter writer;

    public WriteJsonArrayByParts(String fileNameWithPath, String indent) throws Exception {
        OutputStream os = new FileOutputStream(fileNameWithPath, false);
        BufferedOutputStream osb = new BufferedOutputStream(os, 8 * 1024);

        writer = new JsonWriter(new OutputStreamWriter(osb, StringUtil.UTF_8));
        writer.setIndent(indent);
    }

    public void writeStart() throws IOException {
        writer.beginArray();
    }

    @SuppressWarnings("unchecked")
    public void writeObject(T t, Class<?> resultClass) throws IOException {
        ((TypeAdapter<Object>) gson.getAdapter(resultClass)).write(writer, t);
    }

    public void writeEnd() throws IOException {
        writer.endArray();
    }

    public void close() throws IOException {
        writer.close();
    }
}