java - 在 Java 中从数据库写入 (zip) 文件的最节省内存的方法是什么？

Question

我的程序足够快，但我宁愿为了内存优化而放弃这个速度，因为一个用户的最大内存使用量高达 300 MB，这意味着很少有人会不断导致应用程序崩溃。我发现的大多数答案都与速度优化有关，其他答案只是一般性的(“如果直接从数据库写入内存，则不应该有太多内存使用”)。好吧，看来是有:)我正在考虑不发布代码，这样我就不会“锁定”某人的想法，但另一方面，如果你看不到我已经做了什么，我可能会浪费你的时间所以这里是:

// First I get the data from the database in a way that I think can't be more 
// optimized since i've done some testing and it seems to me that the problem 
// isn't in the RS and setting FetchSize and/or direction does not help.

public static void generateAndWriteXML(String query, String oznaka, BufferedOutputStream bos, Connection conn)
        throws Exception
{
    ResultSet rs = null;
    Statement stmt = null;
    try
    {
        stmt = conn.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
        rs = stmt.executeQuery(query);
        writeToZip(rs, oznaka, bos);
    } finally
    {
        ConnectionManager.close(rs, stmt, conn);
    }
}

// then I open up my streams. In the next method I'll generate an XML from the
// ResultSet and I want that XML to be saved in an XML, but since its size takes up
// to 300MB, I want it to be saved in a ZIP. I'm thinking that maybe by writing 
// first to file, then to zip I could get a slower but more efficient program.

private static void writeToZip(ResultSet rs, String oznaka, BufferedOutputStream bos)
        throws SAXException, SQLException, IOException
{
    ZipEntry ze = new ZipEntry(oznaka + ".xml");
    ZipOutputStream zos = new ZipOutputStream(bos);
    zos.putNextEntry(ze);
    OutputStreamWriter writer = new OutputStreamWriter(zos, "UTF8");
    writeXMLToWriter(rs, writer);
    try
    {
        writer.close();
    } catch (IOException e)
    {
    }
    try
    {
        zos.closeEntry();
    } catch (IOException e)
    {
    }
    try
    {
        zos.flush();
    } catch (IOException e)
    {
    }
    try
    {
        bos.close();
    } catch (IOException e)
    {
    }
}

// And finally, the method that does the actual generating and writing. 
// This is the second point I think I could do the memory optimization since the
// DataWriter is custom and it extends a custom XMLWriter that extends the standard
// org.xml.sax.helpers.XMLFilterImpl I've tried with flushing at points in program,
// but the memory that is occupied remains the same, it only takes longer.

public static void writeXMLToWriter(ResultSet rs, Writer writer) throws SAXException, SQLException, IOException
{
    //Set up XML
    DataWriter w = new DataWriter(writer);
    w.startDocument();
    w.setIndentStep(2);
    w.startElement(startingXMLElement);
    // Get the metadata
    ResultSetMetaData meta = rs.getMetaData();
    int count = meta.getColumnCount();
    // Iterate over the set
    while (rs.next())
    {
        w.startElement(rowElement);
        for (int i = 0; i < count; i++)
        {
            Object ob = rs.getObject(i + 1);
            if (rs.wasNull())
            {
                ob = null;
            }
            // XML elements are repeated so they could benefit from caching
            String colName = meta.getColumnLabel(i + 1).intern();
            if (ob != null)
            {
                if (ob instanceof Timestamp)
                {
                    w.dataElement(colName, Util.formatDate((Timestamp) ob, dateFormat));
                }
                else if (ob instanceof BigDecimal)
                {
                    // Possible benefit from writing ints as strings and interning them
                    w.dataElement(colName, Util.transformToHTML(new Integer(((BigDecimal) ob).intValue())));
                }
                else
                {   // there's enough of data that's repeated to validate the use of interning
                    w.dataElement(colName, ob.toString().intern());
                }

            }
            else
            {
                w.emptyElement(colName);
            }
        }
        w.endElement(rowElement);
    }
    w.endElement(startingXMLElement);
    w.endDocument();
}

编辑:这是内存使用情况的示例(使用 VisualVM 获取):

EDIT2:数据库是Oracle 10.2.0.4。我设置了 ResultSet.TYPE_FORWARD_ONLY 并获得了最大 50MB 的使用量!正如我在评论中所说，我会密切关注这一点，但它确实很有希望。

EDIT3:似乎还有另一种可能的优化可用。正如我所说，我正在生成一个 XML，这意味着大量数据被重复(如果没有别的，那么就是标签)，这意味着 String.intern() 可以在这里帮助我，当我测试这个时我会发回来。

Answer 1

是否可以使用 ResultSet.TYPE_FORWARD_ONLY？

您已使用 ResultSet.TYPE_SCROLL_INSENSITIVE。我相信对于某些数据库(您没有说您使用哪一个)这会导致整个结果集加载到内存中。