Java NIO 与非 NIO 性能对比

Question

我花了相当多的时间尝试优化文件哈希算法，以尽可能地提高性能。

查看我之前的 SO 主题:

Get File Hash Performance/Optimization

FileChannel ByteBuffer and Hashing Files

Determining Appropriate Buffer Size

有人多次推荐使用 Java NIO 来提高 native 性能(通过将缓冲区保留在系统中而不是将它们放入 JVM 中)。但是，我的 NIO 代码在基准测试中运行得相当慢(使用每种算法一遍又一遍地散列相同的文件，以否定任何可能扭曲结果的操作系统/驱动器“魔法”。

我现在有两个方法做同样的事情:

这个几乎每次都运行得更快:

/**
 * Gets Hash of file.
 * 
 * @param file String path + filename of file to get hash.
 * @param hashAlgo Hash algorithm to use. <br/>
 *     Supported algorithms are: <br/>
 *     MD2, MD5 <br/>
 *     SHA-1 <br/>
 *     SHA-256, SHA-384, SHA-512
 * @param BUFFER Buffer size in bytes. Recommended to stay in<br/>
 *          multiples of 2 such as 1024, 2048, <br/>
 *          4096, 8192, 16384, 32768, 65536, etc.
 * @return String value of hash. (Variable length dependent on hash algorithm used)
 * @throws IOException If file is invalid.
 * @throws HashTypeException If no supported or valid hash algorithm was found.
 */
public String getHash(String file, String hashAlgo, int BUFFER) throws IOException, HasherException {
    StringBuffer hexString = null;
    try {
        MessageDigest md = MessageDigest.getInstance(validateHashType(hashAlgo));
        FileInputStream fis = new FileInputStream(file);

        byte[] dataBytes = new byte[BUFFER];

        int nread = 0;
        while ((nread = fis.read(dataBytes)) != -1) {
            md.update(dataBytes, 0, nread);
        }
        fis.close();
        byte[] mdbytes = md.digest();

        hexString = new StringBuffer();
        for (int i = 0; i < mdbytes.length; i++) {
            hexString.append(Integer.toHexString((0xFF & mdbytes[i])));
        }

        return hexString.toString();

    } catch (NoSuchAlgorithmException | HasherException e) {
        throw new HasherException("Unsuppored Hash Algorithm.", e);
    }
}

我的 Java NIO 方法大部分时间运行得相当慢:

/**
 * Gets Hash of file using java.nio File Channels and ByteBuffer 
 * <br/>for native system calls where possible. This may improve <br/>
 * performance in some circumstances.
 * 
 * @param fileStr String path + filename of file to get hash.
 * @param hashAlgo Hash algorithm to use. <br/>
 *     Supported algorithms are: <br/>
 *     MD2, MD5 <br/>
 *     SHA-1 <br/>
 *     SHA-256, SHA-384, SHA-512
 * @param BUFFER Buffer size in bytes. Recommended to stay in<br/>
 *          multiples of 2 such as 1024, 2048, <br/>
 *          4096, 8192, 16384, 32768, 65536, etc.
 * @return String value of hash. (Variable length dependent on hash algorithm used)
 * @throws IOException If file is invalid.
 * @throws HashTypeException If no supported or valid hash algorithm was found.
 */
public String getHashNIO(String fileStr, String hashAlgo, int BUFFER) throws IOException, HasherException {

    File file = new File(fileStr);

    MessageDigest md = null;
    FileInputStream fis = null;
    FileChannel fc = null;
    ByteBuffer bbf = null;
    StringBuilder hexString = null;

    try {
        md = MessageDigest.getInstance(hashAlgo);
        fis = new FileInputStream(file);
        fc = fis.getChannel();
        bbf = ByteBuffer.allocateDirect(BUFFER); // allocation in bytes - 1024, 2048, 4096, 8192

        int b;

        b = fc.read(bbf);

        while ((b != -1) && (b != 0)) {
            bbf.flip();

            byte[] bytes = new byte[b];
            bbf.get(bytes);

            md.update(bytes, 0, b);

            bbf.clear();
            b = fc.read(bbf);
        }

        fis.close();

        byte[] mdbytes = md.digest();

        hexString = new StringBuilder();

        for (int i = 0; i < mdbytes.length; i++) {
            hexString.append(Integer.toHexString((0xFF & mdbytes[i])));
        }

        return hexString.toString();

    } catch (NoSuchAlgorithmException e) {
        throw new HasherException("Unsupported Hash Algorithm.", e);
    }
}

我的想法是 Java NIO 尝试使用 native 系统调用等来保持系统中和 JVM 之外的处理和存储(缓冲区)——这(理论上)可以防止程序必须不断地在 JVM 和系统之间来回切换。从理论上讲，这应该更快......但也许我的 MessageDigest 强制 JVM 引入缓冲区，从而否定 native 缓冲区/系统调用可以带来的任何性能改进？我在这个逻辑上是正确的还是我偏离了？

请帮助我理解为什么 Java NIO 在这种情况下不是更好。

Answer 1

可能会使您的 NIO 方法变得更好的两件事:

1. 尝试使用 memory-mapped file而不是将数据读入堆内存。
2. 将数据传递给摘要using a ByteBuffer而不是 byte[] 数组。

第一个应该避免在文件缓存和应用程序堆之间复制数据，而第二个应该避免在缓冲区和字节数组之间复制数据。如果没有这些优化，您可能会比单纯的非 NIO 方法进行更多的复制。