c++ - 使用内存映射文件在 C++ 中解析二进制文件太慢

Question

我正在尝试按整数解析二进制文件，以检查整数值是否满足某个条件，但循环非常慢。
此外，我发现 memory-mapped file 是将文件快速读入内存的最快速度，因此我使用了以下基于 Boost 的代码:

unsigned long long int get_file_size(const char *file_path) {
    const filesystem::path file{file_path};
    const auto generic_path = file.generic_path();
    return filesystem::file_size(generic_path);
}

boost::iostreams::mapped_file_source read_bytes(const char *file_path,
                                         const unsigned long long int offset,
                                         const unsigned long long int length) {
    boost::iostreams::mapped_file_params parameters;
    parameters.path = file_path;
    parameters.length = static_cast<size_t>(length);
    parameters.flags = boost::iostreams::mapped_file::mapmode::readonly;
    parameters.offset = static_cast<boost::iostreams::stream_offset>(offset);

    boost::iostreams::mapped_file_source file;

    file.open(parameters);
    return file;
}

boost::iostreams::mapped_file_source read_bytes(const char *file_path) {
    const auto file_size = get_file_size(file_path);
    const auto mapped_file_source = read_bytes(file_path, 0, file_size);
    return mapped_file_source;
}

我的测试用例大致如下:

inline auto test_parsing_binary_file_performance() {
    const auto start_time = get_time();
    const std::filesystem::path input_file_path = "...";
    const auto mapped_file_source = read_bytes(input_file_path.string().c_str());
    const auto file_buffer = mapped_file_source.data();
    const auto file_buffer_size = mapped_file_source.size();
    LOG_S(INFO) << "File buffer size: " << file_buffer_size;
    auto printed_lap = (long) (file_buffer_size / (double) 1000);
    printed_lap = round_to_nearest_multiple(printed_lap, sizeof(int));
    LOG_S(INFO) << "Printed lap: " << printed_lap;
    std::vector<int> values;
    values.reserve(file_buffer_size / sizeof(int)); // Pre-allocate a large enough vector
    // Iterate over every integer
    for (auto file_buffer_index = 0; file_buffer_index < file_buffer_size; file_buffer_index += sizeof(int)) {
        const auto value = *(int *) &file_buffer[file_buffer_index];
        if (value >= 0x30000000 && value < 0x49000000 - sizeof(int) + 1) {
            values.push_back(value);
        }

        if (file_buffer_index % printed_lap == 0) {
            LOG_S(INFO) << std::setprecision(4) << file_buffer_index / (double) file_buffer_size * 100 << "%";
        }
    }

    LOG_S(INFO) << "Values found count: " << values.size();

    print_time_taken(start_time, false, "Parsing binary file");
}

memory-mapped file 读取几乎按预期立即完成，但尽管有出色的硬件( SSD 等)，但在我的机器上以整数方式迭代它太慢了:

2020-12-20 13:04:35.124 (   0.019s) [main thread     ]Tests.hpp:387   INFO| File buffer size: 419430400
2020-12-20 13:04:35.124 (   0.019s) [main thread     ]Tests.hpp:390   INFO| Printed lap: 419432
2020-12-20 13:04:35.135 (   0.029s) [main thread     ]Tests.hpp:405   INFO| 0%
2020-12-20 13:04:35.171 (   0.065s) [main thread     ]Tests.hpp:405   INFO| 0.1%
2020-12-20 13:04:35.196 (   0.091s) [main thread     ]Tests.hpp:405   INFO| 0.2%
2020-12-20 13:04:35.216 (   0.111s) [main thread     ]Tests.hpp:405   INFO| 0.3%
2020-12-20 13:04:35.241 (   0.136s) [main thread     ]Tests.hpp:405   INFO| 0.4%
2020-12-20 13:04:35.272 (   0.167s) [main thread     ]Tests.hpp:405   INFO| 0.5%
2020-12-20 13:04:35.293 (   0.188s) [main thread     ]Tests.hpp:405   INFO| 0.6%
2020-12-20 13:04:35.314 (   0.209s) [main thread     ]Tests.hpp:405   INFO| 0.7%
2020-12-20 13:04:35.343 (   0.237s) [main thread     ]Tests.hpp:405   INFO| 0.8%
2020-12-20 13:04:35.366 (   0.261s) [main thread     ]Tests.hpp:405   INFO| 0.9%
2020-12-20 13:04:35.399 (   0.293s) [main thread     ]Tests.hpp:405   INFO| 1%
2020-12-20 13:04:35.421 (   0.315s) [main thread     ]Tests.hpp:405   INFO| 1.1%
2020-12-20 13:04:35.447 (   0.341s) [main thread     ]Tests.hpp:405   INFO| 1.2%
2020-12-20 13:04:35.468 (   0.362s) [main thread     ]Tests.hpp:405   INFO| 1.3%
2020-12-20 13:04:35.487 (   0.382s) [main thread     ]Tests.hpp:405   INFO| 1.4%
2020-12-20 13:04:35.520 (   0.414s) [main thread     ]Tests.hpp:405   INFO| 1.5%
2020-12-20 13:04:35.540 (   0.435s) [main thread     ]Tests.hpp:405   INFO| 1.6%
2020-12-20 13:04:35.564 (   0.458s) [main thread     ]Tests.hpp:405   INFO| 1.7%
2020-12-20 13:04:35.586 (   0.480s) [main thread     ]Tests.hpp:405   INFO| 1.8%
2020-12-20 13:04:35.608 (   0.503s) [main thread     ]Tests.hpp:405   INFO| 1.9%
2020-12-20 13:04:35.636 (   0.531s) [main thread     ]Tests.hpp:405   INFO| 2%
2020-12-20 13:04:35.658 (   0.552s) [main thread     ]Tests.hpp:405   INFO| 2.1%
2020-12-20 13:04:35.679 (   0.574s) [main thread     ]Tests.hpp:405   INFO| 2.2%
2020-12-20 13:04:35.702 (   0.597s) [main thread     ]Tests.hpp:405   INFO| 2.3%
2020-12-20 13:04:35.727 (   0.622s) [main thread     ]Tests.hpp:405   INFO| 2.4%
2020-12-20 13:04:35.769 (   0.664s) [main thread     ]Tests.hpp:405   INFO| 2.5%
2020-12-20 13:04:35.802 (   0.697s) [main thread     ]Tests.hpp:405   INFO| 2.6%
2020-12-20 13:04:35.831 (   0.726s) [main thread     ]Tests.hpp:405   INFO| 2.7%
2020-12-20 13:04:35.860 (   0.754s) [main thread     ]Tests.hpp:405   INFO| 2.8%
2020-12-20 13:04:35.887 (   0.781s) [main thread     ]Tests.hpp:405   INFO| 2.9%
2020-12-20 13:04:35.924 (   0.818s) [main thread     ]Tests.hpp:405   INFO| 3%
2020-12-20 13:04:35.956 (   0.850s) [main thread     ]Tests.hpp:405   INFO| 3.1%
2020-12-20 13:04:35.998 (   0.893s) [main thread     ]Tests.hpp:405   INFO| 3.2%
2020-12-20 13:04:36.033 (   0.928s) [main thread     ]Tests.hpp:405   INFO| 3.3%
2020-12-20 13:04:36.060 (   0.955s) [main thread     ]Tests.hpp:405   INFO| 3.4%
2020-12-20 13:04:36.102 (   0.997s) [main thread     ]Tests.hpp:405   INFO| 3.5%
2020-12-20 13:04:36.132 (   1.026s) [main thread     ]Tests.hpp:405   INFO| 3.6%
...
2020-12-20 13:05:03.456 (  28.351s) [main thread     ]Tests.hpp:410   INFO| Values found count: 10650389
2020-12-20 13:05:03.456 (  28.351s) [main thread     ]          benchmark.cpp:31    INFO| Parsing binary file took 28.341 second(s)

解析那些 419 MB 总是需要大约 28 - 70 秒。即使在 Release 模式下编译也无济于事。有什么办法可以减少这个时间吗？我正在执行的操作似乎没有那么低效。
请注意，我正在使用 Linux 64-bit 编译 GCC 10 。
编辑:
正如评论中所建议的，将 memory-mapped file s 与 advise() 一起使用也无助于性能:

boost::interprocess::file_mapping file_mapping(input_file_path.string().data(), boost::interprocess::read_only);
boost::interprocess::mapped_region mapped_region(file_mapping, boost::interprocess::read_only);
mapped_region.advise(boost::interprocess::mapped_region::advice_sequential);
const auto file_buffer = (char *) mapped_region.get_address();
const auto file_buffer_size = mapped_region.get_size();
...

考虑到评论/答案，迄今为止学到的经验教训:

使用 advise(boost::interprocess::mapped_region::advice_sequential) 没有帮助

不调用 reserve() 或以完全正确的大小调用它 可以使性能翻倍

直接迭代 int * 比迭代 char * 慢一些

使用 std::set 收集结果比使用 std::vector 慢一些

进度日志对性能来说无关紧要

Answer 1

正如 xanatos 所暗示的那样memory-mapped file s 在性能上具有欺骗性，因为它们并没有真正立即将整个文件读入内存。在处理过程中，页面丢失导致多次磁盘访问，严重降低了性能。
在这种情况下，首先将整个文件读入内存然后遍历内存会更有效:

inline std::vector<std::byte> load_file_into_memory(const std::filesystem::path &file_path) {
    std::ifstream input_stream(file_path, std::ios::binary | std::ios::ate);

    if (input_stream.fail()) {
        const auto error_message = "Opening " + file_path.string() + " failed";
        throw std::runtime_error(error_message);
    }

    auto current_read_position = input_stream.tellg();
    input_stream.seekg(0, std::ios::beg);

    auto file_size = std::size_t(current_read_position - input_stream.tellg());
    if (file_size == 0) {
        return {};
    }

    std::vector<std::byte> buffer(file_size);

    if (!input_stream.read((char *) buffer.data(), buffer.size())) {
        const auto error_message = "Reading from " + file_path.string() + " failed";
        throw std::runtime_error(error_message);
    }

    return buffer;
}

现在性能更容易接受，大致 3 - 15 seconds总共。