c++ - 如何将迭代器(或类似的东西)存储到硬盘上？

Question

我想存储一个巨大的 std::list<my_big_struct> (但在这个例子中很少)到硬盘上。

此外，我想存储很多 std::list<std::list<my_big_struct>::const_iterator>也到硬盘上。

我如何在没有深度复制的情况下存档此文件(会用完空间)？

下面是一个简短的例子 my_big_struct作为一个简单的 int .

#include <list>
#include <fstream>
#include <boost/serialization/serialization.hpp>
#include <boost/serialization/list.hpp>
#include <boost/archive/binary_oarchive.hpp>

typedef int my_big_struct;

int main()
{
    std::list<my_big_struct> a = {3,2,7,4,9};
    std::list<std::list<my_big_struct>::const_iterator> aa;
    std::list<std::list<my_big_struct>::const_iterator> ab;

    std::list<my_big_struct>::const_iterator find = a.begin();
    aa.emplace(aa.begin(), find++);
    ab.emplace(ab.begin(), find);

    {
        std::ofstream f("test_a.bin", std::ios::binary);
        boost::archive::binary_oarchive o(f);
        o << a; // OK
    }

    // how to store aa and ab onto harddisk without hardcopy ???
}

Answer 1

我不确定这个设计(这听起来像是保留 volatile 内存关系的设计味道)。

如果您确信自己的设计合理，我建议您使用 managed_shared_memory(即 Boost Interprocess)。

然后，您可以依赖 std::list¹ 的迭代器稳定性。

演示

这演示了它是如何工作的:

Live On Coliru

#include <boost/interprocess/managed_mapped_file.hpp>
#include <boost/interprocess/allocators/allocator.hpp>
#include <boost/interprocess/containers/list.hpp>
#include <iostream>
namespace bip = boost::interprocess;

typedef int my_big_struct;

namespace shared {
    using segment = bip::managed_mapped_file;
    using manager = segment::segment_manager;

    template <typename T> using alloc = bip::allocator<T, manager>;
    template <typename T> using list  = boost::interprocess::list<T, alloc<T> >;
}

int main()
{
    using List   = shared::list<my_big_struct>;
    using ItList = shared::list<List::const_iterator>;

    std::remove("test.dat");
    {
        shared::segment segment(bip::open_or_create, "test.dat", 10u<<20); // 10 megabyte

        auto& a  = *segment.find_or_construct<List>("a")(segment.get_segment_manager());
        auto& aa = *segment.find_or_construct<ItList>("aa")(segment.get_segment_manager());
        auto& ab = *segment.find_or_construct<ItList>("ab")(segment.get_segment_manager());

        a.assign({3,2,7,4,9});
        List::const_iterator find = a.begin();
        aa.emplace(aa.begin(), find++);

        ab.emplace(ab.begin(), find);
        ab.emplace(ab.begin(), std::next(find,3));
    }

    // the file is persisted
    {
        shared::segment segment(bip::open_only, "test.dat"); // read it back

        auto& a  = *segment.find_or_construct<List>("a")(segment.get_segment_manager());
        auto& ab = *segment.find_or_construct<ItList>("ab")(segment.get_segment_manager());

        for (auto iterator : ab) {
            std::cout << "Iterator in ab points to " << *iterator << " in list a\n";

            my_big_struct const* pointer = &*iterator;
            for (auto& element : a) {
                if (&element == pointer) {
                    std::cout << "Matching element is found in a (by physical address)\n";
                }
            }
        }

    }
}

打印

Iterator in ab points to 9 in list a
Matching element is found in a (by physical address)
Iterator in ab points to 2 in list a
Matching element is found in a (by physical address)

注意事项

为一些碎片开销做好准备，因此请注意如何构建数据结构(如果可能，请预先管理分配)。

您可以在 Boost Interprocess 上查看我的一些较早的答案以获取灵感

• boost::interprocess::managed_shared_memory
• boost::interprocess::managed_mapped_file
• scoped_allocator_adaptor为了更方便

---

¹ Boost Interprocess 通过使用 offset_ptr 在内部管理抽象(您不必了解此实现细节)

更多示例

如果您还想在 my_big_struct 中包含复杂成员，您也需要在那里使用段分配器:

Live On Coliru

打印

Iterator in ab points to my_big_struct { name:nine, value: 9 } in list a
Matching element is found in a (by physical address)
Iterator in ab points to my_big_struct { name:two, value: 2 } in list a
Matching element is found in a (by physical address)