c++ - boost 二进制文件 - 减少大小

Question

我正在尝试减少 C++ 中 boost 存档的内存大小。

我发现的一个问题是 Boost 的二进制存档默认为任何 int 使用 4 个字节，无论其大小如何。出于这个原因，我得到一个空的 boost 二进制存档占用 62 个字节，而一个空的文本存档占用 40 个字节(空文本存档的文本表示:22 serialization::archive 14 0 0 1 0 0 0 0 0)。

有什么方法可以改变整数的这种默认行为吗？

否则，除了对 vector 使用 make_array 之外，还有其他方法可以优化二进制存档的大小吗？

Answer 1

1. Q. I am trying to reduce the memory size of boost archives in C++.

   参见 Boost C++ Serialization overhead

2. Q. One problem I have found is that Boost's binary archives default to using 4 bytes for any int, regardless of its magnitude.

   那是因为它是序列化库，不是压缩库

3. Q. For this reason, I am getting that an empty boost binary archive takes 62 bytes while an empty text archive takes 40 (text representation of an empty text archive: 22 serialization::archive 14 0 0 1 0 0 0 0 0).

   使用存档标志:例如来自 Boost Serialization : How To Predict The Size Of The Serialized Result? :

   • Tune things (boost::archive::no_codecvt, boost::archive::no_header, disable tracking etc.)

4. Q. Is there any way to change this default behavior for ints?

   没有。有 BOOST_IS_BITWISE_SERIALIZABLE(T)虽然(参见例如 Boost serialization bitwise serializability 的示例和解释)。

5. Q. Else, are there any other ways to optimize the size of a binary archive apart from using make_array for vectors?

   使用 make_array对 vector<int> 没有帮助:

   Live On Coliru

   #include <boost/archive/binary_oarchive.hpp>
   #include <boost/serialization/vector.hpp>
   #include <sstream>
   #include <iostream>
   
   static auto const flags = boost::archive::no_header | boost::archive::no_tracking;
   
   template <typename T>
   std::string direct(T const& v) {
       std::ostringstream oss;
       {
           boost::archive::binary_oarchive oa(oss, flags);
           oa << v;
       }
       return oss.str();
   }
   
   template <typename T>
   std::string as_pod_array(T const& v) {
       std::ostringstream oss;
       {
           boost::archive::binary_oarchive oa(oss, flags);
           oa << v.size() << boost::serialization::make_array(v.data(), v.size());
       }
       return oss.str();
   }
   
   int main() {
       std::vector<int> i(100);
       std::cout << "direct: "       << direct(i).size() << "\n";
       std::cout << "as_pod_array: " << as_pod_array(i).size() << "\n";
   }
   

   打印

   direct: 408
   as_pod_array: 408
   

压缩

最直接的优化方法是压缩生成的流(另请参阅添加的基准 here)。

除此之外，您将不得不覆盖默认序列化并应用您自己的压缩(可以是简单的游程编码、霍夫曼编码或更特定领域的编码)。

演示

Live On Coliru

#include <boost/archive/binary_oarchive.hpp>
#include <boost/serialization/vector.hpp>
#include <sstream>
#include <iostream>
#include <boost/iostreams/filter/bzip2.hpp>
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/device/back_inserter.hpp>
#include <boost/iostreams/copy.hpp>

static auto const flags = boost::archive::no_header | boost::archive::no_tracking;

template <typename T>
size_t archive_size(T const& v)
{
    std::stringstream ss;
    {
        boost::archive::binary_oarchive oa(ss, flags);
        oa << v;
    }

    std::vector<char> compressed;
    {
        boost::iostreams::filtering_ostream fos;
        fos.push(boost::iostreams::bzip2_compressor());
        fos.push(boost::iostreams::back_inserter(compressed));

        boost::iostreams::copy(ss, fos);
    }

    return compressed.size();
}

int main() {
    std::vector<int> i(100);
    std::cout << "bzip2: " << archive_size(i) << "\n";
}

打印

bzip2: 47

这是约 11% 的压缩率(如果您删除存档标志，则为约 19%)。