c++ - 解压缩中如何处理数组分配？

Question

我是一个使用C++开发人员进行项目开发的C#开发人员，我需要对字节数组进行deflate压缩，我发现了一个非常不错的库，称为libdeflate:https://github.com/ebiggers/libdeflate
但是我不知道如何处理输出数组的大小。
在C#中，有一个称为DeflatStream的内部压缩器，我们只传递一个字节数组并取回一个字节数组:

public static byte[] Compress(byte[] data)
{
    MemoryStream output = new MemoryStream();
    using (DeflateStream dstream = new DeflateStream(output, CompressionLevel.Optimal))
    {
        dstream.Write(data, 0, data.Length);
    }
    return output.ToArray();
}
public static byte[] Decompress(byte[] data)
{
    MemoryStream input = new MemoryStream(data);
    MemoryStream output = new MemoryStream();
    using (DeflateStream dstream = new DeflateStream(input, CompressionMode.Decompress))
    {
        dstream.CopyTo(output);
    }
    return output.ToArray();
}

在C#中，我们不需要原始数据大小或将其存储在某个地方。
这就是我在C++中的工作方式:

#include <iostream>
#include <vector>
#include <fstream>

#include "libdeflate.h"
#pragma comment( lib , "libdeflatestatic.lib" )

using namespace std;

#define LOG(x) printf("%s\n",x)
#define LOGVAL(valname,val) printf("Value `%s` = %d\n",valname,val)

int main() 
{
    struct libdeflate_compressor* compressor;
    struct libdeflate_decompressor* decompressor;
    ofstream fout;

    std::ifstream input("inputdata.dta", std::ios::binary);

    std::vector<unsigned char> buffer(std::istreambuf_iterator<char>(input), {});

    const int originalDataSize = buffer.size();

    // Compress
    compressor = libdeflate_alloc_compressor(9);
    std::vector<unsigned char> outdata;
    outdata.resize(originalDataSize);

    LOG("Compressing...");
    int cmpsize = libdeflate_deflate_compress(compressor, buffer.data(), originalDataSize, outdata.data(), originalDataSize);

    LOGVAL("cmpsize",cmpsize);

    fout.open("compressed.bin", ios::binary | ios::out);
    fout.write((char*)outdata.data(), cmpsize);
    fout.close();

    libdeflate_free_compressor(compressor);

    // Decompress
    size_t dec_size;
    std::vector<unsigned char> dec_outdata;
    dec_outdata.resize(originalDataSize);

    decompressor = libdeflate_alloc_decompressor();
    libdeflate_deflate_decompress(decompressor, outdata.data(), cmpsize, dec_outdata.data(), originalDataSize, &dec_size);

    fout.open("decompressed.bin", ios::binary | ios::out);
    fout.write((char*)dec_outdata.data(), dec_size);
    fout.close();

    libdeflate_free_decompressor(decompressor);

    LOG("App finished.");
}

我知道gzip和lz4有两种处理长度的方法，我之前曾使用过它们，我在压缩数据中添加了原始数据大小作为小标题，但是在这种情况下，每个字节对我都很重要。
如您所见，我始终需要 originalDataSize才能使压缩/解压缩正常工作，我不知道自己是否做错了什么，但我需要像C# DeflateStream的工作方式一样进行操作，我需要传递新的 vector 或不知道原始大小并进行解压缩的字节数组。
我看了一下.NET源代码，发现了这一点:

    internal const int DefaultBufferSize = 8192;
    private const int WindowSizeUpperBound = 47;

我该如何像在C#中一样仅将deflate用作byte []输入/输出，而又不将多余的数据存储在某处？
谢谢

Answer 1

您在标题(libdeflate.h)的文档中拥有所需的所有信息，以下是摘录:

/*
 * libdeflate_deflate_decompress() can be used in cases where the actual
 * uncompressed size is known (recommended) or unknown (not recommended):
 *
 *   - If the actual uncompressed size is known, then pass the actual
 *     uncompressed size as 'out_nbytes_avail' and pass NULL for
 *     'actual_out_nbytes_ret'.  This makes libdeflate_deflate_decompress() fail
 *     with LIBDEFLATE_SHORT_OUTPUT if the data decompressed to fewer than the
 *     specified number of bytes.
 *
 *   - If the actual uncompressed size is unknown, then provide a non-NULL
 *     'actual_out_nbytes_ret' and provide a buffer with some size
 *     'out_nbytes_avail' that you think is large enough to hold all the
 *     uncompressed data.  In this case, if the data decompresses to less than
 *     or equal to 'out_nbytes_avail' bytes, then
 *     libdeflate_deflate_decompress() will write the actual uncompressed size
 *     to *actual_out_nbytes_ret and return 0 (LIBDEFLATE_SUCCESS).  Otherwise,
 *     it will return LIBDEFLATE_INSUFFICIENT_SPACE if the provided buffer was
 *     not large enough but no other problems were encountered, or another
 *     nonzero result code if decompression failed for another reason.
 */

因此，您可以做的是使缓冲区尽可能大和/或认为足够大，并调用方法libdeflate_deflate_decompress，传递该缓冲区中有多少字节(在out_nbytes_avail中)，它将返回多少实际使用或需要的字节数(在actual_out_nbytes_ret中)。如果该方法返回成功，则流正常且缓冲区足够大(如果失败)
可能是流不正常或字节不够(请检查返回码)。
伪代码示例:

char *outBuffer=new char[SOME_SIZE];

size_t avail=SOME_SIZE;
size_t needed=0;
auto result=libdeflate_deflate_decompress(DecompresorStruct,
                  inBuffer, inBufferLen,
                  outBuffer, avail,
                  &needed);
if (result==LIBDEFLATE_INSUFFICIENT_SPACE)
{
   // SOME_SIZE is not enough you need at least 'needed' bytes
   // in the output buffer
   if (needed)
       SOME_SIZE=needed;
   else //couldn't determine the needed size
       SOME_SIZE=someHeuristic();
   
   //try again
}
else if (result==LIBDEFLATE_SUCCESS)
{
   // the stream was decompressed fin, you have the real output buffer
   // size in 'needed'
   
} 
else
{
   // some other error, decompression failed
}