python - 如何解压mongo日志文件

Question

正如我所探索的，Mongodb 创建的日志文件是使用 snappy 压缩算法进行压缩的。但我无法解压这个压缩的日志文件。它在尝试解压缩时出错

Error stream missing snappy identifier

我用过的python解压代码如下:

import collections
import bson
from bson.codec_options import CodecOptions
import snappy
from cStringIO import StringIO
try:
    with open('journal/WiredTigerLog.0000000011') as f:
        content = f.readlines()
        fh = StringIO()
        snappy.stream_decompress(StringIO("".join(content)),fh)
        print fh
except Exception,e:
    print str(e)
    pass

请帮帮我，我走不了了

Answer 1

There's two forms of Snappy compression, the basic form and the streaming form. The basic form has the limitation that it all must fit in memory, so the streaming form exists to be able to compress larger amounts of data. The streaming format has a header and then subranges that are compressed. If the header is missing, it sounds like maybe you compressed using the basic form and are trying to uncompress with the streaming form. https://github.com/andrix/python-snappy/issues/40

如果是这种情况，请使用decompress 而不是stream_decompress。

但如果可能是数据根本没有压缩:

with open('journal/WiredTigerLog.0000000011') as f:
    for line in f:
        print line

可以工作。

Minimum log record size for WiredTiger is 128 bytes. If a log record is 128 bytes or smaller, WiredTiger does not compress that record. https://docs.mongodb.com/manual/core/journaling/