.net - 为什么 BCL GZipStream(带有 StreamReader)不能可靠地检测 CRC32 的数据错误？

Question

前几天我遇到了GZipStream doesn't detect corrupt data (even CRC32 passes)?这个问题(这很可能是一个“重复”，我对这个问题百感交集。我也是在标题中添加CRC32的人，但现在回想起来感觉与帖子的其余部分不相称)。在我自己探索了这个问题之后，我认为这个问题比另一个问题最初描述的要严重得多。

我扩展了另一个问题，并使测试代码可以在 LINQPad 下运行，并尝试更好地展示 CRC32 (Cyclic Redundancy Check) 问题(如果确实存在)。 (由于代码只是基于原始代码的轻微修改，因此测试设置/方法可能有缺陷，或者两者都存在另一个奇怪的怪癖/PEBCAK。)

结果很奇怪，因为损坏的数据并不总是导致引发(任何!)异常。请注意，仅有时 CRC32 检查看起来确实“有效”。导致索引超出范围/错误页眉/错误页脚的损坏字节可以被忽略，因为我们可以假设这些字节正在终止 CRC32 检查之前的解压缩(这是完美的)可以理解，即使 IndexOutOfRangeException 可能应该被 InvalidDataException 包装)，所以，

为什么CRC32检查的可靠性明显低于应有的可靠性？(为什么下面会出现“无效数据(无异常)”？)

由于 GZip footer contains CRC32 和未压缩数据的长度似乎是 error detection rate should be "significantly higher" - 也就是说，我不会期望下面是一个失败的案例，更不用说一些未检测到的损坏流了。 (当然，尽快检测到损坏的 Steam 是件好事:但在某些情况下，最终的安全校验和似乎会被完全忽略。)

格式为CorruptByteIndex+FailedDetections:消息:

0+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.1+0: System.IO.InvalidDataException:The magic number in GZip header is not correct. Make sure you are passing in a GZip stream.2+0: System.IO.InvalidDataException:The compression mode specified in GZip header is unknown.3+0: Good data (No Exception)4+0: Good data (No Exception)5+0: Good data (No Exception)6+0: Good data (No Exception)7+0: Good data (No Exception)8+0: Good data (No Exception)9+0: Good data (No Exception)10+0: System.IO.InvalidDataException:Unknown block type. Stream might be corrupted.11+1: Invalid data (No Exception)12+1: System.IO.InvalidDataException:Found invalid data while decoding.13+1: System.IO.InvalidDataException:Found invalid data while decoding.14+1: System.IO.InvalidDataException:Found invalid data while decoding.15+1: System.IO.InvalidDataException:Found invalid data while decoding.16+1: System.IO.InvalidDataException:Found invalid data while decoding.17+2: Invalid data (No Exception)18+2: System.IO.InvalidDataException:Found invalid data while decoding.19+2: System.IndexOutOfRangeException:Index was outside the bounds of the array.20+2: System.IndexOutOfRangeException:Index was outside the bounds of the array.21+3: Invalid data (No Exception)22+3: System.IndexOutOfRangeException:Index was outside the bounds of the array.23+3: System.IndexOutOfRangeException:Index was outside the bounds of the array.24+4: Invalid data (No Exception)25+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.26+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.27+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.28+4: System.IndexOutOfRangeException:Index was outside the bounds of the array.29+5: Invalid data (No Exception)30+5: System.IndexOutOfRangeException:Index was outside the bounds of the array.31+6: Invalid data (No Exception)32+7: Invalid data (No Exception)33+7: System.IndexOutOfRangeException:Index was outside the bounds of the array.34+7: System.IndexOutOfRangeException:Index was outside the bounds of the array.35+7: System.IndexOutOfRangeException:Index was outside the bounds of the array.36+8: Invalid data (No Exception)37+8: System.IndexOutOfRangeException:Index was outside the bounds of the array.38+8: System.IndexOutOfRangeException:Index was outside the bounds of the array.39+9: Invalid data (No Exception)40+9: System.IndexOutOfRangeException:Index was outside the bounds of the array.41+9: System.IndexOutOfRangeException:Index was outside the bounds of the array.42+10: Invalid data (No Exception)43+10: System.IO.InvalidDataException:Found invalid data while decoding.44+10: System.IndexOutOfRangeException:Index was outside the bounds of the array.45+10: System.IO.InvalidDataException:Found invalid data while decoding.46+11: Invalid data (No Exception)47+11: System.IndexOutOfRangeException:Index was outside the bounds of the array.48+11: System.IndexOutOfRangeException:Index was outside the bounds of the array.49+11: System.IndexOutOfRangeException:Index was outside the bounds of the array.50+12: Invalid data (No Exception)51+12: System.IndexOutOfRangeException:Index was outside the bounds of the array.52+12: System.IndexOutOfRangeException:Index was outside the bounds of the array.53+13: Invalid data (No Exception)54+13: System.IndexOutOfRangeException:Index was outside the bounds of the array.55+14: Invalid data (No Exception)56+14: System.IndexOutOfRangeException:Index was outside the bounds of the array.57+15: Invalid data (No Exception)58+15: System.IndexOutOfRangeException:Index was outside the bounds of the array.59+15: System.IndexOutOfRangeException:Index was outside the bounds of the array.60+16: Invalid data (No Exception)61+17: Invalid data (No Exception)62+18: Invalid data (No Exception)63+19: Invalid data (No Exception)64+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.65+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.66+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.67+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.68+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.69+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.70+19: System.IO.InvalidDataException:Found invalid data while decoding.71+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.72+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.73+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.74+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.75+19: System.IO.InvalidDataException:Found invalid data while decoding.76+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.77+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.78+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.79+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.80+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.81+19: System.IO.InvalidDataException:Found invalid data while decoding.82+19: System.IndexOutOfRangeException:Index was outside the bounds of the array.83+20: Invalid data (No Exception)84+21: Invalid data (No Exception)85+22: Invalid data (No Exception)86+22: System.IndexOutOfRangeException:Index was outside the bounds of the array.87+23: Invalid data (No Exception)88+24: Invalid data (No Exception)89+25: Invalid data (No Exception)90+25: System.IndexOutOfRangeException:Index was outside the bounds of the array.91+26: Invalid data (No Exception)92+26: System.IO.InvalidDataException:Found invalid data while decoding.93+26: System.IndexOutOfRangeException:Index was outside the bounds of the array.94+27: Invalid data (No Exception)95+27: System.IndexOutOfRangeException:Index was outside the bounds of the array.96+27: System.IndexOutOfRangeException:Index was outside the bounds of the array.97+28: Invalid data (No Exception)98+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.99+28: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.100+29: Invalid data (No Exception)101+30: Invalid data (No Exception)102+31: Invalid data (No Exception)103+32: Invalid data (No Exception)104+32: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.105+33: Invalid data (No Exception)106+34: Invalid data (No Exception)107+35: Invalid data (No Exception)108+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.109+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.110+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.111+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.112+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.113+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.114+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.115+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.116+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.117+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.118+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.119+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.120+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.121+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.122+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.123+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.124+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.125+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.126+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.127+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.128+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.129+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.130+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.131+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.132+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.133+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.134+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.135+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.136+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.137+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.138+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.139+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.140+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.141+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.142+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.143+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.144+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.145+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.146+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.147+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.148+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.149+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.150+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.151+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.152+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.153+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.154+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.155+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.156+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.157+35: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.158+36: Invalid data (No Exception)159+36: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.160+36: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.161+37: Invalid data (No Exception)162+38: Invalid data (No Exception)163+39: Invalid data (No Exception)164+40: Invalid data (No Exception)165+41: Invalid data (No Exception)166+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.167+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.168+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.169+41: System.IO.InvalidDataException:The CRC in GZip footer does not match the CRC calculated from the decompressed data.170+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.171+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.172+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.173+41: System.IO.InvalidDataException:The stream size in GZip footer does not match the real stream size.

Here is the test which is copy'n'paste runnable in LINQPad (for .NET 3.5 and 4, use "as C# statements" mode):

   string sample = "This is a compression test of microsoft .net gzip compression method and decompression methods";
   var encoding = new ASCIIEncoding();
   var data = encoding.GetBytes(sample);
   string sampleOut = null;
   byte[] cmpData;

   // Compress 
   using (var cmpStream = new MemoryStream())
   {
      using (var hgs = new System.IO.Compression.GZipStream(cmpStream, System.IO.Compression.CompressionMode.Compress))
      {
         hgs.Write(data, 0, data.Length);
      }
      cmpData = cmpStream.ToArray();
   }

   int corruptBytesNotDetected = 0;

   // corrupt data byte by byte
   for (var byteToCorrupt = 0; byteToCorrupt < cmpData.Length; byteToCorrupt++)
   {
      var corruptData = new List<byte>(cmpData).ToArray();
      // corrupt the data
      corruptData[byteToCorrupt]++;

      using (var decomStream = new MemoryStream(corruptData))
      {
         using (var hgs = new System.IO.Compression.GZipStream(decomStream, System.IO.Compression.CompressionMode.Decompress))
         {
            using (var reader = new StreamReader(hgs))
            {
               string message;
               try
               {
                  sampleOut = reader.ReadToEnd();

                  // if we get here, the corrupt data was not detected by GZipStream
                  // ... okay so long as the correct data is extracted

                  if (!sample.SequenceEqual(sampleOut)) {
                    corruptBytesNotDetected++;
                    message = "Invalid data (No Exception)";
                  } else {
                    message = "Good data (No Exception)";
                  }
               }
               catch(Exception ex)
               {
                    message = (ex.GetType() + ":" + ex.Message);
               }
               string.Format("{0}+{1}: {2}",
                    byteToCorrupt, corruptBytesNotDetected, message).Dump();
            }
         }
      }

   }

这是 .NET 3.5 中的压缩数据(众所周知，GZipStream 不擅长“压缩”小有效负载，但这是一个“无法修复”问题，因为该流在技术上仍然有效):

1F 8B 08 00 00 00 00 00 04 00 ED BD 07 60 1C 49 96 25 26 2F6D CA 7B 7F 4A F5 4A D7 E0 74 A1 08 80 60 13 24 D8 90 40 10EC C1 88 CD E6 92 EC 1D 69 47 23 29 AB 2A 81 CA 65 56 65 5D66 16 40 CC ED 9D BC F7 DE 7B EF BD F7 DE 7B EF BD F7 BA 3B9D 4E 27 F7 DF FF 3F 5C 66 64 01 6C F6 CE 4A DA C9 9E 21 80AA C8 1F 3F 7E 7C 1F 3F 22 DE CC 8B 26 A5 FF 65 E9 B4 5A ACEA BC 69 8A 6A 99 B6 79 D3 A6 D5 79 BA 28 A6 75 D5 54 E7 6D3A 5E E6 6D 7A F1 83 62 15 B4 5B E4 ED BC 9A A5 D9 72 96 CEF2 FE 17 CD FF 03 5C 51 5E 27 5E 00 00 00

(而且，只是为了咯咯笑，在 .NET 4 中它会生成一个稍大/不同的压缩流。)

1F 8B 08 00 00 00 00 00 04 00 EC BD 07 60 1C 49 96 25 26 2F6D CA 7B 7F 4A F5 4A D7 E0 74 A1 08 80 60 13 24 D8 90 40 10EC C1 88 CD E6 92 EC 1D 69 47 23 29 AB 2A 81 CA 65 56 65 5D66 16 40 CC ED 9D BC F7 DE 7B EF BD F7 DE 7B EF BD F7 BA 3B9D 4E 27 F7 DF FF 3F 5C 66 64 01 6C F6 CE 4A DA C9 9E 21 80AA C8 1F 3F 7E 7C 1F 3F 22 DE CC 8B 26 A5 FF 65 E9 B4 5A ACEA BC 69 8A 6A 99 B6 79 D3 A6 D5 79 BA 28 A6 75 D5 54 E7 6D3A 5E E6 6D 7A F1 83 62 15 B4 5B E4 ED BC 9A A5 D9 72 96 CEF2 FE 17 CD FF 13 00 00 FF FF 5C 51 5E 27 5E 00 00 00

附加说明:

在这种情况下，测试可能存在细微缺陷。当 GZipStream “未能检测到损坏”(无异常)时，从 StreamReader 读取的数据为“”(空字符串):在这种情况下，为什么 ReadToEnd() 不 引发异常(IOException 或其他)？

这是否是GZipStream，而是这里“古怪”的StreamReader，或者这仍然是GZipStream的问题(因为不抛出异常)？是否有一些正确的方法可以可靠地处理这个用例？ (考虑当前位置的输入流何时确实为空。)

Answer 1

前言

.NET [4 及以前的版本] 用户在任何情况下都不应使用 Microsoft 提供的 GZipStream 或 DeflateStream 类，除非 Microsoft 将它们完全替换为可用的东西。请改用 DotNetZip 库。

更新前言

.NET Framework 4.5 及更高版本已修复压缩问题，并且 GZipStream 和 DeflateStream 在这些版本中使用 zlib。我不知道下面引用的 CRC 问题是否已修复。

另一个更新

CRC 错误不仅没有得到修复，而且微软已经决定 won't fix它!

 

我在 Why does my C# gzip produce a larger file than Fiddler or PHP? 中的回复表明此行为并未反射(reflect)出 gzip 损坏检测的正确实现。在所有测试的情况下，正确的实现都会检测到错误。 (该回复还指出了为什么其中七个案例会产生良好的数据。)

另一个问题是:这种“压缩”方法如何从 94 字节字符串产生 174 字节输出？特别是考虑到字符串是如何压缩的——gzip 将其减少到 84 字节，即使有 18 字节 header 和标尾的开销。您能提供 174 字节的十六进制转储吗？