python - 戈朗 : Unexpected EOF error while reading Gzip Reader

Question

我目前正在做一个项目，客户端 python 脚本使用 gzip 包压缩输入，然后根据 AES 对压缩数据进行加密，最后对加密后的数据进行编码base64 编码方案中的数据。在用 golang 编写的服务器端，我按照相反的顺序执行相同的步骤以获取原始数据。

从输出中可以看出，我可以在我的测试脚本上对从客户端接收到的数据进行解码、解密和解压缩。但是，在读取 gzip.NewReader 类型时，它会抛出 unexpected EOF error。 (查看问题底部的输出)

到目前为止，我在没有修剪填充的情况下完成了它，但是，它给了我一个无效的 gzip header 错误。然后，为了验证有效的 gzip Reader 和脚本创建的阅读器之间是否存在任何显着差异，我将原始数据 gzip 为压缩格式。显然，我从输出中看不到明显的差异，但字节差异很可能是由创建阅读器的时间变化引起的:

Original data: {"i": "MB88", "p": [{"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, 97107, 10350, 55200, 49.7], "t": 1499089212.136789, "dt": "p"}]}

Gzipped data: "\x1f\x8b\x08\x00\xf9w[Y\x02\xff%\x8e=\x0e\xc30\x08F\xaf\x82\x98\x91\x05\xe6\xc7\xa6c\xf7\x9e\xa0\xca\x96\xa5[\x86lQ\xee^\xdcN\xf0\xf4\xc1\x83\x0b?\xf8\x00|=\xe7D\x02<\n\xde\x17\xee\xab\xb85%\x82L\x02\xcb\xa6N\xa0\x7fri\xae\xd5K\xe1$\xe83\xc3\x08\x86Z\x81\xa9g-y\x88\xf6\x9a\xf5E\xde\x99\x7f\x96\xb1\xd5\x99\xb3\xfcb\x99\x121D\x1bG\xe7^.\xdcWPO\xdc\xdb\xfd\x05\x0ev\x15\x1d\x99\x00\x00\x00"

Gzip Reader created from the request by client

GzReader: &{{ [] 2017-07-03 15:40:12 +0200 CEST 255} 0xc42007e150 0xc4200a8000 0 0 [31 139 8 0 60 73 90 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

Gzip Reader created by directly reading the original data

GzReader2: &{{ [] 2017-07-04 13:11:53 +0200 CEST 255} 0xc4200741e0 0xc4200a9300 0 0 [31 139 8 0 249 119 91 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

现在，我很好奇可能导致此错误的原因以及如何解决此问题。由于某些限制，我无法在此处发布原始脚本，但这些是我当前正在测试我的代码的实际测试脚本。我已尽力提供我目前能想到的尽可能多的信息，但是，如果您需要更多解释，请随时问我。如果您能帮我解决这个问题，我将不胜感激!

干杯!

客户端代码(测试人员):

import time
import json
import requests
import random
import gzip
import base64

from Crypto.Cipher import AES

baseurl = 'http://0.0.0.0:80'
key = b'TfvY7I358yospfWKcoviZizOShpm5hyH'
iv = b'mb13KcoviZizvYhp'

MODE = AES.MODE_CFB
BLOCK_SIZE = 16
SEGMENT_SIZE = 128


def http_post(url, data):
    print('Going to make a request to {} with the following data: {}'.format(url, data))
    r = requests.post(url,
                      data=data,
                      headers={'Content-type': 'application/json; charset=utf-8',
                               'Connection': 'keep-alive'}, )
    if r.status_code != 200:
        print('Server returned unexpected response code {}, and content: {}'.format(r.status_code, r.content))
        return False
    else:
        data = r.json()
        return data


def post_sample_data(key, iv):
    fake_device_id = "MB88"
    Load_VA_Total_Mean = random.randint(1000, 100000)
    print('Data that should come back: {}'.format(Load_VA_Total_Mean))
    data = {"i": fake_device_id, "p": [
        {"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, Load_VA_Total_Mean, 10350, 55200, 49.7],
         "t": time.time(), "dt": "p"}]}
    url = baseurl + '/realtimedata'
    encryption_key_reference = 1
    payload = '{}\n{}'.format(convert_pack(data, key, iv), encryption_key_reference)
    return http_post(url, payload)


def check_sample_request():
    fake_device_id = "MB88"
    url = baseurl + '/datapoints/powerzones/status'
    data = [{"name": "P1", "field": "Load_VA_Total_Mean", "devices": [{"id": fake_device_id, "multiplier": 1}]}]
    return http_post(url, json.dumps(data))


def convert_pack(inputdict, key, iv):
    print('input: {}'.format(inputdict))
    print('key: {}'.format(key))

    jsonpayload = json.dumps(inputdict)  # encode dict to json string
    print('json: {}'.format(jsonpayload))

    gzippayload = gzip.compress(jsonpayload.encode('utf-8'))  # compress with gzip
    print('gzip: {}'.format(gzippayload))

    encryptedpayload = base64.b64encode(encrypt(key, iv, gzippayload))
    print('encrypted: {}'.format(encryptedpayload))
    return str(encryptedpayload, encoding='utf-8')    


def _pad_string(value):
    length = len(value)
    pad_size = BLOCK_SIZE - (length % BLOCK_SIZE)
    return value.ljust(length + pad_size, b'\x00')


def encrypt(key, iv, plaintext):
    aes = AES.new(key, MODE, iv, segment_size=SEGMENT_SIZE)
    plaintext = _pad_string(plaintext)
    encrypted_text = aes.encrypt(plaintext)
    return encrypted_text


post_sample_data(key, iv)
check_sample_request()

服务器代码(测试人员):

package main

import (
    "crypto/aes"
    "crypto/cipher"
    "encoding/base64"
    "fmt"
    "compress/gzip"
    "io/ioutil"
    "bytes"
    "strings"
)

func main() {
    //a sample input text generated by the client script
    originalText := "xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv"
    fmt.Println("Original request: ", originalText)
    key := []byte("TfvY7I358yospfWKcoviZizOShpm5hyH")
    text := decrypt(key, originalText)

    gzReader, err1 := gzip.NewReader(bytes.NewReader(text))
    fmt.Println("GzReader: ", gzReader) 
    if err1 != nil {
        fmt.Println("error1: ", err1)
    }

    content, err2 := ioutil.ReadAll(gzReader)   
    if err2 != nil {
        fmt.Println("error2: ",err2)
    }

    fmt.Printf("\nReading: \n -> Content in bytes: %v \n -> Content in string: %s\n", content, string(content)) 
}

func decrypt(key []byte, cryptoText string) []byte {
    ciphertext, _ := base64.StdEncoding.DecodeString(cryptoText)
    fmt.Printf("\nDecoding: \n -> Original: %v \n -> Decoded in bytes: %v \n -> Decoded in string: %s\n", cryptoText, ciphertext, ciphertext)
    block, err := aes.NewCipher(key)
    if err != nil {
        panic(err)
    }

    iv := []byte("mb13KcoviZizvYhp")    
    stream := cipher.NewCFBDecrypter(block, iv)
    stream.XORKeyStream(ciphertext, ciphertext)
    fmt.Printf("\nDecrypting: \n -> Decrypted in bytes: %v \n -> Decrypted in string: %s\n", ciphertext, ciphertext)

    ciphertext = bytes.Trim(ciphertext, "\x00")
    fmt.Printf("\nTrimming the padding: \n -> Decrypted in bytes: %v \n -> Decrypted in string: %s\n\n", ciphertext, ciphertext)    
    return ciphertext
}

终端输出:

Original request:

xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv

Decoding:

-> Original: xhhpLqMJs4g8+8Hb751bcalljkeHDJfbGdMqr9E9im1OkiJzmdLr7pWv/Grh0+uzdeMQBMzefqzTrzetf1fZcBm93IEF6wl8WYE5x2uAZ5XY39bxlRWDt4cTNk/wWisbAMSzNJqQUq2UmqBLaWcI0LU1NGPZd8EKHK36Iwlp8NuP+YsQIJSmQdN2cvNFOLGv

-> Decoded in bytes: [198 24 105 46 163 9 179 136 60 251 193 219 239 157 91 113 169 101 142 71 135 12 151 219 25 211 42 175 209 61 138 109 78 146 34 115 153 210 235 238 149 175 252 106 225 211 235 179 117 227 16 4 204 222 126 172 211 175 55 173 127 87 217 112 25 189 220 129 5 235 9 124 89 129 57 199 107 128 103 149 216 223 214 241 149 21 131 183 135 19 54 79 240 90 43 27 0 196 179 52 154 144 82 173 148 154 160 75 105 103 8 208 181 53 52 99 217 119 193 10 28 173 250 35 9 105 240 219 143 249 139 16 32 148 166 65 211 118 114 243 69 56 177 175]

-> Decoded in string: �i.� ��<����[q�e�G� ���*��=�mN�"s����j���u���~�ӯ7�W�p�܁� |Y�9�k�g��������6O�Z+ij4��R����Kiе54c�w� ��# i�ۏ�� ��A�vr�E8��

Decrypting:

-> Decrypted in bytes: [31 139 8 0 60 73 90 89 2 255 37 142 49 14 2 65 8 69 175 66 168 201 4 6 88 192 210 222 19 152 237 182 177 179 176 219 236 221 101 180 130 151 15 15 78 124 225 13 240 113 207 68 2 124 55 60 79 60 86 113 27 74 192 4 85 4 86 67 157 64 255 228 50 92 187 151 198 36 152 89 155 17 132 90 131 169 215 90 10 225 232 9 86 111 114 159 204 63 75 236 125 230 211 126 177 42 206 154 50 135 232 22 217 90 60 86 208 79 92 251 245 5 234 198 128 61 153 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] �q�D|7

Trimming the padding:

-> Decrypted in bytes: [31 139 8 0 60 73 90 89 2 255 37 142 49 14 2 65 8 69 175 66 168 201 4 6 88 192 210 222 19 152 237 182 177 179 176 219 236 221 101 180 130 151 15 15 78 124 225 13 240 113 207 68 2 124 55 60 79 60 86 113 27 74 192 4 85 4 86 67 157 64 255 228 50 92 187 151 198 36 152 89 155 17 132 90 131 169 215 90 10 225 232 9 86 111 114 159 204 63 75 236 125 230 211 126 177 42 206 154 50 135 232 22 217 90 60 86 208 79 92 251 245 5 234 198 128 61 153] �q�D|7

GzReader: &{{ [] 2017-07-03 15:40:12 +0200 CEST 255} 0xc42007e150 0xc4200a8000 0 0 [31 139 8 0 60 73 90 89 2 255 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] true}

error2: unexpected EOF

Reading:

-> Content in bytes: [123 34 105 34 58 32 34 77 66 56 56 34 44 32 34 112 34 58 32 91 123 34 100 34 58 32 91 53 52 46 51 44 32 48 44 32 57 57 44 32 52 57 46 51 53 44 32 51 44 32 57 57 44 32 53 49 46 53 51 51 44 32 49 44 32 57 56 44 32 50 56 57 54 52 44 32 55 51 52 56 44 32 52 51 53 57 48 44 32 57 55 49 48 55 44 32 49 48 51 53 48 44 32 53 53 50 48 48 44 32 52 57 46 55 93 44 32 34 116 34 58 32 49 52 57 57 48 56 57 50 49 50 46 49 51 54 55 56 57 44 32 34 100 116 34 58 32 34 112 34 125 93 125]

-> Content in string: {"i": "MB88", "p": [{"d": [54.3, 0, 99, 49.35, 3, 99, 51.533, 1, 98, 28964, 7348, 43590, 97107, 10350, 55200, 49.7], "t": 1499089212.136789, "dt": "p"}]}

Answer 1

当您认为您正在“修剪填充”时，您正在删除属于 gzip 流末尾的三个零。仅仅因为它是零并不意味着它是填充。保留 153 之后的三个零。

当您不修剪填充时，gzip 解码器会在成功解压缩第一个 gzip 流后寻找另一个 gzip 流。看到零而不是 gzip header 会导致无效的 gzip header 错误。

您需要跟踪 gzip 流的大小，以便了解什么是填充，什么不是。或者您可以一次解码一个 gzip 流，当您看到后面的零时，您可以断定那是填充。