python - 通过套接字发送包含文件的字典(python)

Question

是否可以通过套接字发送包含文件(图像或文档)作为值的字典？

我试过像下面这样的东西但我失败了..

with open("cat.jpeg", "rb") as f:
    myFile = f.read(2048)

data = {"id": "1283", "filename": "cat.jpeg", "file": myFile}

dataToSend = json.dumps(data).encode("utf-8")

这给出了一个 json 错误，myFile 是一个字节数组，无法序列化。

我尝试使用 base64 编码将 myFile 转换为字符串，但没有成功。

部分起作用的是将 myFile 转换为字符串，例如 str(myFile)。 json 序列化器工作，我通过套接字发送它，dict 没问题，但 myFile 数据已损坏，所以我无法重新创建图片。

那么是否可以使用这种方法或者我应该如何通过套接字发送文件和数据以便在另一端轻松解析？

乐:

仍然无法使用 base64 编码，myFile 仍然是“字节”格式并且json 给出了这个错误:TypeError: Object of type 'bytes' is not JSON serializable

客户端

import os
import base64
import json
import socket

currentPath = os.path.dirname(os.path.abspath(__file__)) + "\\downloads\\"

with open(currentPath + "cat.png", "rb") as f:
    l = f.read()

print(type(l))   #prints <class 'bytes'>

myFile = base64.b64encode(l)

print(type(myFile))    #prints <class 'bytes'>

data = {"id": "12", "filename": "cat.png", "message": "So cute!", "file": myFile}

dataToSend = json.dumps(data).encode("utf-8")   #prints TypeError: Object of type 'bytes' is not JSON serializable

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 1234))
s.sendall(dataToSend)
s.close()

和服务器:

import socket
import json
import os
import sys
import time
import base64

currentPath = os.path.dirname(os.path.abspath(__file__)) + "\\fileCache\\"
tempData = bytearray()

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 1234))
s.listen(5)
conn, addr = s.accept()

while True:
    dataReceived = conn.recv(2048)
    if sys.getsizeof(dataReceived) > 17:
        tempData = tempData + dataReceived
    else:
        data = json.loads(tempData.decode("utf-8"))
        break
    time.sleep(1)

print(data)

myFile = base64.b64decode(data["file"])

with open(currentPath + data["filename"], "wb") as f:
    f.write(myFile)
    f.close()

Answer 1

正如我在评论中所说，将二进制数据打包成字符串格式(如 JSON)是一种浪费——如果你使用 base64，你会增加 33% 的数据传输大小，这也会让 JSON 解码器难以处理正确解码 JSON，因为它需要流式传输整个结构以提取索引。

最好单独发送它们 - JSON 作为 JSON，然后文件内容直接作为二进制。当然，您需要一种方法来区分两者，最简单的方法是在发送 JSON 数据时在其前面加上长度，以便服务器知道要读取多少字节才能获取 JSON，然后读取其余部分作为文件内容。这将使它成为一种非常简单的协议(protocol)，其包的形式如下:

[JSON LENGTH][JSON][FILE CONTENTS]

假设 JSON 永远不会大于 4GB(如果是，您将遇到更大的问题，因为解析它会是一场噩梦)拥有 JSON LENGTH 就足够了固定的 4 个字节(32 位)作为一个无符号整数(如果你不希望 JSON 超过 64KB，你甚至可以选择 16 位)所以整个策略将在客户端工作:

1. 创建负载
2. 将其编码为JSON，然后使用UTF-8编码将其编码为bytes
3. 获取上述包的长度，作为流的前4个字节发送
4. 发送JSON包
5. 读取并发送文件内容

在服务器端你做同样的过程

1. 读取接收到的数据的前4个字节，得到JSON payload长度
2. 读取下一个字节数以匹配此长度
3. 使用 UTF-8 将它们解码为字符串，然后解码 JSON 以获取有效负载
4. 读取其余的流式数据并将其存储到文件中

或者在代码中，客户端:

import json
import os
import socket
import struct

BUFFER_SIZE = 4096  # a uniform buffer size to use for our transfers

# pick up an absolute path from the script folder, not necessary tho
file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "downloads", "cat.png"))

# let's first prepare the payload to send over
payload = {"id": 12, "filename": os.path.basename(file_path), "message": "So cute!"}
# now JSON encode it and then turn it onto a bytes stream by encoding it as UTF-8
json_data = json.dumps(payload).encode("utf-8")
# then connect to the server and send everything
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:  # create a socket
    print("Connecting...")
    s.connect(("127.0.0.1", 1234))  # connect to the server
    # first send the JSON payload length
    print("Sending `{filename}` with a message: {message}.".format(**payload))
    s.sendall(struct.pack(">I", len(json_data)))  # pack as BE 32-bit unsigned int
    # now send the JSON payload itself
    s.sendall(json_data)  # let Python deal with the buffer on its own for the JSON...
    # finally, open the file and 'stream' it to the socket
    with open(file_path, "rb") as f:
        chunk = f.read(BUFFER_SIZE)
        while chunk:
            s.send(chunk)
            chunk = f.read(BUFFER_SIZE)
    # alternatively, if you're using Python 3.5+ you can just use socket.sendfile() instead
    print("Sent.")

和服务器:

import json
import os
import socket
import struct

BUFFER_SIZE = 4096  # a uniform buffer size to use for our transfers

target_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "fileCache"))

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.bind(("127.0.0.1", 1234))  # bind to the 1234 port on localhost
    s.listen(0)  # allow only one connection so we don't have to deal with data separation
    while True:
        print("Waiting for a connection...")
        connection, address = s.accept()  # wait for and accept the incoming connection
        print("Connection from `{}` accepted.".format(address))
        # read the starting 32 bits and unpack them into an int to get the JSON length
        json_length = struct.unpack(">I", connection.recv(4))[0]
        # now read the JSON data of the given size and JSON decode it
        json_data = b""  # initiate an empty bytes structure
        while len(json_data) < json_length:
            chunk = connection.recv(min(BUFFER_SIZE, json_length - len(json_data)))
            if not chunk:  # no data, possibly broken connection/bad protocol
                break  # just exit for now, you should deal with this case in production
            json_data += chunk
        payload = json.loads(json_data.decode("utf-8"))  # JSON decode the payload
        # now read the rest and store it into a file at the target path
        file_path = os.path.join(target_path, payload["filename"])
        with open(file_path, "wb") as f:  # open the target file for writing...
            chunk = connection.recv(BUFFER_SIZE)  # and stream the socket data to it...
            while chunk:
                f.write(chunk)
                chunk = connection.recv(BUFFER_SIZE)
        # finally, lets print out that we received the data
        print("Received `{filename}` with a message: {message}".format(**payload))

注意:请记住，这是 Python 3.x 代码 - 对于 Python 2.x，您必须自己处理上下文管理，而不是打开 with ... block /关闭你的套接字。

这就是它的全部。当然，在真实环境中，您需要处理断开连接、多个客户端等问题。但这是基本过程。