python - asyncio 不会通过 tcp 发送整个图像数据

Question

我正在尝试使用带有 TCP 协议(protocol)的 asyncio 将图像从我的本地计算机发送到云中的计算机。有时我会发送整个图像，有时只发送部分图像。

客户端代码

import os
os.environ['PYTHONASYNCIODEBUG'] = '1'
import asyncio
import logging

logging.basicConfig(level=logging.ERROR)
async def tcp_echo_client(data, loop):
    reader, writer = await asyncio.open_connection(<ip_addr>, <port>,
                                                   loop=loop)
    print('Sending data of size: %r' % str(len(data)))
    writer.write(data)
    await writer.drain()
    #print("Message: %r" %(data))
    print(type(data))
    print('Close the socket')
    writer.write_eof()
    writer.close()

with open('sendpic0.jpg','rb') as f:
    data=f.read()
loop = asyncio.get_event_loop()
loop.run_until_complete(tcp_echo_client(data, loop))
loop.close()

服务器代码:

import os
os.environ['PYTHONASYNCIODEBUG'] = '1'
import asyncio
import logging

logging.basicConfig(level=logging.ERROR)

async def handle_echo(reader, writer):

    data = await reader.read()
    addr = writer.get_extra_info('peername')
    #print("Received %r from %r" % (message, addr))
    print("Length of data recieved: %r" % (str(len(data))))
    #with open('recvpic0.jpg','wb') as f:
    #    f.write(data)
    print("Close the client socket")
    writer.close()
    #print("Message: %r" %(data))
    print("Received data of length: %r" %(str(len(data))))


loop = asyncio.get_event_loop()
data=b''
coro = asyncio.start_server(handle_echo, '', <port_number>, loop=loop)
server = loop.run_until_complete(coro)
print("Received data of length: %r" %(str(len(data))))
# Serve requests until Ctrl+C is pressed
print('Serving on {}'.format(server.sockets[0].getsockname()))
try:
    loop.run_forever()
except KeyboardInterrupt:
    pass

# Close the server
server.close()
loop.run_until_complete(server.wait_closed())
loop.close()

我故意没有提供 IP 地址和端口号，但应该没关系。

这是输出:

服务器输出

Received data of length: '0'
Serving on ('0.0.0.0', 50001)

Length of data recieved: '249216'
Close the client socket
Received data of length: '249216'                                                                              

Length of data recieved: '250624'       
Close the client socket                                                                          
Received data of length: '250624'

Length of data recieved: '256403'                                                                              
Close the client socket                                                  
Received data of length: '256403'                                                                                              

客户端输出

$ python client.py       
Sending data of size: '256403' 
Close the socket
$ python client.py
<class 'bytes'>                                                                               
Close the socket                                                                              
$ python client.py       
Sending data of size: '256403'                                                                
<class 'bytes'>                                                                               
Close the socket                                                                              

我正在使用 Python 3.6。

我不知道我是否应该有检查机制或分 block 发送数据？我假设所有这些都会在读取功能下自动发生。

我调整了这个网站的代码:http://asyncio.readthedocs.io/en/latest/tcp_echo.html

Answer 1

这看起来像详细描述的关闭编写器错误 in this article .

简而言之，writer.close 不是协程，因此您不能等待 close 将数据从 asyncio 的缓冲区实际刷新到操作系统。在 close() 之前等待 writer.drain() 没有帮助，因为它只会暂停，直到后台写入将缓冲区大小减小到“低水位线”，而不是- 正如人们所期望的那样 - 直到缓冲区被清空。

更新:从 2018 年 6 月发布的 Python 3.7 开始，直接的修复是等待 writer.wait_closed()在 tcp_echo_writer 的末尾。

---

在最初写答案时，唯一可用的修复方法是复制 asyncio.open_connection 的实现(并不像听起来那么糟糕，因为它本质上是一个简短的便利函数) 并添加对 transport.set_write_buffer_limits(0) 的调用。这将使 await writer.drain() 实际上等待将所有数据写入操作系统(引用文章认为这无论如何都是正确的做法!):

@asyncio.coroutine
def fixed_open_connection(host=None, port=None, *,
                          loop=None, limit=65536, **kwds):
    if loop is None:
        loop = asyncio.get_event_loop()
    reader = asyncio.StreamReader(limit=limit, loop=loop)
    protocol = asyncio.StreamReaderProtocol(reader, loop=loop)
    transport, _ = yield from loop.create_connection(
        lambda: protocol, host, port, **kwds)
    ###### Following line added to fix buffering issues:
    transport.set_write_buffer_limits(0)
    ######
    writer = asyncio.StreamWriter(transport, protocol, reader, loop)
    return reader, writer

Weird that such a bug us hiding out in main asyncio library.

我怀疑大多数人没有看到这个错误，因为他们让事件循环运行更长时间来做其他事情，所以在 writer.close() 之后数据最终 被写出并关闭套接字。