python - 向 Redis 发送大量数据的最佳实践

Question

我使用python3.6和Redis版本=5.0.7。我正在尝试向 Redis 发送大约 40 Mb 的数据。我有以下错误 ConnectionResetError: [Errno 104] Connection Reset by Peer

如何克服这个问题以及向 Redis 发送大量数据的最佳实践是什么？如果有任何建议，我将不胜感激。

import hashlib
import sys
import redis
client = redis.StrictRedis.from_url('redis://redis:6379/10', encoding="utf-8", decode_responses=True)

gen_data = {}
for i in range(10 ** 6):
    import random
    sku = f'{str(random.random())[:8]}-{random.random()}'
    hash = hashlib.md5(sku.encode()).hexdigest()
    gen_data[sku] = hash

print(f"len: {len(gen_data)} size: {sys.getsizeof(gen_data)/(1024*1024)} Mb")
client.hmset('products', gen_data)

输出

len: 1000000 size: 40.000099182128906 Mb
Traceback (most recent call last):
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/connection.py", line 666, in send_packed_command
    sendall(self._sock, item)
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/_compat.py", line 8, in sendall
    return sock.sendall(*args, **kwargs)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/peter/projects/vostok_projects/catalog/temp.py", line 16, in <module>
    client.hmset('products', gen_data)
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/client.py", line 2761, in hmset
    return self.execute_command('HMSET', name, *items)
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/client.py", line 838, in execute_command
    conn.send_command(*args)
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/connection.py", line 687, in send_command
    check_health=kwargs.get('check_health', True))
  File "/home/peter/.virtualenvs/vostok_projects/lib/python3.6/site-packages/redis/connection.py", line 679, in send_packed_command
    (errno, errmsg))
redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.

Answer 1

我对问题的看法:

根本不允许如此大量的数据。 Redis是单线程的，因此这样的请求会挂起Redis。如果确实需要这样做，您应该使用批量请求。

import itertools

def dict_to_chunks(data, size=10000):
    it = iter(data)
    for _ in range(0, len(data), size):
        yield {k: data[k] for k in itertools.islice(it, size)}


for d in dict_to_chunks(gen_data):
    client.hmset('products', d)

此外，还能够将高达 512 Mb 的序列化二进制字符串存储为 Redis'key 。读写都会比较快。