python-3.x - ConnectionError : Error 104 while writing to socket. 连接被对等方重置

Question

ConnectionError:写入套接字时出现错误 104。对等方重置连接。

环境:Ubuntu的:16.04 python :3.6PC总内存:32G

我已经安装了 redis '3.0.6'。

尝试插入500,000条数据时成功，但尝试插入4000万条大数据时失败。

尝试将Python数据框插入redis时，由于容量太大而失败。

数据插入成功:

 1. r = redis.StrictRedis(host='localhost', port=6379, db=0)
 2.log_df_50.shape
   -> (500000, 6)
 3.r.setex('log_df_50',100,log_df_50.to_json())
   -> True

数据插入失败:

 1.r = redis.StrictRedis(host='localhost', port=6379, db=0)
 2. log_df.shape
   -> (41757802, 6)
 3. r.setex('session',100,log_df.to_json())

---

ConnectionResetError Traceback (most recent call last)~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/connection.pyin send_packed_command(self, command, check_health)705 for item in command:--> 706 sendall(self._sock, item)707 except socket.timeout:

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/_compat.pyin sendall(sock, *args, **kwargs)8 def sendall(sock, *args, **kwargs):----> 9 return sock.sendall(*args, **kwargs)10

ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

ConnectionError Traceback (most recent call last) in ----> 1 r.setex('session',100,log_df.to_json())

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/client.py insetex(self, name, time, value) 1820 if isinstance(time,datetime.timedelta): 1821 time =int(time.total_seconds())-> 1822 return self.execute_command('SETEX', name, time, value) 1823 1824 def setnx(self, name, value):

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/client.py inexecute_command(self, *args, **options)898 conn = self.connection or pool.get_connection(command_name, **options)899 try:--> 900 conn.send_command(*args)901 return self.parse_response(conn, command_name, **options)902 except (ConnectionError, TimeoutError) as e:

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/connection.pyin send_command(self, *args, **kwargs)724 "Pack and send a command to the Redis server"725 self.send_packed_command(self.pack_command(*args),--> 726 check_health=kwargs.get('check_health', True))727728 def can_read(self, timeout=0):

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/connection.pyin send_packed_command(self, command, check_health)716 errmsg = e.args[1]717 raise ConnectionError("Error %s while writing to socket. %s." %--> 718 (errno, errmsg))719 except BaseException:720 self.disconnect()

ConnectionError: Error 104 while writing to socket. Connection resetby peer.

关于原因的任何提示？

如何将python大容量Dataframe插入redis？

我应该怎么做才能解决这个问题？

Answer 1

我认为问题在于 Redis 协议(protocol)中键/值的 512 MB 限制。
https://redis.io/topics/protocol
https://redis.io/topics/data-types-intro

现在你无能为力了。

您可能只需要将数据帧切成更小的部分。

更新:
您也可以使用较新的 Redis(最低 5.0)并在 redis.conf 中增加 proto-max-bulk-len。 https://github.com/redis/redis/issues/7354
我还不得不增加 client-query-buffer-limit。