java - Redis (AWS Elasticache) 在一段时间后停止工作

Question

• 我们以前用于在我们的数据库表 (Postgres RDS) 中存储用户 session

• 我们决定将用户 session 从数据库迁移到 Redis，并对我们的应用程序进行了更改

• 对于 Redis，我们决定使用具有 1 个分片、2 个节点(主 + 副本)和启用多可用区的 Elastic Cache 服务

• 在 Live 环境中，一切都很顺利，直到 session 数超过 50 万(下午 3 点左右)

• 此时Redis Node突然停止响应导致我们的Production环境彻底崩溃(等待连接的线程过多)

• 我们必须重启实例才能恢复服务

• 这在晚上 9 点左右再次发生

Java端(spring)产生的异常

2016/11/22 09:19:31.749 <a href="http-nio-8080-exec-780">http-nio-8080-exec-780</a> <a href="https://forums.aws.amazon.com/">ERROR</a> org.apache.catalina.core.ContainerBase.<a href="https://forums.aws.amazon.com/">Tomcat</a>.<a href="https://forums.aws.amazon.com/">localhost</a>.[/].<a href="https://forums.aws.amazon.com/">dispatcherServlet</a> - Servlet.service() for servlet <a href="https://forums.aws.amazon.com/">dispatcherServlet] in context with path [</a> threw exception

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:140) ~<strike>spring-data-redis-1.4.2.RELEASE.jar!/:1.4.2.RELEASE</strike>

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:229) ~<strike>spring-data-redis-1.4.2.RELEASE.jar!/:1.4.2.RELEASE</strike>

....

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) <strike>na:1.7.0_72</strike>

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) <strike>tomcat-embed-core-8.0.20.jar!/:8.0.20</strike>

at java.lang.Thread.run(Thread.java:745) <strike>na:1.7.0_72</strike>

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

at redis.clients.util.Pool.getResource(Pool.java:42) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.JedisPool.getResource(JedisPool.java:84) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.JedisPool.getResource(JedisPool.java:10) ~<strike>jedis-2.5.2.jar!/:na</strike>

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:133) ~<strike>spring-data-redis-1.4.2.RELEASE.jar!/:1.4.2.RELEASE</strike>

... 55 common frames omitted

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out

at redis.clients.jedis.Connection.connect(Connection.java:150) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:71) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1783) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:65) ~<strike>jedis-2.5.2.jar!/:na</strike>

at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836) ~<strike>commons-pool2-2.2.jar!/:2.2</strike>

at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434) ~<strike>commons-pool2-2.2.jar!/:2.2</strike>

at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~<strike>commons-pool2-2.2.jar!/:2.2</strike>

at redis.clients.util.Pool.getResource(Pool.java:40) ~<strike>jedis-2.5.2.jar!/:na</strike>

... 58 common frames omitted

Caused by: java.net.SocketTimeoutException: connect timed out

at java.net.PlainSocketImpl.socketConnect(Native Method) ~<strike>na:1.7.0_72</strike>

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~<strike>na:1.7.0_72</strike>

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~<strike>na:1.7.0_72</strike>

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~<strike>na:1.7.0_72</strike>

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~<strike>na:1.7.0_72</strike>

at java.net.Socket.connect(Socket.java:579) ~<strike>na:1.7.0_72</strike>

at redis.clients.jedis.Connection.connect(Connection.java:144) ~<strike>jedis-2.5.2.jar!/:na</strike>

... 65 common frames omitted

我们还不知道这到底是什么原因？

有人可以为我们指出正确的方向并帮助我们确定问题的根本原因和解决方案吗？

Answer 1

在某些版本的 Spring 框架上，Spring 不会在事务完成后关闭 redis 连接，因此连接最终会耗尽。如果您使用 setEnableTransactionSupport(true) 初始化 Jedis 模板，这可能会触发错误。将其设置为 false 应该可以修复它。

如果您需要交易，还有其他解决方法。参见本文“Spring Data Redis中的一个事务陷阱”部分； http://www.javaworld.com/article/3062899/big-data/lightning-fast-nosql-with-spring-data-redis.html