cassandra - com.datastax.driver.core.exceptions.BusyPoolException

Question

每当我在 Cassandra 的表中插入数据时，超过 1000 并通过 id 获取数据，它会引发以下异常:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 (com.datastax.driver.core.exceptions.BusyPoolException: [localhost/127.0.0.1] Pool is busy (no available connection and the queue has reached its max size 256)))
    at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:213)
    at com.datastax.driver.core.RequestHandler.access$1000(RequestHandler.java:49)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:277)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution$1.onFailure(RequestHandler.java:340)
    at com.google.common.util.concurrent.Futures$6.run(Futures.java:1764)
    at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:456)
    at com.google.common.util.concurrent.Futures$ImmediateFuture.addListener(Futures.java:153)
    at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1776)
    at com.google.common.util.concurrent.Futures.addCallback(Futures.java:1713)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.query(RequestHandler.java:299)
    at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:274)
    at com.datastax.driver.core.RequestHandler.startNewExecution(RequestHandler.java:117)
    at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:97)
    at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:132)
    at com.outworkers.phantom.builder.query.CassandraOperations$class.scalaQueryStringToPromise(CassandraOperations.scala:67)
    at com.outworkers.phantom.builder.query.InsertQuery.scalaQueryStringToPromise(InsertQuery.scala:31)
    at com.outworkers.phantom.builder.query.CassandraOperations$class.scalaQueryStringExecuteToFuture(CassandraOperations.scala:31)
    at com.outworkers.phantom.builder.query.InsertQuery.scalaQueryStringExecuteToFuture(InsertQuery.scala:31)
    at com.outworkers.phantom.builder.query.ExecutableStatement$class.future(ExecutableQuery.scala:80)
    at com.outworkers.phantom.builder.query.InsertQuery.future(InsertQuery.scala:31)
    at nd.cluster.data.store.Points.upsert(Models.scala:114)

我已经使用 PoolingOptions 解决了上述问题。

val poolingOptions = new PoolingOptions()
    .setConnectionsPerHost(HostDistance.LOCAL, 1, 200)
    .setMaxRequestsPerConnection(HostDistance.LOCAL, 256)
    .setNewConnectionThreshold(HostDistance.LOCAL, 100).setCoreConnectionsPerHost(HostDistance.LOCAL, 200)

  val builder1 =  ContactPoint.local
    .noHeartbeat()
    .withClusterBuilder(_.withoutJMXReporting()
      .withoutMetrics().withPoolingOptions(poolingOptions)).keySpace("nd")

现在它甚至可以使用 1l 数据。但我不确定它的效率。
有人可以帮我吗？

Answer 1

这意味着您提交了太多请求，而不是等待 future 完成后再提交更多请求。

每个连接的默认最大请求数为 1024。如果所有连接都超过此数量，连接池会将一些请求排入队列，最多 256 个。如果队列已满，则会抛出 BusyPoolException。当然，您可以增加每个连接的最大请求数和每个主机的最大连接数。但真正的解决方案当然是限制你的线程。你可以例如以1000个为一组提交请求，然后等待 future 完成再提交，或者使用信号量来调节待处理请求的总数并确保它们不超过一定数量(理论上，这个数字必须保持不变)低于 num_hosts * max_connections_per_host * max_requests_per_connection – 实际上，我不建议超过 1,000，因为它可能不会为您带来更多吞吐量)。

您可能会发现此链接很有用。

https://github.com/redisson/redisson/issues/438
https://groups.google.com/a/lists.datastax.com/forum/#!topic/java-driver-user/p3CwOL0kNrs http://docs.datastax.com/en/developer/java-driver/3.1/manual/pooling