Cassandra 1.1.1 在使用 Hector 1.0.5 插入大量数据时崩溃

Question

我正在使用 Cassandra 1.1.1 和使用 Hector 1.0.5，正在尝试将数据(大量)插入到列族中。在我的程序执行期间，cassandra 服务器崩溃并显示内存不足错误。在那之后，我别无选择，只能退出服务器。对于我试图存储 html 文件内容的一个列族，这种情况会重复出现，但我从未有机会完成它。 html 文件内容从 225 KB 数据到 700 KB 数据不等，我试图插入近 1000 条记录。

在程序中，它抛出以下内容

Exception in thread "main" me.prettyprint.hector.api.exceptions.HectorException: All host pools marked down. Retry burden pushed out to client.
    at me.prettyprint.cassandra.connection.HConnectionManager.getClientFromLBPolicy(HConnectionManager.java:393)
    at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:249)
    at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecuteOperation(ExecutingKeyspace.java:97)
    at me.prettyprint.cassandra.model.MutatorImpl.execute(MutatorImpl.java:243)
    at com.epocrates.soa.rx.util.DiseaseImporter.insertDisease(DiseaseImporter.java:207)
    at com.epocrates.soa.rx.util.DiseaseImporter.batchProcess(DiseaseImporter.java:81)
    at com.epocrates.soa.rx.

util.DiseaseImporter.main(DiseaseImporter.java:37)

在 System.log 中，我找到以下内容

java.io.IOError: java.io.IOException: Map failed
    at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:127)
    at org.apache.cassandra.db.commitlog.CommitLogSegment.freshSegment(CommitLogSegment.java:80)
    at org.apache.cassandra.db.commitlog.CommitLogAllocator.createFreshSegment(CommitLogAllocator.java:244)
    at org.apache.cassandra.db.commitlog.CommitLogAllocator.access$500(CommitLogAllocator.java:49)
    at org.apache.cassandra.db.commitlog.CommitLogAllocator$1.runMayThrow(CommitLogAllocator.java:104)
    at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
    at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.IOException: Map failed
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
    at org.apache.cassandra.db.commitlog.CommitLogSegment.<init>(CommitLogSegment.java:119)
    ... 6 more
Caused by: java.lang.OutOfMemoryError: Map failed
    at sun.nio.ch.FileChannelImpl.map0(Native Method)
    at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
    ... 7 more

java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down
    at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:60)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
    at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
    at org.apache.cassandra.service.StorageProxy.insertLocal(StorageProxy.java:457)
    at org.apache.cassandra.service.StorageProxy.sendToHintedEndpoints(StorageProxy.java:314)
    at org.apache.cassandra.service.StorageProxy$2.apply(StorageProxy.java:119)
    at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:260)
    at org.apache.cassandra.service.StorageProxy.mutate(StorageProxy.java:193)
    at org.apache.cassandra.thrift.CassandraServer.doInsert(CassandraServer.java:637)
    at org.apache.cassandra.thrift.CassandraServer.internal_batch_mutate(CassandraServer.java:587)
    at org.apache.cassandra.thrift.CassandraServer.batch_mutate(CassandraServer.java:595)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3112)
    at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(Cassandra.java:3100)
    at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
    at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
    at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

Answer 1

这意味着您已经用完了将提交日志段映射到的地址空间。

最佳解决方案:升级到 64 位 JVM。

更糟糕的解决方案:在 cassandra.yaml 中，将 commitlog_segment_size_in_mb 和 commitlog_total_space_in_mb 都设置为 16。

这不是第一次出现这种情况。我已开通 https://issues.apache.org/jira/browse/CASSANDRA-4422改进默认设置。