amazon-web-services - 与 Titan over AWS DynamoDB 的索引不一致

Question

我们在我们的一项服务中使用了 Titan 分布式图形数据库。我们有一个使用 CassandraDB 作为存储后端的工作配置。

现在我们正致力于使用 AWS DynamoDB 存储后端运行该服务。如果我们使用 https://github.com/awslabs/dynamodb-titan-storage-backend 在本地模式下运行它一切正常。然而，在迁移到云中的 AWS DynamoDB 之后，我们发现了一些不确定性问题。

我们有在空数据库上运行的单元测试，生成测试数据，执行一些 CRUD，然后清理所有内容。测试通过了 Cassandra。 它们也在本地 DynamoDB 上传递。不幸的是，它们在本地 DynamoDB 和云中的 AWS DynamoDB 上往往会失败。通常在发生故障之前需要运行几次。在这样的失败之后，我们可以在数据库中找到一些幽灵顶点。看起来索引与图形不同步。在结果中，我们可以看到使用索引和不使用索引查询图形时的不同结果。

日志中也有异常。最有趣的是:com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate:

    13:19:12 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet()    13:19:12 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin()    13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit()    13:19:13 [ERROR] [c.h.c.c.h.HystrixDiagnosticLogger] [] HYSTRIX: command category-v1.createCategory failed (com.hybris.caas.category.command.CreateCategoryCommand@7a77b317); execution events: [TIMEOUT, FALLBACK_MISSING]    java.util.concurrent.TimeoutException: null        at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:600) ~[hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:580) ~[hystrix-core-1.5.1.jar:1.5.1]        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:139) ~[rxjava-1.1.1.jar:1.1.1]        at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) [rxjava-1.1.1.jar:1.1.1]        at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) [rxjava-1.1.1.jar:1.1.1]        at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:955) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) [hystrix-core-1.5.1.jar:1.5.1]        at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0]        at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:972) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99) [hystrix-core-1.5.1.jar:1.5.1]        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_102]        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_102]        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]    13:19:13 [ERROR] [c.t.t.g.database.StandardTitanGraph] [] Could not commit transaction [9] due to storage exception in commit    com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) [titan-core-1.0.0.jar:na]        at com.hybris.caas.category.titan.TitanTransactionManager.doCommit(TitanTransactionManager.java:126) [classes/:na]        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at com.sun.proxy.$Proxy41.createCategory(Unknown Source) [na:na]        at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:81) [classes/:na]        at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:35) [classes/:na]        at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:288) [hystrix-core-1.5.1.jar:1.5.1]        at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1]        at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:497) [hystrix-core-1.5.1.jar:1.5.1]        at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1]        at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.5.1.jar:1.5.1]        at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0]        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.5.1.jar:1.5.1]        at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1]        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]    Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent exception while executing backend operation CacheMutation        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na]        ... 40 common frames omitted    Caused by: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate        at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.parallelMutate(DynamoDBDelegate.java:301) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na]        at com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager.mutateMany(DynamoDBStoreManager.java:194) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na]        ... 41 common frames omitted    13:19:13 [INFO ] [c.h.c.c.c.w.m.TimeoutExceptionMapper] [] Mapping TimeoutException    java.util.concurrent.TimeoutException: null        at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:600) ~[hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.AbstractCommand$9.call(AbstractCommand.java:580) ~[hystrix-core-1.5.1.jar:1.5.1]        at rx.internal.operators.OperatorOnErrorResumeNextViaFunction$4.onError(OperatorOnErrorResumeNextViaFunction.java:139) ~[rxjava-1.1.1.jar:1.1.1]        at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.1.jar:1.1.1]        at rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71) ~[rxjava-1.1.1.jar:1.1.1]        at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$1.run(AbstractCommand.java:955) ~[hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:41) ~[hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable$1.call(HystrixContextRunnable.java:37) ~[hystrix-core-1.5.1.jar:1.5.1]        at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) ~[service-sdk-logging-4.7.0.jar:4.7.0]        at com.netflix.hystrix.strategy.concurrency.HystrixContextRunnable.run(HystrixContextRunnable.java:57) ~[hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.AbstractCommand$HystrixObservableTimeoutOperator$2.tick(AbstractCommand.java:972) ~[hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.util.HystrixTimer$1.run(HystrixTimer.java:99) ~[hystrix-core-1.5.1.jar:1.5.1]        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_102]        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) ~[na:1.8.0_102]        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_102]        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) ~[na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_102]        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]    13:19:13 [ERROR] [c.t.t.g.database.StandardTitanGraph] [] Could not commit transaction [9] due to exception    com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) [titan-core-1.0.0.jar:na]        at com.hybris.caas.category.titan.TitanTransactionManager.doCommit(TitanTransactionManager.java:126) [classes/:na]        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at com.sun.proxy.$Proxy41.createCategory(Unknown Source) [na:na]        at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:81) [classes/:na]        at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:35) [classes/:na]        at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:288) [hystrix-core-1.5.1.jar:1.5.1]        at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1]        at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:497) [hystrix-core-1.5.1.jar:1.5.1]        at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1]        at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.5.1.jar:1.5.1]        at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0]        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.5.1.jar:1.5.1]        at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1]        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]    Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent exception while executing backend operation CacheMutation        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na]        ... 40 common frames omitted    Caused by: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate        at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.parallelMutate(DynamoDBDelegate.java:301) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na]        at com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager.mutateMany(DynamoDBStoreManager.java:194) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na]        ... 41 common frames omitted    13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doRollback()    13:19:13 [ERROR] [c.h.c.c.t.TitanTransactionManager] [] Commit exception overridden by rollback exception    com.thinkaurelius.titan.core.TitanException: Could not commit transaction due to exception during persistence        at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1363) ~[titan-core-1.0.0.jar:na]        at com.hybris.caas.category.titan.TitanTransactionManager.doCommit(TitanTransactionManager.java:126) ~[classes/:na]        at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:761) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:730) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:485) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:291) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) [spring-tx-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208) [spring-aop-4.2.5.RELEASE.jar:4.2.5.RELEASE]        at com.sun.proxy.$Proxy41.createCategory(Unknown Source) [na:na]        at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:81) [classes/:na]        at com.hybris.caas.category.command.CreateCategoryCommand.run(CreateCategoryCommand.java:35) [classes/:na]        at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:293) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:288) [hystrix-core-1.5.1.jar:1.5.1]        at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:162) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable$2.call(Observable.java:154) [rxjava-1.1.1.jar:1.1.1]        at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1]        at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:521) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.AbstractCommand$5.call(AbstractCommand.java:497) [hystrix-core-1.5.1.jar:1.5.1]        at rx.Observable.unsafeSubscribe(Observable.java:8314) [rxjava-1.1.1.jar:1.1.1]        at rx.internal.operators.OperatorSubscribeOn$1.call(OperatorSubscribeOn.java:94) [rxjava-1.1.1.jar:1.1.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:56) [hystrix-core-1.5.1.jar:1.5.1]        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction$1.call(HystrixContexSchedulerAction.java:47) [hystrix-core-1.5.1.jar:1.5.1]        at com.sap.cloud.yaas.servicesdk.logging.DelegatingMDCCallable.call(DelegatingMDCCallable.java:58) [service-sdk-logging-4.7.0.jar:4.7.0]        at com.netflix.hystrix.strategy.concurrency.HystrixContexSchedulerAction.call(HystrixContexSchedulerAction.java:69) [hystrix-core-1.5.1.jar:1.5.1]        at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55) [rxjava-1.1.1.jar:1.1.1]        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_102]        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_102]        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_102]        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]    Caused by: com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352) ~[titan-core-1.0.0.jar:na]        ... 34 common frames omitted    Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent exception while executing backend operation CacheMutation        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42) ~[titan-core-1.0.0.jar:na]        ... 40 common frames omitted    Caused by: com.amazon.titan.diskstorage.dynamodb.BackendRuntimeException: was interrupted during parallelMutate        at com.amazon.titan.diskstorage.dynamodb.DynamoDBDelegate.parallelMutate(DynamoDBDelegate.java:301) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na]        at com.amazon.titan.diskstorage.dynamodb.DynamoDBStoreManager.mutateMany(DynamoDBStoreManager.java:194) ~[dynamodb-titan100-storage-backend-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87) ~[titan-core-1.0.0.jar:na]        at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56) ~[titan-core-1.0.0.jar:na]        ... 41 common frames omitted    13:19:13 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/CategoryCreation.json    13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet()    13:19:13 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin()    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit()    13:19:14 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/AssignmentCreation.json    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet()    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin()    13:19:14 [INFO ] [c.h.c.c.r.impl.utils.ReferenceVertex] [] Creating vertex for com.hybris.caas.category.model.Reference@2accc739[id=NEW,type=product,url=product-service.dev.cf.hybris.com/NEW]    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit()    13:19:14 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/AssignmentCreation.json    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet()    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin()    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doRollback()    13:19:14 [INFO ] [c.h.c.c.c.DeleteCategoriesCommand] [] [AUDIT [ remove_all_categories ]] [user=User[name=,roles=[hybris.category_delete_all]], tenant=womdev1s] Delete all categories.    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet()    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin()    13:19:14 [INFO ] [c.h.c.c.r.i.GraphCategoryRepository] [] Deleting all categories in tenant 'womdev1s'    13:19:14 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit()    13:19:16 [INFO ] [c.h.c.c.j.CachedSchemaResolver] [] Resolve schema: resource:/api/schema/CategoryCreation.json    13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet()    13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin()    13:19:16 [INFO ] [c.h.c.c.r.impl.utils.CategoryVertex] [] Creating root category vertex for tenant 'womdev1s'    13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit()    13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doGet()    13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doBegin()    13:19:16 [INFO ] [c.h.c.c.t.TitanTransactionManager] [] TRANSACTION - doCommit()    ......

我知道由于“最终一致性”，我们应该预料到一些问题。 CassandraDB 和 DynamoDB 似乎最终是一致的，但在我们的例子中，行为明显不同。我们还使用了“storage.dynamodb.force-consistent-read=true”标志。

有没有人已经尝试过面对这样的问题？

Answer 1

您遇到的索引一致性问题可能会通过引入 Titan/JanusGraph based locking 得到纠正(practical example)。无论后端选择如何，幽灵顶点始终是 Titan/JanusGraph 中的一个主题，这方面的指导是让 OLAP 作业扫描图形并消除不一致。