java - 无法在远程分区中并行执行

Question

要求:- 使用 Spring Batch 定义远程分区

Looks like all partitions are not performing parallel execution. As per my understanding, for example if gridSize = 3 then all three partition should execute parallel to slave step(reader\process\writer) class.

    Configurations:-
        TaskExecutor:-
           corePoolSize = 3
           maxPoolSize  = 20
           queueCapacity = 100
       gridSize = 3
       chunkSize = 10

我定义了分区器类，它返回与网格大小相同的executionContext。 ExecutionContext={PART3= 开始=0, PART1=开始=0, PART2= 开始=0}

ReaderClass:-

  public void beforeStep(StepExecution stepExecution) {
    log.info(stepExecution);
  }

  public Scope read(){
     --- some operations---
      return new Scope();
  }

  ProcessClass:-

   public Data process(Scope scope){
      -- some operations --
     return new Data(scope);
   } 

   WriterClass :-

    public void write(List<? extends Data> dataItemsSize){

      -- some operations --
     }

日志:-

            [INFO ] 2019-06-13 09:18:13.389 [Container-28] Reader - beforeStep() reader: stepExecution=StepExecution: id=52175, version=1, name=MatchStep:PART2, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=0, rollbackCount=0, exitDescription=
        [INFO ] 2019-06-13 09:18:13.452 [Container-49] Reader - beforeStep() reader: stepExecution=StepExecution: id=52173, version=1, name=MatchStep:PART3, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=0, rollbackCount=0, exitDescription=
        [INFO ] 2019-06-13 09:18:13.455 [Container-28] Reader - read() - END: scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2855, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.455 [Container-28] Reader - read() - END: scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2841, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.456 [Container-28] Reader - read() - END: scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2851, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.456 [Container-28] Reader - read() - END: scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2863, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.476 [Container-30] Reader - beforeStep() reader: stepExecution=StepExecution: id=52171, version=1, name=MatchStep:PART1, status=STARTED, exitStatus=EXECUTING, readCount=0, filterCount=0, writeCount=0 readSkipCount=0, writeSkipCount=0, processSkipCount=0, commitCount=0, rollbackCount=0, exitDescription=
        [INFO ] 2019-06-13 09:18:13.477 [Container-49] Reader - read() - END: scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2845, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.477 [Container-49] Reader - read() - END: scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2847, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.477 [Container-49] Reader - read() - END: scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2861, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.477 [Container-49] Reader - read() - END: scope=null
        [INFO ] 2019-06-13 09:18:13.483 [Container-49] Process - process(): scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2845, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.483 [Container-28] Process - process(): scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2855, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2857, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2843, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2849, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2853, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2859, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2865, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.506 [Container-30] Reader - read() - END: scope=null
        [INFO ] 2019-06-13 09:18:13.507 [Container-30] Process - process(): scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2857, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.509 [Container-28] Process - process(): scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2841, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.509 [Container-49] Process - process(): scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2847, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.513 [Container-28] Process - process(): scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2851, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.513 [Container-49] Process - process(): scope=Scope{partitionerId=PART3, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2861, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.516 [Container-28] Process - process(): scope=Scope{partitionerId=PART2, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2863, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.593 [Container-30] Process - process(): scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2843, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.598 [Container-30] Process - process(): scope=Scope{partitionerId=PART1, openDate=Thu Jun 13 09:18:13 CDT 2019, execScopeId=2849, execId=1321, iTC=CX}
        [INFO ] 2019-06-13 09:18:13.608 [Container-28] Writer - write(): dataItemsSize=4
        [INFO ] 2019-06-13 09:18:13.608 [Container-49] Writer - write(): dataItemsSize=3

Answer 1

看起来您正在使用 TaskExecutorPartitionHandler 并且根据您的日志，分区由不同线程并行执行(Container-28、Container-30, Container-49) 正如预期的那样。

Requirements:- define remote partition using spring batch

如果要配置远程分区，则需要使用MessageChannelPartitionHandler。您可以找到如何使用它的示例 here .