spring batch Hadoop 失败并出现 org.springframework.beans.factory.BeanNotOfRequiredTypeException

Question

我有一个触发 hadoop 作业的 spring 批处理作业。请在下面查看我的工作配置。当我启 Action 业时，出现以下异常...

配置:

<hdp:job id="mr-my-job" 
        input-path="/data/input/"
        output-path="/data/output/"
        jar-by-class="org.test.Main" 
        mapper="org.test.Test1$Map"
        combiner="org.test.Test1$Combiner"
        reducer="org.test.Test1$ReduceFromCombiner"
        number-reducers="7"
        scope="step" />

异常:

org.springframework.beans.factory.BeanNotOfRequiredTypeException: Bean named 'mr-my-job' must be of type [org.apache.hadoop.mapreduce.Job], but was actually of type [$Proxy5]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:375)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
        at org.springframework.data.hadoop.mapreduce.JobExecutor.findJobs(JobExecutor.java:254)
        at org.springframework.data.hadoop.mapreduce.JobExecutor.startJobs(JobExecutor.java:166)
        at org.springframework.data.hadoop.batch.mapreduce.JobTasklet.execute(JobTasklet.java:90)
        at org.springframework.data.hadoop.batch.mapreduce.JobTasklet$$FastClassBySpringCGLIB$$4805a065.invoke(<generated>)
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
        at org.springframework.data.hadoop.batch.mapreduce.JobTasklet$$EnhancerBySpringCGLIB$$3be21557.execute(<generated>)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
        at $Proxy2.execute(Unknown Source)
        at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406)
        at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330)

---

我将 CGLIB 3.1 添加到类路径中并添加 <bean class="org.springframework.batch.core.scope.StepScope" p:proxyTargetClass="true" />在我的 XML 中，但仍然出现相同的错误。

---

我调试并发现在将类型为 org.springframework.aop.framework.JdkDynamicAopProxy@b57ceeff 的 bean 转换为 org.springframework.aop.framework.JdkDynamicAopProxy@b57ceeff 时，在 JobExecutor 类的下面一行抛出异常。我认为这里没有应用来自 CGLIB 的自动代理，因为它仍在使用 JDK 代理。

JobExecutor 类:

for (String name : jobNames) {
    js.add(beanFactory.getBean(name, Job.class));
}

AbstractBeanFactory 类

return getTypeConverter().convertIfNecessary(bean, requiredType);

请指教。

Answer 1

感谢卢卡·巴索·里奇。从“my-mr-job”中删除 scope="step"有效。