java.lang.ClassNotFoundException : com. fastxml.jackson.databind.ser.FilterProvider 当 flink 启动

Question

我在 k8s 集群上运行 Flink 1.11 并在尝试更新 log4j-console.properties 文件时收到以下错误:

Starting Task Manager
Enabling required built-in plugins
Linking flink-s3-fs-hadoop-1.11.1.jar to plugin directory
Successfully enabled flink-s3-fs-hadoop-1.11.1.jar
sed: couldn't open temporary file /opt/flink/conf/sedl2dH0X: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedPLYAzY: Read-only file system
/docker-entrypoint.sh: 72: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml: Permission denied
sed: couldn't open temporary file /opt/flink/conf/sede0G5LW: Read-only file system
/docker-entrypoint.sh: 120: /docker-entrypoint.sh: cannot create /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting taskexecutor as a console application on host flink-taskmanager-c765c947c-qx68t.
Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/databind/ser/FilterProvider
    at org.apache.logging.log4j.core.layout.JsonLayout.<init>(JsonLayout.java:158)
    at org.apache.logging.log4j.core.layout.JsonLayout.<init>(JsonLayout.java:69)
    at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:102)
    at org.apache.logging.log4j.core.layout.JsonLayout$Builder.build(JsonLayout.java:77)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:122)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:1002)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:942)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:934)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:552)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:241)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:288)
    at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:579)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:651)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:668)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:253)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:138)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:45)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:30)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:329)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:349)
    at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<clinit>(TaskManagerRunner.java:89)
Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.databind.ser.FilterProvider
    at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown Source)
    at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown Source)
    at java.base/java.lang.ClassLoader.loadClass(Unknown Source)

我的 log4j-console.properties:

rootLogger.level = INFO
#rootLogger.appenderRef.console.ref = ConsoleAppender

appender.console.type = Console
appender.console.name = STDOUT
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n

appender.kafka.type = Kafka
appender.kafka.name = Kafka
appender.kafka.topic = test
appender.kafka.layout.type = JsonLayout
appender.kafka.layout.complete = false
appender.kafka.additional1.type = KeyValuePair
appender.kafka.additional1.key=app
appender.kafka.additional1.value=TEST
appender.kafka.additional2.type = KeyValuePair
appender.kafka.additional2.key=subsystem
appender.kafka.additional2.value=TEST
appender.kafka.additional3.type = KeyValuePair
appender.kafka.additional3.key=deployment
appender.kafka.additional3.value=TEST
appender.kafka.property.bootstrap.servers=***

rootLogger.appenderRef.console.ref = STDOUT
rootLogger.appenderRef.kafka.ref = Kafka

我使用“flink:1.11.1-scala_2.11-java11”docker 镜像并验证所有 log4j2 依赖项都在类路径中。
我还尝试从上面的基本镜像创建一个新的 docker 镜像，并将缺少的依赖项添加到其中，但没有任何 react 。

Answer 1

我也遭受了这个错误。这里的问题是，当任务管理器和作业管理器启动时，它们使用修改后的类路径运行，而不是您通过构建系统构建的 JAR。
查看 constructFlinkClassPath在 flink source code .为了证明这一点，恢复 JSON 日志记录模式并在启动时检查 tm/jm 日志中的类路径。您会注意到您的 JAR 不在类路径中。
要解决此问题，您需要向 jackson-core 提供依赖项(在这种情况下，您需要 jackson-annotations jackson-databind 和 lib) tm/jm 节点中的文件夹(lib 文件夹默认包含在 flink 类路径中)。
如果您使用 docker，则可以在构建容器 ( RUN wget... ) 时执行此操作。