redis - 使用 apache beam sdk 时无法连接到 Redis 服务器

Question

所以我有一个数据流作业在做

p.apply(RedisIO.read()
    .withEndpoint(<public endpoint>, 6379)
    .withAuth(<password>)
    .withTimeout(60000)
    .withKeyPattern("UID*"))
 .apply(ParDo.of(new Format()))
 .apply(TextIO.write().to(options.getOutput()));

redis 端点是使用没有防火墙设置的密码进行公共(public)身份验证的。当我运行上面的命令时，出现以下错误。

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven- 
plugin:1.6.0:java (default-cli) on project word-count-beam: An 
exception occured while executing the Java class. 
org.apache.beam.sdk.util.UserCodeException: 
redis.clients.jedis.exceptions.JedisConnectionException: 
java.net.ConnectException: Connection refused -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to 
execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java (default- 
cli) on project word-count-beam: An exception occured while executing 
the Java class. org.apache.beam.sdk.util.UserCodeException: 
redis.clients.jedis.exceptions.JedisConnectionException: 
java.net.ConnectException: Connection refused
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:212)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:307)
at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:193)
at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:106)
at org.apache.maven.cli.MavenCli.execute(MavenCli.java:863)
at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main(MavenCli.java:199)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: An exception 
occured while executing the Java class. 
org.apache.beam.sdk.util.UserCodeException: redis.clients.jedis.exceptions.JedisConnectionException: java.net.ConnectException: Connection refused
at org.codehaus.mojo.exec.ExecJavaMojo.execute(ExecJavaMojo.java:339)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:207)
... 20 more
Caused by: org.apache.beam.runners.direct.repackaged.com.google.common.util.concurrent 
.UncheckedExecutionException: 
org.apache.beam.sdk.util.UserCodeException: 
redis.clients.jedis.exceptions.JedisConnectionException: 
java.net.ConnectException: Connection refused

因此未与公共(public) redis 端点建立连接。当我通过 DirectRunner 运行时，我遇到了同样的错误。我在这里遗漏了什么吗？

Answer 1

RedisIO 的 Apache Beam 源代码中存在一个已知错误，其中 withEndpoint 会忽略输入主机并将尝试使用本地主机。在没有本地主机的情况下尝试连接到本地主机上的 Redis 服务器会出现您所看到的错误。

您可以阅读有关问题的更多信息 here ，并查看带有修复的拉取请求 here .

在该拉取请求被合并之前，您应该能够通过复制 RedisIO.java 自己实现更改来解决问题。进入你的项目并改变

.setConnectionConfiguration(connectionConfiguration().withHost(host))
.setConnectionConfiguration(connectionConfiguration().withPort(port))

到

.setConnectionConfiguration(connectionConfiguration().withHost(host).withPort(port))

请注意，同样的错误在 RedisIO 中出现了 3 次，一次是针对 Read(第 168 行)、ReadAll(第 233 行)和 Write(第 365 行)。