java - Hadoop map/reduce 显示错误 : com. mongodb.connection.SslSettings$Builder.context(Ljavax/net/ssl/SSLContext;)

Question

我定期运行 map/reduce 作业，最近我收到了这种日志:

17/09/27 10:38:03 INFO mapreduce.Job: map 27% reduce 0%
17/09/27 10:38:07 INFO mapreduce.Job: Task Id :
attempt_1494368272823_5583_m_000014_0, Status : FAILED Error: com.mongodb.connection.SslSettings$Builder.context(Ljavax/net/ssl/SSLContext;)Lcom/mongodb/connection/SslSettings$Builder;
17/09/27 10:38:08 INFO mapreduce.Job: map 30% reduce 0%
17/09/27 10:38:10 INFO mapreduce.Job: map 30% reduce 6%
17/09/27 10:38:43 INFO mapreduce.Job: map 32% reduce 6%
17/09/27 10:38:45 INFO mapreduce.Job: map 33% reduce 6%
17/09/27 10:38:46 INFO mapreduce.Job: map 33% reduce 7%
17/09/27 10:38:49 INFO mapreduce.Job: map 35% reduce 7%
17/09/27 10:38:52 INFO mapreduce.Job: map 35% reduce 8%

17/09/27 10:39:15 INFO mapreduce.Job: Task Id : attempt_1494368272823_5583_m_000008_0, Status : FAILED Error: com.mongodb.connection.SslSettings$Builder.context(Ljavax/net/ssl/SSLContext;)Lcom/mongodb/connection/SslSettings$Builder;
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/09/27 10:39:16 INFO mapreduce.Job: map 40% reduce 10%

结束于:

17/09/27 10:44:12 INFO mapreduce.Job: Job job_1494368272823_5583 failed with state FAILED due to: Task failed task_1494368272823_5583_m_000025 Job failed as tasks failed. failedMaps:1 failedReduces:0

17/09/27 10:44:12 INFO mapreduce.Job: Counters: 41 File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=5481350
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=3430808431
HDFS: Number of bytes written=2495944
HDFS: Number of read operations=5540
HDFS: Number of large read operations=0
HDFS: Number of write operations=118 Job Counters
Failed map tasks=21
Killed map tasks=9
Killed reduce tasks=1
Launched map tasks=47
Launched reduce tasks=1
Other local map tasks=20
Data-local map tasks=26
Rack-local map tasks=1
Total time spent by all maps in occupied slots (ms)=34635900
Total time spent by all reduces in occupied slots (ms)=3707820
Total time spent by all map tasks (ms)=3463590
Total time spent by all reduce tasks (ms)=370782
Total vcore-milliseconds taken by all map tasks=3463590
Total vcore-milliseconds taken by all reduce tasks=370782
Total megabyte-milliseconds taken by all map tasks=34635900000
Total megabyte-milliseconds taken by all reduce tasks=3707820000
Map-Reduce Framework
Map input records=18
Map output records=9722
Map output bytes=2572358
Map output materialized bytes=2608684
Input split bytes=3387
Combine input records=0
Spilled Records=9722
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=62408
CPU time spent (ms)=1809020
Physical memory (bytes) snapshot=56575217664
Virtual memory (bytes) snapshot=155695132672
Total committed heap usage (bytes)=60439920640
File Input Format Counters
Bytes Read=0
MapReduce failed with 1

有时作业实际上会成功完成。有时就像上面那样。日志中没有提供更多信息。对于导致此失败的到底发生了什么，我将不胜感激？据我所知，这似乎不是超时或内存问题。集群没有重载(我检查过)...

编辑:进一步调试后，当我连接到 MongoDB 时，问题似乎出现在“connect”方法中。但是为什么会这样呢？当我尝试手动连接时连接成功，并且有时它可以正常工作。

Answer 1

我想您使用的是 MongoClient 实例。

在这种情况下，请验证与 SSLContext 相关的参数是否按照以下引用中所述进行了设置: http://mongodb.github.io/mongo-java-driver/3.6/driver/tutorials/ssl/

“Java 安全套接字扩展 (JSSE) 引用指南”将是了解 SSLContext 的好书: https://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html

您会发现大量关于创建 SSLContext 实例的示例。

否则，很可能您遇到了这个问题: https://jira.mongodb.org/browse/JAVA-2590

摘自该链接:

an older version of mongodb-driver-core is being pulled in

请清理你的依赖并使用最新的。