gpt4 book ai didi

java - 在 hadoop 上运行 JNI 的 Apache Pig 无法在多个 map 上加载共享对象(但仅在一个 map 上运行良好)

转载 作者:可可西里 更新时间:2023-11-01 16:34:11 26 4
gpt4 key购买 nike

<分区>

所以我们已经编写了一个 JNI,然后我们编写了一个 UDF 以便能够从 Apache Pig 调用我们的 JNI。现在的问题是我们得到这个错误,但只有当我们用多个 map 运行它时(只有一个 map 一切正常):

Native code library failed to load. java.lang.UnsatisfiedLinkError:
Can't load library: /usr/lib/libdclassjni.so

我认为这与 following piece in Apache Pig's manual : 有关

One problem that users run into is when they make assumption about how many times a constructor for their UDF is called. For instance, they might be creating side files in the store function and doing it in the constructor seems like a good idea. The problem with this approach is that in most cases Pig instantiates functions on the client side to, for instance, examine the schema of the data.

Users should not make assumptions about how many times a function is instantiated; instead, they should make their code resilient to multiple instantiations. For instance, they could check if the files exist before creating them.

在 UDF 中,我们有一个构造函数,它依次实例化另一个类,而那个类确实如此

System.load("/usr/lib/libdclassjni.so").

我可以保证我们在所有节点上都安装了 lidbclassjni.so。所以主要问题是为什么 Apache Pig 无法加载共享对象。此外,UDF 应该如何看待才能避免这个问题。

TL;DR =>

在 hadoop 上的 Apahce Pig UDF 中使用 JNI。在一张 map 上工作正常,但在多张 map 上我得到

Native code library failed to load. java.lang.UnsatisfiedLinkError: Can't load library: /usr/lib/libdclassjni.so

感谢您对此的任何见解

更新:

固定

26 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com