scala - 'new HiveContext' 需要 X11 显示？ com.trend.iwss.jscan?

Question

Spark 1.6.2( yarn 大师)

包名:com.example.spark.Main

---

基本的SparkSQL代码

val conf = new SparkConf()
conf.setAppName("SparkSQL w/ Hive")
val sc = new SparkContext(conf)

val hiveContext = new HiveContext(sc)
import hiveContext.implicits._

// val rdd = <some RDD making>
val df = rdd.toDF()
df.write.saveAsTable("example")

还有堆栈跟踪...

No X11 DISPLAY variable was set, but this program performed an operation which requires it.
         at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
         at java.awt.Window.<init>(Window.java:536)
         at java.awt.Frame.<init>(Frame.java:420)
         at java.awt.Frame.<init>(Frame.java:385)
         at com.trend.iwss.jscan.runtime.BaseDialog.getActiveFrame(BaseDialog.java:75)
         at com.trend.iwss.jscan.runtime.AllowDialog.make(AllowDialog.java:32)
         at com.trend.iwss.jscan.runtime.PolicyRuntime.showAllowDialog(PolicyRuntime.java:325)
         at com.trend.iwss.jscan.runtime.PolicyRuntime.stopActionInner(PolicyRuntime.java:240)
         at com.trend.iwss.jscan.runtime.PolicyRuntime.stopAction(PolicyRuntime.java:172)
         at com.trend.iwss.jscan.runtime.PolicyRuntime.stopAction(PolicyRuntime.java:165)
         at com.trend.iwss.jscan.runtime.NetworkPolicyRuntime.checkURL(NetworkPolicyRuntime.java:284)
         at com.trend.iwss.jscan.runtime.NetworkPolicyRuntime._preFilter(NetworkPolicyRuntime.java:164)
         at com.trend.iwss.jscan.runtime.PolicyRuntime.preFilter(PolicyRuntime.java:132)
         at com.trend.iwss.jscan.runtime.NetworkPolicyRuntime.preFilter(NetworkPolicyRuntime.java:108)
         at org.apache.commons.logging.LogFactory$5.run(LogFactory.java:1346)
         at java.security.AccessController.doPrivileged(Native Method)
         at org.apache.commons.logging.LogFactory.getProperties(LogFactory.java:1376)
         at org.apache.commons.logging.LogFactory.getConfigurationFile(LogFactory.java:1412)
         at org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:455)
         at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
         at org.apache.hadoop.hive.shims.HadoopShimsSecure.<clinit>(HadoopShimsSecure.java:60)
         at java.lang.Class.forName0(Native Method)
         at java.lang.Class.forName(Class.java:264)
         at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:146)
         at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:141)
         at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:100)
         at org.apache.spark.sql.hive.client.ClientWrapper.overrideHadoopShims(ClientWrapper.scala:116)
         at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:69)
         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
         at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
         at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
         at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:249)
         at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:345)
         at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:255)
         at org.apache.spark.sql.hive.HiveContext.setConf(HiveContext.scala:459)
         at org.apache.spark.sql.hive.HiveContext.defaultOverrides(HiveContext.scala:233)
         at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:236)
         at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
         at com.example.spark.Main1$.main(Main.scala:52)
         at com.example.spark.Main.main(Main.scala)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:498)
         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738)
         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
 ivysettings.xml file not found in HIVE_HOME or HIVE_CONF_DIR,/etc/hive/2.5.3.0-37/0/ivysettings.xml will be used

---

同样的代码一周前在一个新的 HDP 集群上运行，它在沙箱中运行良好......我记得唯一做的就是试图改变 JAVA_HOME 变量，但我很确定我取消了那些变化。
我不知所措 - 不确定如何开始追踪问题。

集群是 headless 的，所以它当然没有 X11 显示，但是 new HiveContext 的哪一部分？甚至需要弹出任何 JFrame ？

根据日志，我会说这是一个 Java 配置问题，我搞砸了，而且在 org.apache.hadoop.hive.shims.HadoopShimsSecure.<clinit>(HadoopShimsSecure.java:60) 中。被触发，因此出现 Java 安全对话框，但我不知道。

无法进行X11 转发，并尝试进行export SPARK_OPTS="-Djava.awt.headless=true"在 spark-submit 之前，但这没有帮助。

这些都试过了，还是不能转发，没有显示

• Getting a HeadlessException: No X11 DISPLAY variable was set
• "No X11 DISPLAY variable" - what does it mean?

该错误似乎可以在两个 Spark 客户端上重现。
我只在一台机器上尝试更改 JAVA_HOME .

进行了 Ambari Hive 服务检查。没修好。
可以通过 Hive/Beeline CLI 正常连接到 Hive 数据库

Answer 1

就 Spark 代码而言，这似乎减轻了错误。

val conf = SparkConf()
conf.set("spark.executor.extraJavaOptions" , "-Djava.awt.headless=true")

---

原始答案

找到这篇文章。 Spring 3.0.5 - java.awt.HeadlessException - com.trend.iwss.jscan

基本上，趋势科技将一些 com.trend.iwss.jscan 包插入到通过公司防火墙通过 Maven 下载的 JAR 文件中，我无法控制它。

(链接无效)http://esupport.trendmicro.com/Pages/IWSx-3x-Some-files-and-folders-are-added-to-the-Jar-files-after-passin.aspx

Wayback Machine 来拯救...

如果其他人有意见，我也想听听。

Problem:
When downloading some .JAR files via IWSA, a directory filled with .class file, which is not related to what is being downloaded, is added to the jar file (com\trend\iwss\jscan\runtime\).

Solution:
This happens because if a JAR file is originally unsigned, IWSA will insert some code into the applet to monitor and restrict potential harmful actions.

For IWSS/IWSA, every "get" request is the same so it will not know if you are trying to download an archive or an applet, which will be executed by your browser.

This code is added for security reason to monitor the behavior of the "possible" applet to be sure that it does not do any harm to the machine and its environment.

To prevent this issue, please follow these steps:

1. Log on to the IWSS web console.

2. Go to HTTP > Applets and ActiveX > Policies > Java Applet Security Rules.

3. Under Java Applet Security, change the value of "No signature" to either "Pass" or "Block", depending on what you want to do with the unsigned .JAR files.

4. Click Save.