r - 避免在运行时在 HDP 上修改 mapred.child.env，以便 R 可以使用 RHive 建立与 hiveserver2 的连接

Question

我正在尝试让 R 的 RHive 包与 hiveserver2 很好地通信。

我在尝试使用以下方式连接到 hiveserver2 时收到错误消息:

>rhive.connect(host="localhost",port=10000, hiveServer2=TRUE, user="root", password="hadoop")

初始运行的输出:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/hadoop/client/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/hadoop/client/slf4j-log4j12.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/hive/lib/hive-jdbc-0.14.0.2.2.0.0-2041-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/2.2.0.0-2041/hive/lib/hive-jdbc.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/03/19 07:08:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/03/19 07:08:23 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
15/03/19 07:08:24 INFO jdbc.Utils: Supplied authorities: localhost:10000
15/03/19 07:08:24 INFO jdbc.Utils: Resolved authority: localhost:10000
15/03/19 07:08:24 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default

这会导致错误:

错误:org.apache.hive.service.cli.HiveSQLException:处理语句时出错:无法在运行时修改 mapred.child.env。它不在允许在运行时修改的参数列表中

在同一命令的后续运行中，输出减少为:

15/03/19 07:16:24 INFO jdbc.Utils: Supplied authorities: localhost:10000
15/03/19 07:16:24 INFO jdbc.Utils: Resolved authority: localhost:10000
15/03/19 07:16:24 INFO jdbc.HiveConnection: Will try to open client transport with JDBC Uri: jdbc:hive2://localhost:10000/default
Error: org.apache.hive.service.cli.HiveSQLException: Error while processing statement: Cannot modify mapred.child.env at runtime. It is not in list of params that are allowed to be modified at runtime

这向我表明我可能在某处没有足够的权限...但是，我正在使用 root 运行它。所以，我不确定我缺少哪些权限...

我已经使用安装指南安装了 RHive via README .

注意:如果我使用 CRAN version of the package 也会出现同样的错误.

我目前正在使用 Hortonworks Data Platform 2.2 (HDP 2.2)的虚拟框图像。结果，hadoop 和 hiveserver2 已经安装好了。我已经安装了 R 版本 3.1.2。

以下是我安装 RHive 的方式:

# Set up paths for HIVE_HOME, HADOOP_HOME, and HADOOP_CONF
export HIVE_HOME=/usr/hdp/2.2.0.0-2041/hive

export HADOOP_HOME=/usr/hdp/2.2.0.0-2041/hadoop

export HADOOP_CONF_DIR=/etc/hadoop/conf

# R Location via RHOME
R_HOME=/usr/lib64/R

# Place R_HOME into hadoop config location
sudo sh -c "echo \"R_HOME='$R_HOME'\" >> $HADOOP_HOME/conf/hadoop-env.sh"

# Add remote enable to Rserve config.
sudo sh -c "echo 'remote enable' >> /etc/Rserv.conf"

# Launch the daemon
R CMD Rserve

# Confirm launch
netstat -nltp

# Install ant to build java files
sudo yum -y install ant

# Install package dependencies
sudo R --no-save << EOF
install.packages( c('rJava','Rserve','RUnit'), repos='http://cran.us.r-project.org', INSTALL_opts=c('--byte-compile') )
EOF

# Install RHive package
git clone https://github.com/nexr/RHive.git
cd RHive
ant build
sudo R CMD INSTALL RHive

要检查打开 R 并使用 EOF 之间的语句或直接从 shell 运行命令:

sudo R --no-save << EOF
Sys.setenv(HIVE_HOME="/usr/hdp/2.2.0.0-2041/hive")
Sys.setenv(HADOOP_HOME="/usr/hdp/2.2.0.0-2041/hadoop")
Sys.setenv(HADOOP_CONF_DIR="/etc/hadoop/conf")
library(RHive)
rhive.connect(host="localhost",port=10000, hiveServer2=TRUE, user="root", password="hadoop")
EOF

Answer 1

答案在此 link 中提到.

基本上，您必须在/etc/hive/conf/hive-site.xml 中添加值为“mapred.child.env”的属性“hive.security.authorization.sqlstd.confwhitelist.append”

此解决方案对我有用，但我使用 Ambari UI 来更改此配置。