hadoop - 如何在写入文件时禁用 Hive 中的日志

Question

我有一个用例，我正在执行配置单元查询并将输出存储到文件中。

hive -S -e "SELECT * from test.employee where empid=1"  > /mapr/Piyush/test/output.txt

查询执行正常，但我也收到日志以及文件中的数据。我猜这是因为 log4j 属性。这里的问题是我无权访问 log4j 配置文件，因此我无法对其进行任何更改。我尝试设置几个配置。

set hive.root.logger=ERROR, console

和

set hive.root.logger=INFO,console

和

set hive.server2.logging.operation.enabled=false

但没有任何帮助。我只需要文件中表中的数据。如果我遗漏了任何内容或此问题的任何解决方案，请告诉我。

示例输出文件

2017-06-09 11:49:18,265 main ERROR Cannot access RandomAccessFile {}) java.io.FileNotFoundException: /mapr/pankaj-hive.log (Permission denied)
2017-06-09 11:49:18,272 main ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender for element RollingRandomAccessFile. java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:813)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:753)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:745)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:389)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:169)
    at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:158)
    at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:43)
    at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:149)
    at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:46)
    at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:413)
    at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:257)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:519)
    at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:536)
    at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:214)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:146)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:41)
    at org.apache.logging.log4j.LogManager.getContext(LogManager.java:185)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:103)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
    at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:42)
    at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
    at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284)
    at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:155)
    at org.apache.commons.logging.impl.SLF4JLogFactory.getInstance(SLF4JLogFactory.java:132)
    at org.apache.commons.logging.LogFactory.getLog(LogFactory.java:655)
    at org.apache.hadoop.util.ShutdownHookManager.<clinit>(ShutdownHookManager.java:44)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:200)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$RollingRandomAccessFileManagerFactory@48aca48b] unable to create manager for [/mapr/pankaj-hive.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$FactoryData@13fd2ccd]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:61)
    at org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager.getRollingRandomAccessFileManager(RollingRandomAccessFileManager.java:84)
    at org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender.createAppender(RollingRandomAccessFileAppender.java:206)
    ... 33 more

2017-06-09 11:49:18,277 main ERROR Null object returned for RollingRandomAccessFile in Appenders.
2017-06-09 11:49:18,281 main ERROR Unable to locate appender "DRFA" for logger config "root"
2017-06-09 11:49:19,640 main ERROR Cannot access RandomAccessFile {}) java.io.FileNotFoundException: /mapr/pankaj-hive.log (Permission denied)
2017-06-09 11:49:19,644 main ERROR Unable to invoke factory method in class class org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender for element RollingRandomAccessFile. java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.logging.log4j.core.config.plugins.util.PluginBuilder.build(PluginBuilder.java:136)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createPluginObject(AbstractConfiguration.java:813)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:753)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.createConfiguration(AbstractConfiguration.java:745)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.doConfigure(AbstractConfiguration.java:389)
    at org.apache.logging.log4j.core.config.AbstractConfiguration.initialize(AbstractConfiguration.java:169)
    at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:158)
    at org.apache.logging.log4j.core.config.builder.impl.DefaultConfigurationBuilder.build(DefaultConfigurationBuilder.java:43)
    at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:149)
    at org.apache.logging.log4j.core.config.properties.PropertiesConfigurationFactory.getConfiguration(PropertiesConfigurationFactory.java:46)
    at org.apache.logging.log4j.core.config.ConfigurationFactory.getConfiguration(ConfigurationFactory.java:236)
    at org.apache.logging.log4j.core.config.ConfigurationFactory$Factory.getConfiguration(ConfigurationFactory.java:445)
    at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:228)
    at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:140)
    at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:113)
    at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:98)
    at org.apache.logging.log4j.core.config.Configurator.initialize(Configurator.java:156)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jDefault(LogUtils.java:155)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:91)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4jCommon(LogUtils.java:83)
    at org.apache.hadoop.hive.common.LogUtils.initHiveLog4j(LogUtils.java:66)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:661)
    at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:646)
    at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:641)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.IllegalStateException: ManagerFactory [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$RollingRandomAccessFileManagerFactory@48aca48b] unable to create manager for [/mapr/pankaj-hive.log] with data [org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager$FactoryData@bff34c6]
    at org.apache.logging.log4j.core.appender.AbstractManager.getManager(AbstractManager.java:73)
    at org.apache.logging.log4j.core.appender.OutputStreamManager.getManager(OutputStreamManager.java:61)
    at org.apache.logging.log4j.core.appender.rolling.RollingRandomAccessFileManager.getRollingRandomAccessFileManager(RollingRandomAccessFileManager.java:84)
    at org.apache.logging.log4j.core.appender.RollingRandomAccessFileAppender.createAppender(RollingRandomAccessFileAppender.java:206)
    ... 34 more

2017-06-09 11:49:19,649 main ERROR Null object returned for RollingRandomAccessFile in Appenders.
2017-06-09 11:49:19,652 main ERROR Unable to locate appender "DRFA" for logger config "root"
1   piyush  bangalore   20

我想要的输出只是

1   piyush  bangalore   20

Answer 1

您好，根据配置单元文档您不能通过“设置”命令设置 hive.root.logger 来更改日志记录属性，因为配置单元在初始化时读取记录器属性，即在打开 cli 之前。

你可以这样做:

hive --hiveconf hive.root.logger=OFF -S -e "SELECT * from test.employee where empid=1">/mapr/Piyush/test/output.txt

P.S 我假设您不需要任何日志记录。您可以根据需要更改日志记录级别。

下面我附上带有链接的官方文档截图

Click for official Documentation