- android - RelativeLayout 背景可绘制重叠内容
- android - 如何链接 cpufeatures lib 以获取 native android 库?
- java - OnItemClickListener 不起作用,但 OnLongItemClickListener 在自定义 ListView 中起作用
- java - Android 文件转字符串
我构建了一个 hadoop 和 hive 集群并尝试做一些测试。但它真的很慢。
表格
表 value_count
+--------------------------------------------------------------+--+
| createtab_stmt |
+--------------------------------------------------------------+--+
| CREATE TABLE `value_count`( |
| `key` int, |
| `count` int, |
| `create_date` date COMMENT '????') |
| COMMENT 'This is a group table' |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
| LOCATION |
| 'hdfs://avatarcluster/hive/warehouse/test.db/value_count' |
| TBLPROPERTIES ( |
| 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}', |
| 'numFiles'='7', |
| 'numRows'='7', |
| 'rawDataSize'='448', |
| 'totalSize'='2297', |
| 'transient_lastDdlTime'='1496217645') |
+--------------------------------------------------------------+--+
sql执行
insert into value_count values (5, 1, '2017-05-06');
这个sql我已经执行了好几次了,每次大概需要4、5分钟
hadoop 容器日志
2017-05-31 16:00:45,041 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Central Dispatcher queue size after DAG completion, before cleanup: 0
2017-05-31 16:00:45,041 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Waiting for next DAG to be submitted.
2017-05-31 16:00:45,042 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Cleaning up DAG: name=insert into value_count valu...'2017-05-06')(Stage-1), with id=dag_1490688643958_53401_1
2017-05-31 16:00:45,042 [INFO] [Dispatcher thread {Central}] |container.AMContainerMap|: Cleaned up completed containers on dagComplete. Removed=0, Remaining=1
2017-05-31 16:00:45,044 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Completed cleanup for DAG: name=insert into value_count valu...'2017-05-06')(Stage-1), with id=dag_1490688643958_53401_1
2017-05-31 16:00:50,749 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_1490688643958_53401_01_000002, containerExpiryTime=1496217650577, idleTimeout=5000, taskRequestsCount=0, heldContainers=1, delayedContainers=0, isNew=false
2017-05-31 16:00:50,752 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1490688643958_53401_1][Event:CONTAINER_STOPPED]: containerId=container_1490688643958_53401_01_000002, stoppedTime=1496217650751, exitStatus=0
2017-05-31 16:00:50,753 [INFO] [ContainerLauncher #1] |launcher.TezContainerLauncherImpl|: Stopping container_1490688643958_53401_01_000002
2017-05-31 16:00:50,753 [INFO] [ContainerLauncher #1] |impl.ContainerManagementProtocolProxy|: Opening proxy : app08.hp.sp.tst.bmsre.com:51640
2017-05-31 16:00:51,628 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_1490688643958_53401_01_000002 exited with diagnostics set to Container failed, exitCode=-105. Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
2017-05-31 16:01:29,678 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 51 lastPreemptionHeartbeat: 50
2017-05-31 16:02:19,740 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 101 lastPreemptionHeartbeat: 100
2017-05-31 16:03:09,801 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 151 lastPreemptionHeartbeat: 150
2017-05-31 16:03:59,858 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 201 lastPreemptionHeartbeat: 200
2017-05-31 16:04:49,915 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 251 lastPreemptionHeartbeat: 250
2017-05-31 16:05:39,971 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 301 lastPreemptionHeartbeat: 300
2017-05-31 16:06:09,581 [INFO] [DAGSubmissionTimer] |rm.TaskSchedulerManager|: TaskScheduler notified that it should unregister from RM
2017-05-31 16:06:09,581 [INFO] [DAGSubmissionTimer] |app.DAGAppMaster|: No current running DAG, shutting down the AM
2017-05-31 16:06:09,581 [INFO] [DAGSubmissionTimer] |app.DAGAppMaster|: DAGAppMasterShutdownHandler invoked
2017-05-31 16:06:09,581 [INFO] [DAGSubmissionTimer] |app.DAGAppMaster|: Handling DAGAppMaster shutdown
2017-05-31 16:06:09,582 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Sleeping for 5 seconds before shutting down
2017-05-31 16:06:14,582 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Calling stop for all the services
2017-05-31 16:06:14,582 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Initiating stop of YarnTaskScheduler
2017-05-31 16:06:14,582 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Releasing held containers
2017-05-31 16:06:14,583 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Removing all pending taskRequests
2017-05-31 16:06:14,583 [INFO] [AMShutdownThread] |history.HistoryEventHandler|: Stopping HistoryEventHandler
2017-05-31 16:06:14,583 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Stopping RecoveryService
2017-05-31 16:06:14,583 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Handle the remaining events in queue, queue size=0
2017-05-31 16:06:14,584 [INFO] [RecoveryEventHandlingThread] |recovery.RecoveryService|: EventQueue take interrupted. Returning
2017-05-31 16:06:14,584 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Closing Summary Stream
2017-05-31 16:06:14,611 [INFO] [AMShutdownThread] |impl.SimpleHistoryLoggingService|: Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2017-05-31 16:06:14,611 [INFO] [HistoryEventHandlingThread] |impl.SimpleHistoryLoggingService|: EventQueue take interrupted. Returning
2017-05-31 16:06:14,613 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: AllocatedContainerManager Thread interrupted
2017-05-31 16:06:14,615 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=0, successfulDAGs=1, failedDAGs=0, killedDAGs=0
, trackingURL=
2017-05-31 16:06:14,620 [INFO] [AMShutdownThread] |impl.AMRMClientImpl|: Waiting for application to be successfully unregistered.
2017-05-31 16:06:14,720 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Successfully unregistered application from RM
2017-05-31 16:06:14,721 [INFO] [AMShutdownThread] |rm.TaskSchedulerManager|: Shutting down AppCallbackExecutor
2017-05-31 16:06:14,721 [INFO] [AMRM Callback Handler Thread] |impl.AMRMClientAsyncImpl|: Interrupted while waiting for queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
2017-05-31 16:06:14,726 [INFO] [AMShutdownThread] |mortbay.log|: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0
2017-05-31 16:06:14,826 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 49703
2017-05-31 16:06:14,827 [INFO] [IPC Server listener on 49703] |ipc.Server|: Stopping IPC Server listener on 49703
2017-05-31 16:06:14,827 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 43709
2017-05-31 16:06:14,827 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder
2017-05-31 16:06:14,827 [INFO] [IPC Server listener on 43709] |ipc.Server|: Stopping IPC Server listener on 43709
2017-05-31 16:06:14,827 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder
2017-05-31 16:06:14,830 [INFO] [Thread-2] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked
2017-05-31 16:06:14,830 [INFO] [Thread-2] |app.DAGAppMaster|: The shutdown handler is still running, waiting for it to complete
2017-05-31 16:06:14,844 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Completed deletion of tez scratch data dir, path=hdfs://avatarcluster/tmp/hive/hadoop/_tez_session_dir/46c45420-9bdf-40a5-83a5-c8d1d496abb8/.tez/application_1490688643958_53401
2017-05-31 16:06:14,844 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Exiting DAGAppMaster..GoodBye!
2017-05-31 16:06:14,844 [INFO] [Thread-2] |app.DAGAppMaster|: The shutdown handler has completed
hadoop 集群架构
app05: NameNode/ResourceManager/JournalNode
app10: NameNode(Standby)/JournalNode
app08: DataNode/NodeManager/JournalNode
app09: DataNode/NodeManager
虽然 app05/08/09/10 是我的测试版机器,每个都有 32 个 vcore 和 48GB ram。
hadoop 配置
核心站点.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://avatarcluster</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-data/</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>app02.hp.sp.tst.bmsre.com:2181</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
</configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.nameservices</name>
<value>avatarcluster</value>
</property>
<property>
<name>dfs.ha.namenodes.avatarcluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.avatarcluster.nn1</name>
<value>app05.hp.sp.tst.bmsre.com:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.avatarcluster.nn2</name>
<value>app10.hp.sp.tst.bmsre.com:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.avatarcluster.nn1</name>
<value>app05.hp.sp.tst.bmsre.com:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.avatarcluster.nn2</name>
<value>app10.hp.sp.tst.bmsre.com:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://app05.hp.sp.tst.bmsre.com:8485;app10.hp.sp.tst.bmsre.com:8485;app08.hp.sp.tst.bmsre.com:8485/avatarcluster
</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.avatarcluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/hadoop/hadoop/journal-data</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/hadoop/namenode</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>100</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
maprd-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>app05.hp.sp.tst.bmsre.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>app05.hp.sp.tst.bmsre.com:19888</value>
</property>
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/mr-history/tmp</value>
</property>
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/mr-history/done</value>
</property>
<property>
<name>mapred.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.output.compression.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>3048</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3048</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx2024m</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2024m</value>
</property>
</configuration>
tez-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/apps/tez-0.8.5.tar.gz</value>
</property>
<property>
<name>tez.am.resource.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>tez.task.resource.memory.mb</name>
<value>2048</value>
</property>
</configuration>
yarn-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.admin.acl</name>
<value>*</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>app05.hp.sp.tst.bmsre.com:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>app05.hp.sp.tst.bmsre.com:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>app05.hp.sp.tst.bmsre.com:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>app05.hp.sp.tst.bmsre.com:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>app05.hp.sp.tst.bmsre.com:8088</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/hadoop/hadoop/nodemanager-workdir</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/hadoop/hadoop/nodemanager-logs</value>
</property>
<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/home/hadoop/hadoop/nodemanager-remote-app-logs</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>
</configuration>
最佳答案
我不确定你的集群配置(节点、内存、cpus),如果你能用这些信息更新帖子会很有用。同时,第一个猜测是您的 Tez 配置错误,因此我建议将 MB 减少到更小的值(对于小数据来说,几 mb 应该足够了)。另外,尝试使用 mapred 引擎运行相同的命令以限制问题
hive.execution.engine=mr
关于hadoop - Hive 执行 "insert into ... values ..."非常慢,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/44280532/
我有一个“有趣”的问题,即以两种不同的方式运行 wine 会导致: $> wine --version /Applications/Wine.app/Contents/Resources/bin/wi
我制作了这个网络抓取工具来获取网页中的表格。我使用 puppeteer (不知道 crontab 有问题)、Python 进行清理并处理数据库的输出 但令我惊讶的是,当我执行它时 */50 * * *
JavaScript 是否被调用或执行取决于什么?准确地说,我有两个函数,它们都以相同的方式调用: [self.mapView stringByEvaluatingJavaScriptFromStri
我目前正在使用 python 做一个机器学习项目(这里是初学者,从头开始学习一切)。 只是想知道 statsmodels 的 OLS 和 scikit 的 PooledOlS 使用我拥有的相同面板数据
在使用集成对象模型 (IOM) 后,我可以执行 SAS 代码并将 SAS 数据集读入 .Net/C# 数据集 here . 只是好奇,使用 .Net 作为 SAS 服务器的客户端与使用 Enterpr
有一些直接的 jQuery 在单击时隐藏打开的 div 未显示,但仍将高度添加到导航中以使其看起来好像要掉下来了。 这个脚本工作正常: $(document).ready(funct
这个问题已经有答案了: How do I compare strings in Java? (23 个回答) 已关闭 4 年前。 这里是 Java 新手,我正在使用 NetBeans 尝试一些简单的代
如果我将它切换到 Python 2.x,它执行 10。这是为什么? 训练逻辑回归模型 import keras.backend as
我有两个脚本,它们包含在 HTML 正文中。在第一个脚本中,我初始化一个 JS 对象,该对象在第二个脚本标记中引用。 ... obj.a = 1000; obj.
每当我运行该方法时,我都会收到一个带有数字的错误 以下是我的代码。 public String getAccount() { String s = "Listing the accounts";
我已经用 do~while(true) 创建了我的菜单;但是每次用户输入一个数字时,它不会运行程序,而是再次显示菜单!你怎么看? //我的主要方法 public static void main(St
执行命令后,如何让IPython通知我?我可以使用铃声/警报还是通过弹出窗口获取它?我正在OS X 10.8.5的iTerm上运行Anaconda。 最佳答案 使用最新版本的iTerm,您可以在she
您好,我刚刚使用菜单栏为 Swing 编写了代码。但是问题出现在运行中。我输入: javac Menu.java java Menu 它没有给出任何错误,但 GUI 没有显示。这是我的源代码以供引用:
我觉得这里缺少明显的东西,但是我看不到它写在任何地方。 我使用Authenticode证书对可执行文件进行签名,但是当我开始学习有关它的更多信息时,我对原样的值(value)提出了质疑。 签名的exe
我正在设计一个应用程序,它使用 DataTables 中的预定义库来创建数据表。我想对数据表执行删除操作,为此应在按钮单击事件上执行 java 脚本。 $(document).ready(functi
我是 Haskell 新手,如果有人愿意帮助我,我会很高兴!我试图让这个程序与 do while 循环一起工作。 第二个 getLine 命令的结果被放入变量 goGlenn 中,如果 goGlenn
我有一个用 swing 实现迷你游戏的程序,在主类中我有一个循环,用于监听游戏 map 中的 boolean 值。使用 while 实现的循环不会执行一条指令,如果它是唯一的一条指令,我不知道为什么。
我正在尝试开发一个连接到 Oracle 数据库并执行函数的 Java 应用程序。如果我在 Eclipse 中运行该应用程序,它可以工作,但是当我尝试在 Windows 命令提示符中运行 .jar 时,
我正在阅读有关 Java 中的 Future 和 javascript 中的 Promises 的内容。下面是我作为示例编写的代码。我的问题是分配给 future 的任务什么时候开始执行? 当如下行创
我有一个常见的情况,您有两个变量(xSpeed 和 ySpeed),当它们低于 minSpeed 时,我想将它们独立设置为零,并在它们都为零时退出。 最有效的方法是什么?目前我有两种方法(方法2更干净
我是一名优秀的程序员,十分优秀!