- android - RelativeLayout 背景可绘制重叠内容
- android - 如何链接 cpufeatures lib 以获取 native android 库?
- java - OnItemClickListener 不起作用,但 OnLongItemClickListener 在自定义 ListView 中起作用
- java - Android 文件转字符串
我构建了一个 hadoop 和 hive 集群并尝试做一些测试。但它真的很慢。
表格
表 value_count
+--------------------------------------------------------------+--+
| createtab_stmt |
+--------------------------------------------------------------+--+
| CREATE TABLE `value_count`( |
| `key` int, |
| `count` int, |
| `create_date` date COMMENT '????') |
| COMMENT 'This is a group table' |
| ROW FORMAT SERDE |
| 'org.apache.hadoop.hive.ql.io.orc.OrcSerde' |
| STORED AS INPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat' |
| OUTPUTFORMAT |
| 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat' |
| LOCATION |
| 'hdfs://avatarcluster/hive/warehouse/test.db/value_count' |
| TBLPROPERTIES ( |
| 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}', |
| 'numFiles'='7', |
| 'numRows'='7', |
| 'rawDataSize'='448', |
| 'totalSize'='2297', |
| 'transient_lastDdlTime'='1496217645') |
+--------------------------------------------------------------+--+
sql执行
insert into value_count values (5, 1, '2017-05-06');
这个sql我已经执行了好几次了,每次大概需要4、5分钟
hadoop 容器日志
2017-05-31 16:00:45,041 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Central Dispatcher queue size after DAG completion, before cleanup: 0
2017-05-31 16:00:45,041 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Waiting for next DAG to be submitted.
2017-05-31 16:00:45,042 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Cleaning up DAG: name=insert into value_count valu...'2017-05-06')(Stage-1), with id=dag_1490688643958_53401_1
2017-05-31 16:00:45,042 [INFO] [Dispatcher thread {Central}] |container.AMContainerMap|: Cleaned up completed containers on dagComplete. Removed=0, Remaining=1
2017-05-31 16:00:45,044 [INFO] [Dispatcher thread {Central}] |app.DAGAppMaster|: Completed cleanup for DAG: name=insert into value_count valu...'2017-05-06')(Stage-1), with id=dag_1490688643958_53401_1
2017-05-31 16:00:50,749 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_1490688643958_53401_01_000002, containerExpiryTime=1496217650577, idleTimeout=5000, taskRequestsCount=0, heldContainers=1, delayedContainers=0, isNew=false
2017-05-31 16:00:50,752 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1490688643958_53401_1][Event:CONTAINER_STOPPED]: containerId=container_1490688643958_53401_01_000002, stoppedTime=1496217650751, exitStatus=0
2017-05-31 16:00:50,753 [INFO] [ContainerLauncher #1] |launcher.TezContainerLauncherImpl|: Stopping container_1490688643958_53401_01_000002
2017-05-31 16:00:50,753 [INFO] [ContainerLauncher #1] |impl.ContainerManagementProtocolProxy|: Opening proxy : app08.hp.sp.tst.bmsre.com:51640
2017-05-31 16:00:51,628 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_1490688643958_53401_01_000002 exited with diagnostics set to Container failed, exitCode=-105. Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
2017-05-31 16:01:29,678 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 51 lastPreemptionHeartbeat: 50
2017-05-31 16:02:19,740 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 101 lastPreemptionHeartbeat: 100
2017-05-31 16:03:09,801 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 151 lastPreemptionHeartbeat: 150
2017-05-31 16:03:59,858 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 201 lastPreemptionHeartbeat: 200
2017-05-31 16:04:49,915 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 251 lastPreemptionHeartbeat: 250
2017-05-31 16:05:39,971 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:6144, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 301 lastPreemptionHeartbeat: 300
2017-05-31 16:06:09,581 [INFO] [DAGSubmissionTimer] |rm.TaskSchedulerManager|: TaskScheduler notified that it should unregister from RM
2017-05-31 16:06:09,581 [INFO] [DAGSubmissionTimer] |app.DAGAppMaster|: No current running DAG, shutting down the AM
2017-05-31 16:06:09,581 [INFO] [DAGSubmissionTimer] |app.DAGAppMaster|: DAGAppMasterShutdownHandler invoked
2017-05-31 16:06:09,581 [INFO] [DAGSubmissionTimer] |app.DAGAppMaster|: Handling DAGAppMaster shutdown
2017-05-31 16:06:09,582 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Sleeping for 5 seconds before shutting down
2017-05-31 16:06:14,582 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Calling stop for all the services
2017-05-31 16:06:14,582 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Initiating stop of YarnTaskScheduler
2017-05-31 16:06:14,582 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Releasing held containers
2017-05-31 16:06:14,583 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Removing all pending taskRequests
2017-05-31 16:06:14,583 [INFO] [AMShutdownThread] |history.HistoryEventHandler|: Stopping HistoryEventHandler
2017-05-31 16:06:14,583 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Stopping RecoveryService
2017-05-31 16:06:14,583 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Handle the remaining events in queue, queue size=0
2017-05-31 16:06:14,584 [INFO] [RecoveryEventHandlingThread] |recovery.RecoveryService|: EventQueue take interrupted. Returning
2017-05-31 16:06:14,584 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Closing Summary Stream
2017-05-31 16:06:14,611 [INFO] [AMShutdownThread] |impl.SimpleHistoryLoggingService|: Stopping SimpleHistoryLoggingService, eventQueueBacklog=0
2017-05-31 16:06:14,611 [INFO] [HistoryEventHandlingThread] |impl.SimpleHistoryLoggingService|: EventQueue take interrupted. Returning
2017-05-31 16:06:14,613 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: AllocatedContainerManager Thread interrupted
2017-05-31 16:06:14,615 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=0, successfulDAGs=1, failedDAGs=0, killedDAGs=0
, trackingURL=
2017-05-31 16:06:14,620 [INFO] [AMShutdownThread] |impl.AMRMClientImpl|: Waiting for application to be successfully unregistered.
2017-05-31 16:06:14,720 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Successfully unregistered application from RM
2017-05-31 16:06:14,721 [INFO] [AMShutdownThread] |rm.TaskSchedulerManager|: Shutting down AppCallbackExecutor
2017-05-31 16:06:14,721 [INFO] [AMRM Callback Handler Thread] |impl.AMRMClientAsyncImpl|: Interrupted while waiting for queue
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
2017-05-31 16:06:14,726 [INFO] [AMShutdownThread] |mortbay.log|: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:0
2017-05-31 16:06:14,826 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 49703
2017-05-31 16:06:14,827 [INFO] [IPC Server listener on 49703] |ipc.Server|: Stopping IPC Server listener on 49703
2017-05-31 16:06:14,827 [INFO] [AMShutdownThread] |ipc.Server|: Stopping server on 43709
2017-05-31 16:06:14,827 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder
2017-05-31 16:06:14,827 [INFO] [IPC Server listener on 43709] |ipc.Server|: Stopping IPC Server listener on 43709
2017-05-31 16:06:14,827 [INFO] [IPC Server Responder] |ipc.Server|: Stopping IPC Server Responder
2017-05-31 16:06:14,830 [INFO] [Thread-2] |app.DAGAppMaster|: DAGAppMasterShutdownHook invoked
2017-05-31 16:06:14,830 [INFO] [Thread-2] |app.DAGAppMaster|: The shutdown handler is still running, waiting for it to complete
2017-05-31 16:06:14,844 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Completed deletion of tez scratch data dir, path=hdfs://avatarcluster/tmp/hive/hadoop/_tez_session_dir/46c45420-9bdf-40a5-83a5-c8d1d496abb8/.tez/application_1490688643958_53401
2017-05-31 16:06:14,844 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Exiting DAGAppMaster..GoodBye!
2017-05-31 16:06:14,844 [INFO] [Thread-2] |app.DAGAppMaster|: The shutdown handler has completed
hadoop 集群架构
app05: NameNode/ResourceManager/JournalNode
app10: NameNode(Standby)/JournalNode
app08: DataNode/NodeManager/JournalNode
app09: DataNode/NodeManager
虽然 app05/08/09/10 是我的测试版机器,每个都有 32 个 vcore 和 48GB ram。
hadoop 配置
核心站点.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://avatarcluster</value>
</property>
<property>
<name>io.file.buffer.size</name>
<value>131072</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hadoop.groups</name>
<value>*</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop-data/</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>app02.hp.sp.tst.bmsre.com:2181</value>
</property>
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
</configuration>
hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.nameservices</name>
<value>avatarcluster</value>
</property>
<property>
<name>dfs.ha.namenodes.avatarcluster</name>
<value>nn1,nn2</value>
</property>
<property>
<name>dfs.namenode.rpc-address.avatarcluster.nn1</name>
<value>app05.hp.sp.tst.bmsre.com:8020</value>
</property>
<property>
<name>dfs.namenode.rpc-address.avatarcluster.nn2</name>
<value>app10.hp.sp.tst.bmsre.com:8020</value>
</property>
<property>
<name>dfs.namenode.http-address.avatarcluster.nn1</name>
<value>app05.hp.sp.tst.bmsre.com:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.avatarcluster.nn2</name>
<value>app10.hp.sp.tst.bmsre.com:50070</value>
</property>
<property>
<name>dfs.namenode.shared.edits.dir</name>
<value>qjournal://app05.hp.sp.tst.bmsre.com:8485;app10.hp.sp.tst.bmsre.com:8485;app08.hp.sp.tst.bmsre.com:8485/avatarcluster
</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.avatarcluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/hadoop/hadoop/journal-data</value>
</property>
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/hadoop/namenode</value>
</property>
<property>
<name>dfs.blocksize</name>
<value>134217728</value>
</property>
<property>
<name>dfs.namenode.handler.count</name>
<value>100</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
</configuration>
maprd-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>app05.hp.sp.tst.bmsre.com:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>app05.hp.sp.tst.bmsre.com:19888</value>
</property>
<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/mr-history/tmp</value>
</property>
<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/mr-history/done</value>
</property>
<property>
<name>mapred.output.compress</name>
<value>true</value>
</property>
<property>
<name>mapred.output.compression.codec</name>
<value>org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
<property>
<name>mapreduce.map.memory.mb</name>
<value>3048</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3048</value>
</property>
<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx2024m</value>
</property>
<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx2024m</value>
</property>
</configuration>
tez-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>tez.lib.uris</name>
<value>${fs.defaultFS}/apps/tez-0.8.5.tar.gz</value>
</property>
<property>
<name>tez.am.resource.memory.mb</name>
<value>2048</value>
</property>
<property>
<name>tez.task.resource.memory.mb</name>
<value>2048</value>
</property>
</configuration>
yarn-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.admin.acl</name>
<value>*</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>app05.hp.sp.tst.bmsre.com:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>app05.hp.sp.tst.bmsre.com:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>app05.hp.sp.tst.bmsre.com:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>app05.hp.sp.tst.bmsre.com:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>app05.hp.sp.tst.bmsre.com:8088</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/hadoop/hadoop/nodemanager-workdir</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/hadoop/hadoop/nodemanager-logs</value>
</property>
<property>
<name>yarn.nodemanager.log.retain-seconds</name>
<value>3600</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/home/hadoop/hadoop/nodemanager-remote-app-logs</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir-suffix</name>
<value>logs</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>4096</value>
</property>
<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>
</configuration>
最佳答案
我不确定你的集群配置(节点、内存、cpus),如果你能用这些信息更新帖子会很有用。同时,第一个猜测是您的 Tez 配置错误,因此我建议将 MB 减少到更小的值(对于小数据来说,几 mb 应该足够了)。另外,尝试使用 mapred 引擎运行相同的命令以限制问题
hive.execution.engine=mr
关于hadoop - Hive 执行 "insert into ... values ..."非常慢,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/44280532/
我正在开发一些用于计费的数据库项目(PHP/MySQL)。 每当创建新账单时,我想生成一个由年、周和增量编号组成的账单编号。我想用触发器来做到这一点。触发器将使用现有的账单编号来查找增量编号,或者从新
我有一个 MySQL 插入,我正在使用 RAND 生成随机 INT 值问题是它不会插入到数据库中,因为该列接受 TINYINT ,如何将输出转换为 TINYINT。代码示例如下: INSERT INT
如果我想从单个插入中保存主键 (mytable_id),我已完成以下操作: CREATE OR REPLACE FUNCTION myfunct(ownerid text) RETURNS void
为了简单起见,假设我有两个表 用户表(id,email) 用户日志表(id, date) 无论 id 被插入到 user 表中,相同的 id 也应该被插入到 user_log 表中,否则事务应该失败。
为了简单起见,假设我有两个表 用户表(id,email) 用户日志表(id, date) 无论 id 被插入到 user 表中,相同的 id 也应该被插入到 user_log 表中,否则事务应该失败。
我知道在触发器中 - 至少对于 SQL Server - 人们永远不应该假设插入的表只有一行,这意味着触发器中这样的 SQL 通常是不好的: select @UserID = ID from inse
我正在使用 bigquery 对象中的方法 tabledata().insertAll 更新行列表。执行后,返回显示没有错误。但是,我的表仍然继续,没有写入任何数据。 可能是权限问题。如果是这样,为什
这是一个扩展 F# Recursive Tree Validation 的问题,我昨天已经很好地回答了。 这个问题涉及在现有树中插入一个 child 。这是我想使用的更新类型: type Name
我有 2 个表:用户和照片(在 mysql 数据库中)。 在这里你可以看到两个表之间的关系 User Photos -------------
我试图同时在不同的表上插入两行。 子查询INSERT INTO的AUTO_INCRMENT或id的值(如果已经存在)应该写入主查询中。 目前我有这个(仅用 3 个值简化),但它不起作用。我想知道是否有
我有一个 900 万行的表,由于其庞大的规模,我正在努力处理所有这些数据。 我想做的是在不覆盖数据的情况下将 IMPORT 一个 CSV 添加到表中。 在我做这样的事情之前; INSERT if no
我正在写新闻并将其插入到我的数据库中,我在 3 年前构建了代码并且运行良好,但我不能再插入了,我不明白为什么: $insert=mysqli_query($co,"INSERT INTO articl
我正在尝试编写一个简单的 INSERT 语句来将新用户添加到数据库中,但它不起作用,这意味着,我尝试插入到表中的数据都没有被插入。几个小时以来,我一直在尝试解决此问题,但没有成功。我尝试编写插入语句的
所以我有这个表格: http://i.imgur.com/vZYssQy.png 现在 ID、First Name、Last Name、DOB、Address、Phone Number 和 Post
在控制台中运行查询(SELECT 语句)时,从数据库检索到的数据以表格格式显示在数据库控制台工具窗口的结果 Pane 中。 我已经搜索过 datagrip Help我只是想知道是否有任何方法可以用于为
每当使用触发器插入行时,我都试图将另一行插入表中,但收到以下错误消息: The target table 'EDDSDBO.Redaction' of the DML statement cannot
关闭。这个问题不符合Stack Overflow guidelines .它目前不接受答案。 这个问题似乎与 help center 中定义的范围内的编程无关。 . 关闭 6 年前。 Improve
我有以下代码片段: $get_data = mysqli_query ($connect, "SELECT * FROM users WHERE username = '$username'");
情况:需要向 SQLite 数据库中插入大量数据。 问题:我们可以使用两个语句来插入数据 - data = [("111", "222", "333"), ("AAA", "BBB", "CCC"),
我的数据库中有一个表 Teacher: TABLE Teacher ( ID CHAR (7) NOT NULL , name
我是一名优秀的程序员,十分优秀!