- mongodb - 在 MongoDB mapreduce 中,如何展平值对象?
- javascript - 对象传播与 Object.assign
- html - 输入类型 ="submit"Vs 按钮标签它们可以互换吗?
- sql - 使用 MongoDB 而不是 MS SQL Server 的优缺点
我正在尝试通过 Sqoop 将包含 3200 万条记录的表从 SQL Server 导入 Hive。连接是 SQL Server 成功的。但是 Map/Reduce 作业没有成功执行。它给出了以下错误:
18/07/19 04:00:11 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
18/07/19 04:00:27 DEBUG db.DBConfiguration: Fetching password from job credentials store
18/07/19 04:00:27 INFO db.DBInputFormat: Using read commited transaction isolation
18/07/19 04:00:27 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '1=1' and upper bound '1=1'
18/07/19 04:00:28 INFO mapreduce.JobSubmitter: number of splits:1
18/07/19 04:00:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1531917395459_0002
18/07/19 04:00:30 INFO impl.YarnClientImpl: Submitted application application_1531917395459_0002
18/07/19 04:00:30 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1531917395459_0002/
18/07/19 04:00:30 INFO mapreduce.Job: Running job: job_1531917395459_0002
18/07/19 04:43:02 INFO mapreduce.Job: Job job_1531917395459_0002 running in uber mode : false
18/07/19 04:43:03 INFO mapreduce.Job: map 0% reduce 0%
18/07/19 04:43:04 INFO mapreduce.Job: Job job_1531917395459_0002 failed with state FAILED due to: Application application_1531917395459_0002 failed 2 times due to ApplicationMaster for attempt appattempt_1531917395459_0002_000002 timed out. Failing the application.
18/07/19 04:43:08 INFO mapreduce.Job: Counters: 0
18/07/19 04:43:08 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead
18/07/19 04:43:09 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 2,576.6368 seconds (0 bytes/sec)
18/07/19 04:43:10 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
18/07/19 04:43:10 INFO mapreduce.ImportJobBase: Retrieved 0 records.
18/07/19 04:43:10 ERROR tool.ImportTool: Error during import: Import job failed!
这里是来自 yarn-site.xml 文件的配置
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.log-aggregation-enable</name>
<value>true</value>
</property>
<property>
<name>yarn.dispatcher.exit-on-error</name>
<value>true</value>
</property>
<property>
<description>List of directories to store localized files in.</description>
<name>yarn.nodemanager.local-dirs</name>
<value>/var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir</value>
</property>
<property>
<description>Where to store container logs.</description>
<name>yarn.nodemanager.log-dirs</name>
<value>/var/log/hadoop-yarn/containers</value>
</property>
<property>
<description>Where to aggregate logs to.</description>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/var/log/hadoop-yarn/apps</value>
</property>
<property>
<description>Classpath for typical applications.</description>
<name>yarn.application.classpath</name>
<value>
$HADOOP_CONF_DIR,
$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,
$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,
$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,
$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*
</value>
</property>
<!-- added by me -->
<property>
<name>yarn.resourcemanager.hostname</name>
<value>localhost</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:8031</value>
</property>
</configuration>
起初,作业在通过 0.0.0.0:8032 与资源管理器连接时卡住了。于是我把主机改成了127.0.0.1。然后行刑继续进行。但随后出现上述错误。即使我尝试只用 1000 行来执行这项工作,但同样的错误。此外,有时工作会被终止。
这是我的 sqoop 命令
sqoop import --connect "jdbc:sqlserver://system-ip;databaseName=TEST" --driver com.microsoft.sqlserver.jdbc.SQLServerDriver --username user1 --password password --hive-import --create-hive-table --hive-table "customer_data_1000" --table "customer_data_1000" --split-by Account_Branch_Converted -m 1 --verbose
这是我的 docker 命令以防万一:
docker run --hostname=quickstart.cloudera --privileged=true -t -p 127.0.0.1:8888:8888 -p 127.0.
0.1:7180:7180 -p 127.0.0.1:50070:50070 -i 7c41929668d8/usr/bin/docker-quickstart
这是资源管理器日志:
2018-07-26 07:18:26,439 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:appattempt_1532588462827_0001_000001 Timed out after 600 secs
2018-07-26 07:24:03,059 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1532588462827_0001_000001 with final state: FAILED, and exit status: -1000
2018-07-26 07:35:46,609 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000001 State change from LAUNCHED to FINAL_SAVING
2018-07-26 07:35:49,502 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:quickstart.cloudera:36003 Timed out after 600 secs
2018-07-26 07:39:44,485 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Deactivating Node quickstart.cloudera:36003 as it is now LOST
2018-07-26 07:44:39,238 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: quickstart.cloudera:36003 Node Transitioned from RUNNING to LOST
2018-07-26 07:45:09,895 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1532588462827_0001_000001
2018-07-26 07:49:43,848 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found resyncing quickstart.cloudera:36003
2018-07-26 07:49:43,916 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1532588462827_0001_000001
2018-07-26 07:49:45,738 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000001 State change from FINAL_SAVING to FAILED
2018-07-26 07:49:47,095 WARN org.apache.hadoop.ipc.Server: IPC Server handler 12 on 8032, call org.apache.hadoop.yarn.api.ApplicationClientProtocolPB.getApplicationReport from 127.0.0.1:45162 Call#608 Retry#0: output error
2018-07-26 07:49:47,100 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 1. The max attempts is 2
2018-07-26 07:49:47,887 INFO org.apache.hadoop.ipc.Server: IPC Server handler 12 on 8032 caught an exception
java.nio.channels.ClosedChannelException
at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:265)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:474)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:2621)
at org.apache.hadoop.ipc.Server.access$1900(Server.java:134)
at org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:989)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:1054)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2141)
2018-07-26 07:49:49,127 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Registering app attempt : appattempt_1532588462827_0001_000002
2018-07-26 07:49:49,127 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from NEW to SUBMITTED
2018-07-26 07:49:49,127 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1532588462827_0001_000001
2018-07-26 07:49:50,458 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_01_000001 Container Transitioned from RUNNING to KILLED
2018-07-26 07:49:50,459 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt: Completed container: container_1532588462827_0001_01_000001 in state: KILLED event:KILL
2018-07-26 07:49:50,460 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_01_000001
2018-07-26 07:49:50,550 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1532588462827_0001_01_000001 of capacity <memory:2048, vCores:1> on host quickstart.cloudera:36003, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2018-07-26 07:49:50,563 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application attempt appattempt_1532588462827_0001_000001 released container container_1532588462827_0001_01_000001 on node: host: quickstart.cloudera:36003 #containers=0 available=8192 used=0 with event: KILL
2018-07-26 07:49:50,580 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Removed node quickstart.cloudera:36003 cluster capacity: <memory:0, vCores:0>
2018-07-26 07:49:50,580 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application appattempt_1532588462827_0001_000001 is done. finalState=FAILED
2018-07-26 07:49:50,581 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1532588462827_0001 requests cleared
2018-07-26 07:49:51,860 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added Application Attempt appattempt_1532588462827_0001_000002 to scheduler from user: root
2018-07-26 07:49:52,125 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from SUBMITTED to SCHEDULED
2018-07-26 07:50:04,533 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: quickstart.cloudera:36003 Node Transitioned from NEW to RUNNING
2018-07-26 07:50:04,534 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Added node quickstart.cloudera:36003 cluster capacity: <memory:8192, vCores:8>
2018-07-26 07:50:05,883 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: received container statuses on node manager register :[container_id { app_attempt_id { application_id { id: 1 cluster_timestamp: 1532588462827 } attemptId: 1 } id: 1 } container_state: C_COMPLETE resource { memory: 2048 virtual_cores: 1 } priority { priority: 0 } diagnostics: "Container Killed by ResourceManager\nContainer killed on request. Exit code is 143\nContainer exited with a non-zero exit code 143\n" container_exit_status: -106 creation_time: 1532588806652]
2018-07-26 07:50:05,883 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: NodeManager from node quickstart.cloudera(cmPort: 36003 httpPort: 8042) registered with capability: <memory:8192, vCores:8>, assigned nodeId quickstart.cloudera:36003
2018-07-26 07:50:06,020 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:06,022 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000001 Container Transitioned from NEW to ALLOCATED
2018-07-26 07:50:06,023 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_02_000001
2018-07-26 07:50:06,025 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1532588462827_0001_02_000001 of capacity <memory:2048, vCores:1> on host quickstart.cloudera:36003, which has 1 containers, <memory:2048, vCores:1> used and <memory:6144, vCores:7> available after allocation
2018-07-26 07:50:06,025 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : quickstart.cloudera:36003 for container : container_1532588462827_0001_02_000001
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000001 Container Transitioned from ALLOCATED to ACQUIRED
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Clear node set for appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Storing attempt: AppId: application_1532588462827_0001 AttemptId: appattempt_1532588462827_0001_000002 MasterContainer: Container: [ContainerId: container_1532588462827_0001_02_000001, NodeId: quickstart.cloudera:36003, NodeHttpAddress: quickstart.cloudera:8042, Resource: <memory:2048, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.17.0.2:36003 }, ]
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from SCHEDULED to ALLOCATED_SAVING
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from ALLOCATED_SAVING to ALLOCATED
2018-07-26 07:50:06,026 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Launching masterappattempt_1532588462827_0001_000002
2018-07-26 07:50:06,027 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Setting up container Container: [ContainerId: container_1532588462827_0001_02_000001, NodeId: quickstart.cloudera:36003, NodeHttpAddress: quickstart.cloudera:8042, Resource: <memory:2048, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.17.0.2:36003 }, ] for AM appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,027 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command to launch container container_1532588462827_0001_02_000001 : $JAVA_HOME/bin/java -Dlog4j.configuration=container-log4j.properties -Dyarn.app.container.log.dir=<LOG_DIR> -Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA -Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster 1><LOG_DIR>/stdout 2><LOG_DIR>/stderr
2018-07-26 07:50:06,027 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Create AMRMToken for ApplicationAttempt: appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,027 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Creating password for appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,128 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Done launching container Container: [ContainerId: container_1532588462827_0001_02_000001, NodeId: quickstart.cloudera:36003, NodeHttpAddress: quickstart.cloudera:8042, Resource: <memory:2048, vCores:1>, Priority: 0, Token: Token { kind: ContainerToken, service: 172.17.0.2:36003 }, ] for AM appattempt_1532588462827_0001_000002
2018-07-26 07:50:06,129 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from ALLOCATED to LAUNCHED
2018-07-26 07:50:06,953 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000001 Container Transitioned from ACQUIRED to RUNNING
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:32,951 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for appattempt_1532588462827_0001_000002 (auth:SIMPLE)
2018-07-26 07:50:33,014 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: AM registration appattempt_1532588462827_0001_000002
emanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:34,887 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:34,893 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000002 Container Transitioned from NEW to ALLOCATED
2018-07-26 07:50:34,893 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Allocated Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_02_000002
2018-07-26 07:50:34,894 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Assigned container container_1532588462827_0001_02_000002 of capacity <memory:1024, vCores:1> on host quickstart.cloudera:36003, which has 2 containers, <memory:3072, vCores:2> used and <memory:5120, vCores:6> available after allocation
2018-07-26 07:50:36,467 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:36,478 INFO org.apache.hadoop.yarn.server.resourcemanager.security.NMTokenSecretManagerInRM: Sending NMToken for nodeId : quickstart.cloudera:36003 for container : container_1532588462827_0001_02_000002
2018-07-26 07:50:36,479 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000002 Container Transitioned from ALLOCATED to ACQUIRED
2018-07-26 07:50:36,898 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_01_000001 completed with event FINISHED
2018-07-26 07:50:38,113 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000002 Container Transitioned from ACQUIRED to RUNNING
2018-07-26 07:50:38,113 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: checking for deactivate...
2018-07-26 07:50:54,379 INFO org.apache.hadoop.yarn.server.resourcemanager.rmcontainer.RMContainerImpl: container_1532588462827_0001_02_000002 Container Transitioned from RUNNING to COMPLETED
2018-07-26 07:50:54,525 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt: Completed container: container_1532588462827_0001_02_000002 in state: COMPLETED event:FINISHED
2018-07-26 07:50:54,553 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_02_000002
2018-07-26 07:50:54,555 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1532588462827_0001_02_000002 of capacity <memory:1024, vCores:1> on host quickstart.cloudera:36003, which currently has 1 containers, <memory:2048, vCores:1> used and <memory:6144, vCores:7> available, release resources=true
2018-07-26 07:50:54,558 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application attempt appattempt_1532588462827_0001_000002 released container container_1532588462827_0001_02_000002 on node: host: quickstart.cloudera:36003 #containers=1 available=6144 used=2048 with event: FINISHED
2018-07-26 07:50:55,386 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_02_000002 completed with event FINISHED
438 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSAppAttempt: Completed container: container_1532588462827_0001_02_000001 in state: COMPLETED event:FINISHED
2018-07-26 07:51:00,438 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=AM Released Container TARGET=SchedulerApp RESULT=SUCCESS APPID=application_1532588462827_0001 CONTAINERID=container_1532588462827_0001_02_000001
2018-07-26 07:51:00,438 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerNode: Released container container_1532588462827_0001_02_000001 of capacity <memory:2048, vCores:1> on host quickstart.cloudera:36003, which currently has 0 containers, <memory:0, vCores:0> used and <memory:8192, vCores:8> available, release resources=true
2018-07-26 07:51:00,438 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: Updating application attempt appattempt_1532588462827_0001_000002 with final state: FAILED, and exit status: 0
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from RUNNING to FINAL_SAVING
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: Unregistering app attempt : appattempt_1532588462827_0001_000002
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.security.AMRMTokenSecretManager: Application finished, removing password for appattempt_1532588462827_0001_000002
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: appattempt_1532588462827_0001_000002 State change from FINAL_SAVING to FAILED
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: The number of failed attempts is 2. The max attempts is 2
2018-07-26 07:51:00,439 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Updating application application_1532588462827_0001 with final state: FAILED
2018-07-26 07:51:00,457 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1532588462827_0001 State change from RUNNING to FINAL_SAVING
2018-07-26 07:51:00,458 INFO org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore: Updating info for app: application_1532588462827_0001
2018-07-26 07:51:00,458 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Application appattempt_1532588462827_0001_000002 is done. finalState=FAILED
2018-07-26 07:51:00,458 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.AppSchedulingInfo: Application application_1532588462827_0001 requests cleared
2018-07-26 07:51:00,458 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Cleaning master appattempt_1532588462827_0001_000002
2018-07-26 07:51:05,760 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Application application_1532588462827_0001 failed 2 times due to AM Container for appattempt_1532588462827_0001_000002 exited with exitCode: 0
For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1532588462827_0001/Then, click on links to logs of each attempt.
Diagnostics: Failing this attempt. Failing the application.
2018-07-26 07:51:05,781 INFO org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: application_1532588462827_0001 State change from FINAL_SAVING to FAILED
2018-07-26 07:51:05,785 WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=root OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1532588462827_0001 failed 2 times due to AM Container for appattempt_1532588462827_0001_000002 exited with exitCode: 0
For more detailed output, check application tracking page:http://quickstart.cloudera:8088/proxy/application_1532588462827_0001/Then, click on links to logs of each attempt.
Diagnostics: Failing this attempt. Failing the application. APPID=application_1532588462827_0001
2018-07-26 07:51:05,819 INFO org.apache.hadoop.yarn.server.resourcemanager.RMAppManager$ApplicationSummary: appId=application_1532588462827_0001,name=customer_data_1000.jar,user=root,queue=root.root,state=FAILED,trackingUrl=http://quickstart.cloudera:8088/cluster/app/application_1532588462827_0001,appMasterHost=N/A,startTime=1532588804719,finishTime=1532591460451,finalStatus=FAILED
2018-07-26 07:51:05,821 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_02_000001 completed with event FINISHED
2018-07-26 07:51:05,822 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler: Container container_1532588462827_0001_02_000002 completed with event FINISHED
我哪里错了?
最佳答案
我不能给你一个精确的解决方案,但我能做的就是告诉你根本原因是什么:
由于上述原因之一,您的工作失败了。由于您有足够的 v-cores 和可用内存,并且容器也正在创建中。所以,处理端一切正常,但肯定有配置错误。
关于docker - Sqoop - 导入作业失败,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/51414922/
当我们使用以下设置时,Sqoop 导入会导致重复/部分记录 --query - 自定义查询 --split-by - 非整数列(字符) --num-mappers - 超过 2 Verified th
我安装了 Hadoop、Hive、HBase、Sqoop 并将它们添加到 PATH 中。 当我尝试执行 sqoop 命令时,出现此错误: Error: Could not find or load m
当我连接到一个集群 ( aaaaaa1 ) 上的 mysql 和不同服务器 ( aaaaaa2 ) 上的 sqooop 时,我收到以下错误,尽管在 mySql 中创建了一个表,并授予另一个集群的完全权
我发现以下命令对于查看我的源数据库的样子非常有用: sqoop-list-databases sqoop-list-tables 但是,似乎没有列出表中列的命令,这将是一个合乎逻辑的步骤。 我现在的问
我很困惑虽然导入库(com.cloudera.sqoop 和 org.apache.sqoop)并在 eclipse 中获取它(包含 jar sqoop-1.4.4-hadoop200.jar)- I
sqoop import命令中$conditions子句的意义是什么? select col1, col2 from test_table where \$CONDITIONS 最佳答案 Sqoop
sqoop import命令中$conditions子句的意义是什么? select col1, col2 from test_table where \$CONDITIONS 最佳答案 Sqoop
我正在使用共享节点集群 Hadoop 2.5.0-cdh5.3.2 请共享要加载的所有兼容版本的 MySql jar 文件的名称以及 HDFS 和 MySQL 之间成功导入和导出的所有路径文件夹。 我
我在Azkaban中使用shell命令,并将Sqoop命令放在shell脚本中。 今天 Sqoop 任务无缘无故卡住了,sqoop_task1。 几天前发生在另一个 sqoop 任务上,我们称它为 s
我已经在计算机上安装了sqoop (来自http://www.apache.org/dist/sqoop/1.4.4/的sqoop-1.4.4.bin__hadoop-1.0.0.tar.gz)。 当
我刚刚下载了Sqoop安装文件sqoop-1.99.3-bin-hadoop100.tar.gz。我无法在其中找到文件 addtowar.sh。我按照此处的安装说明进行操作 - https://sqo
我有一个 Oozie 工作流,它应该每 X 分钟运行一次。它从 HBase 表中读取一个值。在此之后,基于上一步从 HBase 读取的值运行(增量)Sqoop 操作。为了使工作流正常工作,我需要以某种
[cloudera@quickstart ~]$ **sqoop import-all-tables -m=4 --connect "jdbc:mysql://quickstart.cloudera:
我正在使用 hadoop 2.6.0,现在我正在尝试 sqoop-1.4.5.bin__hadoop-2.0.4-alpha.tar.gz。我正在使用 sqoop 版本 sqoop version 2
我发现在运行时覆盖 sqoop 作业属性时,我只能覆盖一个属性。 例子1:如果我提交 sqoop job --exec test123 -- --query "select * from test w
如何使用sqoop job自动化sqoop增量导入? 据我所知,sqoop job 会记住最后一个值。如果我们创建一个类似 的 sqoop 作业 sqoop job --create myjob --
我正在尝试使用 sqoop export 将记录从 S3 导出到 Mysql Aurora。 S3 中的数据类型之一是 clob,它的长文本和 XML 文件作为 string 存储在其中。 当我运行我
我正在对由计算量大的自定义查询生成的大型记录集进行原型(prototype)迁移。此查询大约需要 1-2 小时才能在 SQL Developer 中返回结果集 我正在尝试将此查询传递给一个简单的 Sq
谁能告诉我这个命令的输出是什么:这里的 departments 表有默认的 6 行(从 dept_id 2 到 7),然后我向 Mysql db 'retail_db.departments' 表(d
根据 sqoop.apache.org 的说法,Sqoop 2 的功能并不完整,不应该用于生产系统。很公平,有些人可能想在他们的测试环境中测试 Sqoop 2 的新功能。 Cloudera 对 Sqo
我是一名优秀的程序员,十分优秀!