gpt4 book ai didi

apache - 带有Nutch的Apache Hadoop 2.7.4尝试由于任何原因失败

转载 作者:行者123 更新时间:2023-12-02 20:49:00 27 4
gpt4 key购买 nike

我已经使用Nutch 2.3.1配置了Apache Hadoop 2.7.4,Hbase 1.2.6和Solr 6.6.1。我在所有节点上的mapred-site.xml

<property>
<name>mapreduce.jobtracker.address</name>
<value>master:9001</value>
</property>

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

<property>
<name>mapreduce.task.timeout</name>
<value>10000</value>
<description>The number of milliseconds before a task will be
terminated if it neither reads an input, writes an output, nor
updates its status string. A value of 0 disables the timeout.
</description>
</property>

<property>
<name>mapreduce.reduce.memory.mb</name>
<value>2048</value>
<description>The amount of memory to request from the scheduler for each
reduce task.
</description>
</property>

yarn-site.xml在所有节点上具有以下配置。
 <property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
</property>

<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>3072</value>
</property>

<property>
<name>yarn.nodemanager.resource.cpu-vcores</name>
<value>2</value>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>1024</value>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-vcores</name>
<value>1</value>
</property>

所有作业均成功运行,但在solrindex时失败,并出现以下异常
17/10/13 12:30:43 INFO mapreduce.Job:  map 0% reduce 0%
17/10/13 12:35:12 INFO mapreduce.Job: Task Id : attempt_1507879413631_0001_m_000000_1, Status : FAILED
AttemptID:attempt_1507879413631_0001_m_000000_1 Timed out after 10 secs
17/10/13 12:39:42 INFO mapreduce.Job: Task Id : attempt_1507879413631_0001_m_000000_2, Status : FAILED
AttemptID:attempt_1507879413631_0001_m_000000_2 Timed out after 10 secs
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

17/10/13 12:44:13 INFO mapreduce.Job: map 100% reduce 0%
17/10/13 12:44:13 INFO mapreduce.Job: Job job_1507879413631_0001 failed with state FAILED due to: Task failed task_1507879413631_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0

从纱日志之一具有以下信息
2017-10-13 12:30:12,442 INFO [IPC Server handler 0 on 37429] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1507879413631_0001_m_000000_0 is : 0.0
2017-10-13 12:30:15,450 INFO [IPC Server handler 5 on 37429] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1507879413631_0001_m_000000_0 is : 0.0
2017-10-13 12:30:41,870 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1507879413631_0001_m_000000_0: AttemptID:attempt_1507879413631_0001_m_000000_0 Timed out after 10 secs
2017-10-13 12:30:41,871 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507879413631_0001_m_000000_0 TaskAttempt Transitioned from RUNNING to FAIL_CONTAINER_CLEANUP
2017-10-13 12:30:41,872 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1507879413631_0001_01_000002 taskAttempt attempt_1507879413631_0001_m_000000_0
2017-10-13 12:30:41,873 INFO [ContainerLauncher #1] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1507879413631_0001_m_000000_0
2017-10-13 12:30:41,873 INFO [ContainerLauncher #1] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: Opening proxy : node1:39015
2017-10-13 12:30:41,896 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507879413631_0001_m_000000_0 TaskAttempt Transitioned from FAIL_CONTAINER_CLEANUP to FAIL_TASK_CLEANUP
2017-10-13 12:30:41,896 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT
2017-10-13 12:30:41,897 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507879413631_0001_m_000000_0 TaskAttempt Transitioned from FAIL_TASK_CLEANUP to FAILED
2017-10-13 12:30:41,902 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved node3 to /default-rack
2017-10-13 12:30:41,903 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 1 failures on node node1
2017-10-13 12:30:41,904 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1507879413631_0001_m_000000_1 TaskAttempt Transitioned from NEW to UNASSIGNED
2017-10-13 12:30:41,905 INFO [Thread-52] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added attempt_1507879413631_0001_m_000000_1 to list of failed maps
2017-10-13 12:30:42,847 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2017-10-13 12:30:42,865 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1507879413631_0001: ask=1 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:7168, vCores:1> knownNMs=3
2017-10-13 12:30:42,865 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1507879413631_0001_01_000002
2017-10-13 12:30:42,866 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:1 ContRel:0 HostLocal:0 RackLocal:1
2017-10-13 12:30:42,866 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1507879413631_0001_m_000000_0: Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

哪里有问题。我已经检查了所有日志,但是没有错误消息比上面的错误之一。其次,当我在独立计算机上运行此作业(伪模式)时,它已完成,没有任何问题。

最佳答案

您可以在日志中看到以下错误:
AttemptID:attempt_1507879413631_0001_m_000000_0 Timed out after 10 secs
您可以通过在此处修改设置来尝试增加超时:

<property>
<name>mapreduce.task.timeout</name>
<value>10000</value>
<description>The number of milliseconds before a task will be
terminated if it neither reads an input, writes an output, nor
updates its status string. A value of 0 disables the timeout.
</description>
</property>

您的时间设置为10秒,通常使用600的超时时间(10分钟)。因此您可以将其更改为600000。

关于apache - 带有Nutch的Apache Hadoop 2.7.4尝试由于任何原因失败,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/46725078/

27 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com