gpt4 book ai didi

python - Mrjob 步骤失败。怎么调试呢?

转载 作者:太空宇宙 更新时间:2023-11-03 21:44:02 25 4
gpt4 key购买 nike

我正在尝试在 EMR 集群中运行示例 mrjob。我已在 AWS 仪表板中手动创建 EMR 集群并启动 mrjob,如下所示

python keywords.py -r emr s3://commoncrawl/crawl-data/CC-MAIN-2018-34/wet.paths.gz --cluster-id j-22GFG1FUGS12L

作业失败并出现以下错误消息

Using configs in /etc/mrjob.conf
Using s3://mrjob-07d6e1cbb9127021/tmp/ as our temp dir on S3
emr_api_params is deprecated and does nothing. Please use extra_cluster_params instead
Could not infer endpoint for bucket commoncrawl; assuming defaults
Copying local files to s3://mrjob-07d6e1cbb9127021/tmp/keywords.ec2-user.20181002.164319.430013/files/...
Adding our job to existing cluster j-22GFG1FUGS12L
Creating temp directory /tmp/phonenumers.ec2-user.20181002.164319.430013
Connect to resource manager at: http://localhost:40750/cluster
Waiting for Step 1 of 1 (s-2OZF2A4TZTS06) to complete...
RUNNING for 0:00:18
FAILED
Cluster j-22GFG1FUGS12L is WAITING: Cluster ready after last step failed.
Attempting to fetch counters from logs...
Waiting 10 minutes for logs to transfer to S3... (ctrl-c to skip)

如何查看失败消息?

最佳答案

请参阅EMR docs如何获取作业和任务日志。因为深入研究集群日志并不简单,所以我建议使用 mrjob's local runner 彻底测试 Python 代码。 。

关于python - Mrjob 步骤失败。怎么调试呢?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/52626504/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com