python - Pig Hadoop Stream 帮助

Question

我在运行 pig 流媒体时遇到问题。当我启动一个交互式 pig 实例时(仅供引用，我正在通过 SSH/Putty 在交互式 pig AWS EMR 实例的主节点上执行此操作)只有一台机器我的 pig 流媒体工作完美(它也适用于我的 Windows Cloudera VM 镜像).但是，当我切换到使用多台计算机时，它只是停止工作并出现各种错误。

注意:

• 我能够在多计算机实例上毫无问题地运行没有任何流命令的 Pig 脚本。
• 我所有的 pig 工作都是在 pig MapReduce 模式下完成的，而不是 –x 本地模式。
• 我的 python 脚本 (stream1.py) 在上面有这个 #!/usr/bin/env python

下面是我到目前为止尝试过的选项的小示例(以下所有命令都是在主节点/主节点上的 grunt shell 中完成的，我通过 ssh/putty 访问):

这就是我如何将 python 文件放到 mater 节点上以便可以使用它:

cp s3n://darin.emr-logs/stream1.py stream1.py
copyToLocal stream1.py /home/hadoop/stream1.py
chmod 755 stream1.py

这些是我的各种直播尝试:

cooc = stream ct_pag_ph through `stream1.py`
dump coco;
ERROR 2090: Received Error while processing the reduce plan: 'stream1.py ' failed with exit status: 127

cooc = stream ct_pag_ph through `python stream1.py`;
dump coco;
ERROR 2090: Received Error while processing the reduce plan: 'python stream1.py ' failed with exit status: 2

DEFINE X `stream1.py`; 
cooc = stream ct_bag_ph through X;
dump coco;
ERROR 2090: Received Error while processing the reduce plan: 'stream1.py ' failed with exit status: 127

DEFINE X `stream1.py`; 
cooc = stream ct_bag_ph through `python X`;
dump coco;
ERROR 2090: Received Error while processing the reduce plan: 'python X ' failed with exit status: 2

DEFINE X `stream1.py` SHIP('stream1.py');
cooc = STREAM ct_bag_ph THROUGH X;
dump cooc;
ERROR 2017: Internal error creating job configuration.

DEFINE X `stream1.py` SHIP('/stream1.p');
cooc = STREAM ct_bag_ph THROUGH X;
dump cooc;

DEFINE X `stream1.py` SHIP('stream1.py') CACHE('stream1.py');
cooc = STREAM ct_bag_ph THROUGH X;
ERROR 2017: Internal error creating job configuration.

define X 'python /home/hadoop/stream1.py' SHIP('/home/hadoop/stream1.py');
cooc = STREAM ct_bag_ph THROUGH X;

Answer 1

DEFINE X `stream1.py` SHIP('stream1.py');

根据您的先决条件并且在您当前的本地目录中有 stream1.py 对我来说似乎有效。

确定这一点的方法:

DEFINE X `python stream1.py` SHIP('/local/path/stream1.py');

SHIP的目标就是复制所有任务的工作目录下的命令。