hadoop - 使用 oozie 在 S3 中移动文件

Question

我想使用 AWS oozie 在 S3 中移动文件。我要运行

aws s3 mv s3://temp/*.zip s3://temp/processed_files/. --recursive

我如何在 oozie 中执行此操作？

编辑 1

2015-11-12 10:18:55,758  WARN ShellActionExecutor:542 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] Launcher exception: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
java.io.IOException: Cannot run program "move.sh" (in directory "/mnt1/yarn/usercache/hadoop/appcache/application_1446129655727_0421/container_1446129655727_0421_01_000002"): error=2, No such file or directory
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1047)
    at org.apache.oozie.action.hadoop.ShellMain.execute(ShellMain.java:93)
    at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:55)
    at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37)
    at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:226)
    at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:65)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:452)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:344)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.IOException: error=2, No such file or directory
    at java.lang.UNIXProcess.forkAndExec(Native Method)
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:187)
    at java.lang.ProcessImpl.start(ProcessImpl.java:130)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1028)
    ... 17 more

2015-11-12 10:18:55,838  INFO ActionEndXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@sh] ERROR is considered as FAILED for SLA
2015-11-12 10:18:55,880  INFO ActionStartXCommand:539 - USER[hadoop] GROUP[-] TOKEN[] APP[rad_workflow] JOB[0000118-151029144311676-oozie-oozi-W] ACTION[0000118-151029144311676-oozie-oozi-W@killemail] Start action [0000118-151029144311676-oozie-oozi-W@killemail] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]

Hue Oozie 屏幕截图和错误日志的屏幕截图。

Answer 1

如果您按照 Helmut Zechmann 所说的进行操作但仍然无法正常工作，请检查问题是否与您的 shell 脚本有关，该脚本具有 windows 行结尾而不是 unix 行结尾。

尝试使用 Notepad++ 或其他工具将其转换为正确的格式。