Hadoop 1.0.4 - Windows 7 + cygwin - Tasktracker 抛出异常

Question

执行 map reduce 程序时，Tasktracker 不工作。tasktraker 日志:

2014-01-31 15:58:51,021 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201401311556_0001_m_-523923709
2014-01-31 15:58:51,021 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201401311556_0001_m_-523923709 spawned.
2014-01-31 15:58:51,026 INFO org.apache.hadoop.mapred.JvmManager: JVM Not killed jvm_201401311556_0001_m_-523923709 but just removed
2014-01-31 15:58:51,026 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201401311556_0001_m_-523923709 exited with exit code -1. Number of tasks it ran: 0
2014-01-31 15:58:51,028 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201401311556_0001_m_000003_0 : Child Error
java.io.IOException: Task process exit with nonzero status of -1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
2014-01-31 15:58:54,029 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 2
2014-01-31 15:58:56,321 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201401311556_0001_m_000003_1 task's state:UNASSIGNED
2014-01-31 15:58:56,322 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201401311556_0001_m_000003_1 which needs 1 slots
2014-01-31 15:58:56,322 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 2 and trying to launch attempt_201401311556_0001_m_000003_1 which needs 1 slots
2014-01-31 15:58:56,367 INFO org.apache.hadoop.mapred.JvmManager: In JvmRunner constructed JVM ID: jvm_201401311556_0001_m_-1949280651
2014-01-31 15:58:56,368 INFO org.apache.hadoop.mapred.JvmManager: JVM Runner jvm_201401311556_0001_m_-1949280651 spawned.
2014-01-31 15:58:56,372 INFO org.apache.hadoop.mapred.JvmManager: JVM Not killed jvm_201401311556_0001_m_-1949280651 but just removed
2014-01-31 15:58:56,372 INFO org.apache.hadoop.mapred.JvmManager: JVM : jvm_201401311556_0001_m_-1949280651 exited with exit code -1. Number of tasks it ran: 0
2014-01-31 15:58:56,372 WARN org.apache.hadoop.mapred.TaskRunner: attempt_201401311556_0001_m_000003_1 : Child Error
java.io.IOException: Task process exit with nonzero status of -1.
    at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:258)
2014-01-31 15:58:57,285 WARN org.apache.hadoop.mapred.TaskLog: Index file for the log of attempt_201401311556_0001_m_000003_0 does not exist.
2014-01-31 15:58:57,327 WARN org.apache.hadoop.mapred.TaskLog: Failed to retrieve stdout log for task: attempt_201401311556_0001_m_000003_0
java.io.FileNotFoundException: C:\cygwin\home\LalmaniG\hadoop-1.0.4\logs\userlogs\job_201401311556_0001\attempt_201401311556_0001_m_000003_0\stdout (The system cannot find the path specified)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
    at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:455)
    at org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
    at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
    at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)
    at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
    at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
    at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
    at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
    at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
    at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2014-01-31 15:58:57,381 WARN org.apache.hadoop.mapred.TaskLog: Index file for the log of attempt_201401311556_0001_m_000003_0 does not exist.
2014-01-31 15:58:57,422 WARN org.apache.hadoop.mapred.TaskLog: Failed to retrieve stderr log for task: attempt_201401311556_0001_m_000003_0
java.io.FileNotFoundException: C:\cygwin\home\LalmaniG\hadoop-1.0.4\logs\userlogs\job_201401311556_0001\attempt_201401311556_0001_m_000003_0\stderr (The system cannot find the path specified)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
    at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:455)
    at org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81)
    at org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
    at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
    at org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:835)
    at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
    at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
    at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
    at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
    at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
    at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
    at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    at org.mortbay.jetty.Server.handle(Server.java:326)

Answer 1

问题是在 Windows 中，Java IO 不能使用符号链接(symbolic link)或联结。因此，您需要创建一个将/cygwin 指向/的符号链接(symbolic link)。您不能在路径中使用驱动器号，因此不要使用

C:\cygwin\home\LalmaniG\hadoop-1.0.4...

使用

/cygwin/home/LalmaniG/hadoop-1.0.4/...

确保对 Hadoop 临时目录也执行此操作。