windows - 在 yarn cluster (linux) : Error no sheme for Filesystem "C" 上从客户端 (windows) 执行 spark

Question

我想将一个 spark 应用程序(只是一个简单的 Hello World 应用程序)部署到我的 hadoop 集群。在我的 Windows 机器上使用 spark 提交，我使用 --master yarn 在客户端模式下执行应用程序。连接到 hadoop 集群是成功的，在集群上的日志文件中可以看到。 (hadoop conf文件已经从集群下载下来，保存在客户端windows机器上，环境变量已经设置好)。

使用 hadoop 2.7 和 spark 1.6

这是使用的 spark-submit 命令:

>spark-submit --master yarn --class "SimpleApp" ..\..\SimpleApp\target\scala-2.11\simple-project_2.11-1.0.jar --deploy-mode client

这是返回的错误信息:

16/08/10 10:52:45 ERROR SparkContext: Error initializing SparkContext.
java.io.IOException: No FileSystem for scheme: C

完整的错误信息:

Warning: Ignoring non-spark config property: fs.hdfs.impl=org.apache.hadoop.hdfs.DistributedFileSystem
16/08/10 10:52:32 INFO SparkContext: Running Spark version 1.6.2
16/08/10 10:52:32 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/08/10 10:52:32 INFO SecurityManager: Changing view acls to: sensored,sensored
16/08/10 10:52:32 INFO SecurityManager: Changing modify acls to: sensored,sensored
16/08/10 10:52:32 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sensored,sensored); users with modify permissions: Set(sensored,sensored)
16/08/10 10:52:33 INFO Utils: Successfully started service 'sparkDriver' on port 52340.
16/08/10 10:52:33 INFO Slf4jLogger: Slf4jLogger started
16/08/10 10:52:33 INFO Remoting: Starting remoting
16/08/10 10:52:34 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@sensored,sensored]
16/08/10 10:52:34 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 52353.
16/08/10 10:52:34 INFO SparkEnv: Registering MapOutputTracker
16/08/10 10:52:34 INFO SparkEnv: Registering BlockManagerMaster
16/08/10 10:52:34 INFO DiskBlockManager: Created local directory at C:\Users\sensored,sensored\AppData\Local\Temp\blockmgr-35b65f9a-47a3-4d1e-940f-be8bc9c78b8f
16/08/10 10:52:34 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
16/08/10 10:52:34 INFO SparkEnv: Registering OutputCommitCoordinator
16/08/10 10:52:34 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/08/10 10:52:34 INFO SparkUI: Started SparkUI at http://10.61.156.198:4040
16/08/10 10:52:34 INFO HttpFileServer: HTTP File server directory is C:\Users\sensored,sensored\AppData\Local\Temp\spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e\httpd-e39c2eda-b9aa-40df-95c9-eae44b53c4ee
16/08/10 10:52:34 INFO HttpServer: Starting HTTP Server
16/08/10 10:52:34 INFO Utils: Successfully started service 'HTTP file server' on port 52356.
16/08/10 10:52:34 INFO SparkContext: Added JAR file:/C:/Users/sensored,sensored/Documents/spark-1.6.2-bin-hadoop2.6/bin/../../SimpleApp/target/scala-2.11/simple-project_2.11-1.0.jar at http://sensored,sensored:52356/jars
/simple-project_2.11-1.0.jar with timestamp 1470819154823
16/08/10 10:52:35 INFO TimelineClientImpl: Timeline service address: http://sensored,sensored:8188/ws/v1/timeline/
16/08/10 10:52:35 INFO RMProxy: Connecting to ResourceManager at sensored,sensored/sensored,sensored.188:8050
16/08/10 10:52:36 INFO Client: Requesting a new application from cluster with 2 NodeManagers
16/08/10 10:52:36 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (221184 MB per container)
16/08/10 10:52:36 INFO Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
16/08/10 10:52:36 INFO Client: Setting up container launch context for our AM
16/08/10 10:52:36 INFO Client: Setting up the launch environment for our AM container
16/08/10 10:52:36 INFO Client: Preparing resources for our AM container
16/08/10 10:52:36 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because UNIX Domain sockets are not available on Windows.
16/08/10 10:52:36 INFO Client: Uploading resource file:/C:/Users/sensored,sensored/Documents/spark-1.6.2-bin-hadoop2.6/lib/spark-assembly-1.6.2-hadoop2.6.0.jar -> hdfs://sensored,sensored:8020/user/sensored,sensored/.sparkSt
aging/application_1468399950865_0375/spark-assembly-1.6.2-hadoop2.6.0.jar
16/08/10 10:52:41 INFO Client: Uploading resource file:/C:/Users/sensored,sensored/AppData/Local/Temp/spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e/__spark_conf__7713144618669536696.zip -> hdfs://sensored,sensored:80
20/user/sensored,sensored/.sparkStaging/application_1468399950865_0375/__spark_conf__7713144618669536696.zip
16/08/10 10:52:41 INFO SecurityManager: Changing view acls to: sensored,sensored
16/08/10 10:52:41 INFO SecurityManager: Changing modify acls to: sensored,sensored
16/08/10 10:52:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(sensored,sensored); users with modify permissions: Set(sensored,sensored)
16/08/10 10:52:41 INFO Client: Submitting application 375 to ResourceManager
16/08/10 10:52:41 INFO YarnClientImpl: Submitted application application_1468399950865_0375
16/08/10 10:52:42 INFO Client: Application report for application_1468399950865_0375 (state: ACCEPTED)
16/08/10 10:52:42 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: N/A
         ApplicationMaster RPC port: -1
         queue: default
         start time: 1470819161610
         final status: UNDEFINED
         tracking URL: http://sensored,sensored:8088/proxy/application_1468399950865_0375/
         user: sensored,sensored
16/08/10 10:52:43 INFO Client: Application report for application_1468399950865_0375 (state: ACCEPTED)
16/08/10 10:52:44 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null)
16/08/10 10:52:44 INFO YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> sensored,sensored, PROXY_URI_BASES -> http://sensored,sensored:8088/proxy/application_1468399950865_0375), /proxy/application_1468399950865_0375
16/08/10 10:52:44 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
16/08/10 10:52:44 INFO Client: Application report for application_1468399950865_0375 (state: RUNNING)
16/08/10 10:52:44 INFO Client:
         client token: N/A
         diagnostics: N/A
         ApplicationMaster host: 10.3.131.188
         ApplicationMaster RPC port: 0
         queue: default
         start time: 1470819161610
         final status: UNDEFINED
         tracking URL: http://sensored,sensored:8088/proxy/application_1468399950865_0375/
         user: sensored,sensored
16/08/10 10:52:44 INFO YarnClientSchedulerBackend: Application application_1468399950865_0375 has started running.
16/08/10 10:52:44 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 52383.
16/08/10 10:52:44 INFO NettyBlockTransferService: Server created on 52383
16/08/10 10:52:44 INFO BlockManagerMaster: Trying to register BlockManager
16/08/10 10:52:44 INFO BlockManagerMasterEndpoint: Registering block manager 10.61.156.198:52383 with 511.1 MB RAM, BlockManagerId(driver, sensored,sensored, 52383)
16/08/10 10:52:44 INFO BlockManagerMaster: Registered BlockManager
16/08/10 10:52:45 ERROR SparkContext: Error initializing SparkContext.
java.io.IOException: No FileSystem for scheme: C
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1686)
        at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:66)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:547)
        at SimpleApp$.main(SimpleApp.scala:10)
        at SimpleApp.main(SimpleApp.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/10 10:52:45 INFO SparkUI: Stopped Spark web UI at http://10.61.156.198:4040
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Shutting down all executors
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Interrupting monitor thread
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Asking each executor to shut down
16/08/10 10:52:45 INFO YarnClientSchedulerBackend: Stopped
16/08/10 10:52:45 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/08/10 10:52:45 INFO MemoryStore: MemoryStore cleared
16/08/10 10:52:45 INFO BlockManager: BlockManager stopped
16/08/10 10:52:45 INFO BlockManagerMaster: BlockManagerMaster stopped
16/08/10 10:52:45 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
16/08/10 10:52:45 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
16/08/10 10:52:45 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
16/08/10 10:52:45 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" java.io.IOException: No FileSystem for scheme: C
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
        at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1686)
        at org.apache.spark.scheduler.EventLoggingListener.<init>(EventLoggingListener.scala:66)
        at org.apache.spark.SparkContext.<init>(SparkContext.scala:547)
        at SimpleApp$.main(SimpleApp.scala:10)
        at SimpleApp.main(SimpleApp.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/08/10 10:52:45 INFO ShutdownHookManager: Shutdown hook called
16/08/10 10:52:45 INFO ShutdownHookManager: Deleting directory C:\Users\sensored,sensored\AppData\Local\Temp\spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e\httpd-e39c2eda-b9aa-40df-95c9-eae44b53c4ee
16/08/10 10:52:45 INFO ShutdownHookManager: Deleting directory C:\Users\sensored,sensored\AppData\Local\Temp\spark-ec3f2bdc-e11e-48b5-a3f6-7d8ae97b8a5e

Answer 1

改用spark 2.0后，问题没有出现了!