scala - Spark 和 Scala : saveAsTextFile() exception-6ren

scala - Spark 和 Scala : saveAsTextFile() exception

转载作者：可可西里更新时间：2023-11-01 14:18:41

我是 Spark & Scala 的新手，调用 saveAsTextFile() 后出现异常。希望有人能帮忙...

这是我的输入.txt:

Hello World, I'm a programmer 
Hello World, I'm a programmer

这是在 CMD 上运行“spark-shell”后的信息:

C:\Users\Nhan Tran>spark-shell
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
Spark context Web UI available at http://DLap:4040
Spark context available as 'sc' (master = local[*], app id = local-1526374303208).
Spark session available as 'spark'.
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.3.0
      /_/

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_172)
Type in expressions to have them evaluated.
Type :help for more information.

scala>

这是我的代码:

scala> val inputfile = sc.textFile("input.txt")
scala> val counts = inputfile.flatMap(line => line.split(" ")).map(word => (word, 1)).reduceByKey(_+_);
scala> counts.toDebugString
scala> counts.cache()
scala> counts.saveAsTextFile("output")

一切正常，直到我调用 counts.saveAsTextFile("output")，这是我的异常(exception)情况:

2018-05-15 14:28:27 ERROR Executor:91 - Exception in task 0.0 in stage 3.0 (TID 4)
    java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native Method)
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
            at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
            at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
            at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
            at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:224)
            at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:118)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:79)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
            at org.apache.spark.scheduler.Task.run(Task.scala:109)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
    2018-05-15 14:28:27 ERROR Executor:91 - Exception in task 1.0 in stage 3.0 (TID 5)
    java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native Method)
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
            at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
            at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
            at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
            at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:224)
            at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:118)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:79)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
            at org.apache.spark.scheduler.Task.run(Task.scala:109)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
    2018-05-15 14:28:27 WARN  TaskSetManager:66 - Lost task 0.0 in stage 3.0 (TID 4, localhost, executor driver): java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native Method)
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
            at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
            at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
            at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
            at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:224)
            at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:118)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:79)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
            at org.apache.spark.scheduler.Task.run(Task.scala:109)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

    2018-05-15 14:28:27 ERROR TaskSetManager:70 - Task 0 in stage 3.0 failed 1 times; aborting job
    2018-05-15 14:28:27 ERROR SparkHadoopWriter:91 - Aborting job job_20180515142827_0007.
    org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 4, localhost, executor driver): java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native Method)
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
            at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
            at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
            at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
            at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:224)
            at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:118)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:79)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
            at org.apache.spark.scheduler.Task.run(Task.scala:109)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

    Driver stacktrace:
            at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1599)
            at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1587)
            at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1586)
            at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
            at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
            at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1586)
            at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
            at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:831)
            at scala.Option.foreach(Option.scala:257)
            at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:831)
            at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1820)
            at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1769)
            at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1758)
            at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
            at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:642)
            at org.apache.spark.SparkContext.runJob(SparkContext.scala:2027)
            at org.apache.spark.SparkContext.runJob(SparkContext.scala:2048)
            at org.apache.spark.SparkContext.runJob(SparkContext.scala:2080)
            at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:78)
            at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1096)
            at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1094)
            at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1094)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
            at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
            at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1094)
            at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1067)
            at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)
            at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
            at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
            at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1032)
            at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:958)
            at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)
            at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
            at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
            at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:957)
            at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1493)
            at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1472)
            at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1472)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
            at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
            at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
            at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1472)
            at $line30.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:26)
            at $line30.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:31)
            at $line30.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(<console>:33)
            at $line30.$read$$iw$$iw$$iw$$iw$$iw.<init>(<console>:35)
            at $line30.$read$$iw$$iw$$iw$$iw.<init>(<console>:37)
            at $line30.$read$$iw$$iw$$iw.<init>(<console>:39)
            at $line30.$read$$iw$$iw.<init>(<console>:41)
            at $line30.$read$$iw.<init>(<console>:43)
            at $line30.$read.<init>(<console>:45)
            at $line30.$read$.<init>(<console>:49)
            at $line30.$read$.<clinit>(<console>)
            at $line30.$eval$.$print$lzycompute(<console>:7)
            at $line30.$eval$.$print(<console>:6)
            at $line30.$eval.$print(<console>)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:786)
            at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1047)
            at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:638)
            at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
            at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
            at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
            at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
            at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
            at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
            at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)
            at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:681)
            at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:395)
            at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:415)
            at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:923)
            at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
            at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:909)
            at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
            at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:909)
            at org.apache.spark.repl.Main$.doMain(Main.scala:76)
            at org.apache.spark.repl.Main$.main(Main.scala:56)
            at org.apache.spark.repl.Main.main(Main.scala)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.lang.reflect.Method.invoke(Method.java:498)
            at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
            at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
            at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
            at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
            at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
            at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native Method)
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
            at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
            at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
            at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
            at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:224)
            at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:118)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:79)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
            at org.apache.spark.scheduler.Task.run(Task.scala:109)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
    org.apache.spark.SparkException: Job aborted.
      at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:96)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1096)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1094)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1.apply(PairRDDFunctions.scala:1094)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
      at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
      at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1094)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1067)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1032)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
      at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
      at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1032)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:958)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)
      at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:958)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
      at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
      at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:957)
      at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1493)
      at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1472)
      at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1472)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
      at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
      at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
      at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1472)
      ... 49 elided
    Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 1 times, most recent failure: Lost task 0.0 in stage 3.0 (TID 4, localhost, executor driver): java.lang.UnsatisfiedLinkError: org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Ljava/lang/String;JJJI)Ljava/io/FileDescriptor;
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileWithMode0(Native Method)
            at org.apache.hadoop.io.nativeio.NativeIO$Windows.createFileOutputStreamWithMode(NativeIO.java:559)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:219)
            at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:209)
            at org.apache.hadoop.fs.RawLocalFileSystem.createOutputStreamWithMode(RawLocalFileSystem.java:307)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:296)
            at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:328)
            at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:398)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:461)
            at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
            at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
            at org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
            at org.apache.spark.internal.io.HadoopMapRedWriteConfigUtil.initWriter(SparkHadoopWriter.scala:224)
            at org.apache.spark.internal.io.SparkHadoopWriter$.org$apache$spark$internal$io$SparkHadoopWriter$$executeTask(SparkHadoopWriter.scala:118)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:79)
            at org.apache.spark.internal.io.SparkHadoopWriter$$anonfun$3.apply(SparkHadoopWriter.scala:78)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
            at org.apache.spark.scheduler.Task.run(Task.scala:109)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)

最佳答案

来自错误日志。好像您在 Windows 上使用 Spark。您可以尝试以下故障排除步骤:-

search for winuntils.exe path

cd path

winutils.exe chmod 777 output (directory where you store output)

Lets try again to store output (counts.saveAsTextFile("output"))

关于scala - Spark 和 Scala : saveAsTextFile() exception，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/50344874/

文章推荐： c++ - mingw 构建错误 : undefined reference to `__chkstk_ms'

文章推荐： c# - 如何在Windows任务栏中制作一个文本框？

scala - Scala 可以在参数中允许自由类型参数吗(Scala 类型参数是一等公民吗？)？
我有一些 Scala 代码，它用两个不同版本的类型参数化函数做了一些漂亮的事情。我已经从我的应用程序中简化了很多，但最后我的代码充满了形式 w(f[Int],f[Double]) 的调用。哪里w()是
scala - Scala 脚本可以引用同一目录中其他未编译的 scala 代码吗？
如果我在同一目录中有两个单独的未编译的 scala 文件: // hello.scala object hello { def world() = println("hello world") }
scala - Spark Scala 获取类未找到 scala.Any
val schema = df.schema val x = df.flatMap(r => (0 until schema.length).map { idx => ((idx, r.g
scala - 错误 : scala: No 'scala-library*.jar' in Scala compiler library
环境: Play 2.3.0/Scala 2.11.1/IntelliJ 13.1 我使用 Typesafe Activator 1.2.1 用 Scala 2.11.1 创建一个新项目。项目创建好后
scala - 如何使我的 Scala jar 库中的类可以在 Scala 控制台和 Scala 脚本中访问？
我只是想知道如何使用我自己的类扩展 Scala 控制台和“脚本”运行程序，以便我可以通过使用实际的 Scala 语言与其通信来实际使用我的代码？我应将 jar 放在哪里，以便无需临时配置即可从每个 S
scala - ensime scala 错误(未找到类 scala.Array，未找到对象 scala)
我已经根据 README.md 文件安装了 ensime，但是，我在低级 ensime-server 缓冲区中出现以下错误: 信息: fatal error :scala.tools.nsc.Miss
scala - Scala 中的函数相等，是 Scala 中的函数对象吗？
我正在阅读《Scala 编程》一书。在书中，它说“一个函数文字被编译成一个类，当在运行时实例化时它是一个函数值”。并且它提到“函数值是对象，因此您可以根据需要将它们存储在变量中”。所以我尝试检查函数
scala - 如何在 Scala 原生应用程序中运行 Scala 测试？
我有 hello world scala native 应用程序，想对此应用程序运行小型 scala 测试我使用通常的测试命令，但它抛出异常: NativeMain.scala object Nati
scala - 从 Scala 编译器插件生成 Scala 代码树
有few resources在网络上，在编写与代码模式匹配的 Scala 编译器插件方面很有指导意义，但这些对生成代码(构建符号树)没有帮助。我应该从哪里开始弄清楚如何做到这一点？ (如果有比手动构建
scala - 使用仅适用于较旧 Scala 版本的 Scala 库
我是 Scala 的新手。但是，我用创建了一个中等大小的程序。斯卡拉 2.9.0 .现在我想使用一个仅适用于的开源库斯卡拉 2.7.7 . 是吗可能在我的 Scala 2.9.0 程序中使用这个
scala - Scala 酸洗是否适用于 Scala 2.11？
有没有办法在 Scala 2.11 中使用 scala-pickling？我在 sonatype 存储库中尝试了唯一的 scala-pickling_2.11 工件，但它似乎不起作用。我收到消息:
scala - 如何从 Scala 本身获取 Scala 版本？
这与命令行编译器选项无关。如何以编程方式获取代码内的 Scala 版本？或者，Eclipse Scala 插件 v2 在哪里存储 scalac 的路径？最佳答案这无需访问 scala-compi
scala - 避免 Scala 内存泄漏 - Scala 构造函数
我正在阅读《Scala 编程》一书，并在第 6 章中的类 Rational 实现中遇到了一些问题。这是我的 Rational 类的初始版本(基于本书) class Rational(numerato
scala - 是否有必要在新的 scala 项目中添加我的自定义 scala 库依赖项？
我是 Scala 新手，我正在尝试开发一个使用自定义库的小项目。我在库内创建了一个mysql连接池。这是我的库的build.sbt organization := "com.learn" name :
scala - 如何在编译 Scala 文件之前在 SBT Build.scala 中运行 Scala 代码？
我正在尝试运行一些 Scala 代码，只是暂时打印出“Hello”，但我希望在 SBT 项目中编译 Scala 代码之前运行 Scala 代码。我发现在 build.sbt 中有以下工作。 compi
scala - maven Scala 插件默认使用什么 Scala 版本？
Here链接到 maven Scala 插件使用。但没有提到它使用的究竟是什么 Scala 版本。我创建了具有以下配置的 Maven Scala 项目: org.scala-tools
scala - Scala 上的类型不匹配用于理解 : scala. concurrent.Future
我对 Scala 还很陌生，请多多包涵。我有一堆包裹在一个大数组中的 future 。 future 已经完成了查看几 TB 数据的辛勤工作，在我的应用程序结束时，我想总结上述 future 的所有结
scala - 带有 scala 宏的非 scala 源位置
我有一个 scala 宏，它依赖于通过包含其位置的静态字符串指定的任意 xml 文件。 def myMacro(path: String) = macro myMacroImpl def myMacr
scala - 缺少扩展函数的参数类型 (Scala)
这是我的功能: def sumOfSquaresOfOdd(in: Seq[Int]): Int = { in.filter(_%2==1).map(_*_).reduce(_+_) } 为什么我
scala - Scala 中两个时间戳之间的秒数差异
这个问题在这里已经有了答案: Calculating the difference between two Java date instances (45 个答案) 关闭 5 年前。所以我有一个这

可可西里

个人简介

我是一名优秀的程序员,十分优秀！

作者热门文章

滴滴打车优惠券免费领取

全站热门文章

首页

博学

6Ren·AI

商城

scala - Spark 和 Scala : saveAsTextFile() exception