gpt4 book ai didi

scala - 使用 HDFS 的 Scalding 教程 : Data is missing from one or more paths in: List(tutorial/data/hello. txt)

转载 作者:可可西里 更新时间:2023-11-01 16:26:15 28 4
gpt4 key购买 nike

当我尝试使用命令运行 Scalding 教程(https://github.com/Cascading/scalding-tutorial/)时配置 ssh 和 rsync 之后:

$ scripts/scald.rb --hdfs tutorial/Tutorial0.scala

我收到以下错误:

com.twitter.scalding.InvalidSourceException: [com.twitter.scalding.TextLineWrappedArray(tutorial/data/hello.txt)] Data is missing from one or more paths in: List(tutorial/data/hello.txt)

尽管文件 tutorial/data/hello.txt 确实存在,但仍会发生此错误。

如何解决这个问题?

标准输出:

$ scripts/scald.rb --hdfs tutorial/Tutorial0.scala
scripts/scald.rb:194: warning: already initialized constant SCALA_LIB_DIR
dkondratev@hadoop-n002.maxus.lan's password:
dkondratev@hadoop-n002.maxus.lan's password:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/phoenix/phoenix-4.0.0.2.1.2.1-471-client.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
14/07/07 19:05:45 INFO Configuration.deprecation: mapred.min.split.size is deprecated. Instead, use mapreduce.input.fileinputformat.split.minsize
14/07/07 19:05:45 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
Exception in thread "main" java.lang.Throwable: GUESS: Data is missing from the path you provided.
If you know what exactly caused this error, please consider contributing to GitHub via following link.
https://github.com/twitter/scalding/wiki/Common-Exceptions-and-possible-reasons#comtwitterscaldinginvalidsourceexception
at com.twitter.scalding.Tool$.main(Tool.scala:132)
at com.twitter.scalding.Tool.main(Tool.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

最佳答案

我认为您遇到的问题是您告诉 Scalding 在 HDFS 中运行,但是您作为输入提供的文件在您的本地文件系统中,而不是在 HDFS 中。在运行示例之前,将文件上传到您的 HDFS:

hadoop fs -mkdir tutorial
hadoop fs -mkdir tutorial/data
hadoop fs -put tutorial/data/hello.txt tutorial/data/hello.txt

关于scala - 使用 HDFS 的 Scalding 教程 : Data is missing from one or more paths in: List(tutorial/data/hello. txt),我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/24614595/

28 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com