gpt4 book ai didi

hadoop - sqoop 错误 [主] tool.ImportTool

转载 作者:可可西里 更新时间:2023-11-01 16:22:56 28 4
gpt4 key购买 nike

您好,当我尝试将 MySql 表导入 hdfs 时出现错误:

sqoop import --connect jdbc:mysql://dba-virtual-machine/test --username socio --table textlines --columns line --target-dir /textlines

 2014-06-16 07:43:24,308 INFO  [main] manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2014-06-16 07:43:24,319 INFO [main] tool.CodeGenTool: Beginning code generation
2014-06-16 07:43:25,004 INFO [main] manager.SqlManager: Executing SQL statement: SELECT t.* FROM <code>textlines</code> AS t LIMIT 1
2014-06-16 07:43:25,026 INFO [main] manager.SqlManager: Executing SQL statement: SELECT t.* FROM <code>textlines</code> AS t LIMIT 1
2014-06-16 07:43:25,060 INFO [main] orm.CompilationManager: HADOOP_MAPRED_HOME is /home/socio/hadoop
Note: /tmp/sqoop-socio/compile/4d35eb51b72ffcde0e815ad25857257c/textlines.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2014-06-16 07:43:26,755 INFO [main] orm.CompilationManager: Writing jar file: /tmp/sqoop-socio/compile/4d35eb51b72ffcde0e815ad25857257c/textlines.jar
2014-06-16 07:43:26,869 WARN [main] manager.MySQLManager: It looks like you are importing from mysql.
2014-06-16 07:43:26,870 WARN [main] manager.MySQLManager: This transfer can be faster! Use the --direct
2014-06-16 07:43:26,870 WARN [main] manager.MySQLManager: option to exercise a MySQL-specific fast path.
2014-06-16 07:43:26,870 INFO [main] manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
2014-06-16 07:43:26,877 INFO [main] mapreduce.ImportJobBase: Beginning import of textlines
2014-06-16 07:43:26,878 INFO [main] Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/socio/hbase/lib/slf4j-log4j12-1.6.4.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/socio/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See <a href="http://www.slf4j.org/codes.html#multiple_bindings" rel="noreferrer noopener nofollow">http://www.slf4j.org/codes.html#multiple_bindings</a> for an explanation.
2014-06-16 07:43:27,894 WARN [main] util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2014-06-16 07:43:27,961 INFO [main] Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
2014-06-16 07:43:28,945 INFO [main] Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
2014-06-16 07:43:29,030 INFO [main] Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
2014-06-16 07:43:29,031 INFO [main] jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
2014-06-16 07:43:29,944 INFO [main] mapreduce.JobSubmitter: Cleaning up the staging area file:/tmp/hadoop-socio/mapred/staging/socio1192068052/.staging/job_local1192068052_0001
2014-06-16 07:43:29,945 ERROR [main] security.UserGroupInformation: PriviledgedActionException as:socio (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://mac:9000/home/socio/sqoop/lib/commons-io-1.4.jar
2014-06-16 07:43:29,945 ERROR [main] tool.ImportTool: Encountered IOException running import job: java.io.FileNotFoundException: File does not exist: hdfs://mac:9000/home/socio/sqoop/lib/commons-io-1.4.jar
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110)
at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:264)
at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:300)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:387)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)
at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:186)
at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:159)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:239)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:600)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:413)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:502)
at org.apache.sqoop.Sqoop.run(Sqoop.java:145)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229)
at org.apache.sqoop.Sqoop.main(Sqoop.java:238)

此文件 hdfs://mac:9000/home/socio/sqoop/lib/commons-io-1.4.jar在这里:home/socio/sqoop/lib/commons-io-1.4.jar .我不知道我要修改什么

等待回复

最佳答案

我在mappred-site.xml中添加解决了这个问题:

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>

另外,我遇​​到了hadoop和sqoop的兼容性问题

关于hadoop - sqoop 错误 [主] tool.ImportTool,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/24246423/

28 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com