mysql - Talend 上的 SqoopImport 组件错误

Question

我正在尝试在 Talend (v.6.0.1) 中运行一个非常简单的作业，它基本上包含一个 Sqoop 导入，用于将数据从 MySql 迁移到在 Cloudera 发行版(CDH5.4 YARN 模式)上运行的 Hive 数据库。我在本地运行 Talend，数据库在单独的机器上运行。我已将 Cloudera 连接创建为 Talend 元数据，它似乎工作正常。我已按照说明的步骤进行操作 here创造工作。有一个连接 Cloudera 集群(也运行 Sqoop)和 MySQL 数据库的 ssh 隧道，所以在 MySQL 的连接路径中我使用的是本地主机。

当我尝试运行我的作业时产生以下错误堆栈跟踪:

    Exception in component tSqoopImport_1
java.lang.Exception: The Sqoop import job has failed. Please check the logs.
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.tSqoopImport_1Process(recent_orders_migration.java:512)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.tLibraryLoad_1Process(recent_orders_migration.java:365)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.runJobInTOS(recent_orders_migration.java:786)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.main(recent_orders_migration.java:643)
[ERROR]: org.apache.sqoop.manager.SqlManager - Error executing statement: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1118)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:343)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2308)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2122)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:774)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:49)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:409)
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:375)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:289)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:880)
    at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:739)
    at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:762)
    at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:270)
    at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
    at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
    at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
    at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.tSqoopImport_1Process(recent_orders_migration.java:508)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.tLibraryLoad_1Process(recent_orders_migration.java:365)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.runJobInTOS(recent_orders_migration.java:786)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.main(recent_orders_migration.java:643)
Caused by: java.net.ConnectException: Connection refused
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at java.net.Socket.connect(Socket.java:538)
    at java.net.Socket.<init>(Socket.java:434)
    at java.net.Socket.<init>(Socket.java:244)
    at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256)
    at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:292)
    ... 33 more
[ERROR]: org.apache.sqoop.tool.ImportTool - Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
    at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
    at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
    at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
    at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
    at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.tSqoopImport_1Process(recent_orders_migration.java:508)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.tLibraryLoad_1Process(recent_orders_migration.java:365)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.runJobInTOS(recent_orders_migration.java:786)
    at dw_migration.recent_orders_migration_0_1.recent_orders_migration.main(recent_orders_migration.java:643)

关于可能导致此问题的原因的任何想法？

谢谢，塔玛拉。

Answer 1

尝试使用最新的 mysql jar 文件。可能会奏效。

我尝试了不同的方式，我在元数据连接中创建了 mysql 连接。查看下图进行配置

然后使用下图所示的 JDBC 属性

如果您需要更多信息，请告诉我