hadoop - 尝试使用 Sqoop 将数据从 Cassandra 传输到 Hadoop 时出错

Question

我使用 tar ball 安装了 Cassandra-2.2.7。 Cassandra 运行良好，我在键空间中创建了表。现在我想将这个表数据传输到 HDFS 中。我正在使用 Sqoop-1.4.6。我将以下库文件保存在 $SQOOP_HOME/lib/下，

apache-cassandra-2.2.7.jar
apache-cassandra-thrift-2.2.7.jar
cassandra-jdbc-1.2.5.jar
cassandra-all-1.2.0.jar
libthrift-0.8.0.jar
thrift-server-0.3.7.jar

然后我运行命令，

bin/sqoop list-tables --driver org.apache.cassandra.cql.jdbc.CassandraDriver --connect jdbc:cassandra://127.0.0.1:7199/sample --username cassandra --password cassandra

当我运行这个命令时，出现以下错误，

Warning: /usr/lib/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
16/08/15 16:52:41 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
16/08/15 16:52:41 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/08/15 16:52:41 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
16/08/15 16:52:41 INFO manager.SqlManager: Using default fetchSize of 1000
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/hadoop-1.2.1/lib/sf2hadoop.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/hadoop-1.2.1/lib/slf4j-log4j12-1.4.3.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [ch.qos.logback.classic.util.ContextSelectorStaticBinder]
16/08/15 16:52:41 ERROR manager.SqlManager: Error reading database metadata: java.sql.SQLNonTransientConnectionException: org.apache.thrift.transport.TTransportException
java.sql.SQLNonTransientConnectionException: org.apache.thrift.transport.TTransportException
	at org.apache.cassandra.cql.jdbc.CassandraConnection.<init>(CassandraConnection.java:159)
	at org.apache.cassandra.cql.jdbc.CassandraDriver.connect(CassandraDriver.java:92)
	at java.sql.DriverManager.getConnection(DriverManager.java:664)
	at java.sql.DriverManager.getConnection(DriverManager.java:247)
	at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:885)
	at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52)
	at org.apache.sqoop.manager.SqlManager.listTables(SqlManager.java:520)
	at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)
	at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
	at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: org.apache.thrift.transport.TTransportException
	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
	at org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:129)
	at org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
	at org.apache.cassandra.thrift.Cassandra$Client.recv_describe_cluster_name(Cassandra.java:1247)
	at org.apache.cassandra.thrift.Cassandra$Client.describe_cluster_name(Cassandra.java:1235)
	at org.apache.cassandra.cql.jdbc.CassandraConnection.<init>(CassandraConnection.java:130)
	... 13 more
Could not retrieve tables list from server
16/08/15 16:52:41 ERROR tool.ListTablesTool: manager.listTables() returned null

我认为这是某种 Thrift 错误，但即使我将 lib-thrift jar 替换为其他几个版本也无法纠正。

Answer 1

那个 Cassandra JDBC 驱动程序很旧。上次我尝试使用它时，我需要针对最新的 cassandra thrift 库手动编译它并修复一些编译错误。

我建议使用 Spark传输数据的作业，或者如果您必须使用 sqoop，则取决于您在做什么 DataStax Enterprise 包括一个用于 Cassandra 的自定义 sqoop 驱动程序。