hadoop - 如何将Sqoop连接到多个hadoop集群

Question

有没有办法让Sqoop连接到不同的Hadoop集群，从而可以创建多个Sqoop作业将数据导出到多个hadoop集群？

Answer 1

to export data to multiple hadoop clusters

如果数据进入 Hadoop，那在技术上就是 Sqoop 导入

不清楚您目前如何从一台机器管理不同的集群，但您需要让 Sqoop 读取所有环境的 conf 文件夹

The sqoop command-line program is a wrapper which runs the bin/hadoop script shipped with Hadoop. If you have multiple installations of Hadoop present on your machine, you can select the Hadoop installation by setting the $HADOOP_HOME environment variable.

For example:

   $ HADOOP_HOME=/path/to/some/hadoop sqoop import --arguments...

or:

   $ export HADOOP_HOME=/some/path/to/hadoop
   $ sqoop import --arguments...

If $HADOOP_HOME is not set, Sqoop will use the default installation location for Cloudera’s Distribution for Hadoop, /usr/lib/hadoop.
The active Hadoop configuration is loaded from $HADOOP_HOME/conf/, unless the $HADOOP_CONF_DIR environment variable is set

https://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html#_controlling_the_hadoop_installation

根据您设置 Hadoop 的方式，Hortonworks 只有 Sqoop 1，而 Cloudera(可能还有 MapR)有 Sqoop2，这些指令可能不同，因为 Sqoop2 架构不同。