azure - 如何在 HDInsight Spark/Jupyter 上使用 Avro？

Question

我试图读取 HDInsight Spark/Jupyter 集群内的 avro 文件，但得到了

u'Failed to find data source: com.databricks.spark.avro. Please find an Avro package at http://spark.apache.org/third-party-projects.html;'
Traceback (most recent call last):
  File "/usr/hdp/current/spark2-client/python/pyspark/sql/readwriter.py", line 159, in load
    return self._df(self._jreader.load(path))
  File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
    answer, self.gateway_client, self.target_id, self.name)
  File "/usr/hdp/current/spark2-client/python/pyspark/sql/utils.py", line 69, in deco
    raise AnalysisException(s.split(': ', 1)[1], stackTrace)
AnalysisException: u'Failed to find data source: com.databricks.spark.avro. Please find an Avro package at http://spark.apache.org/third-party-projects.html;'

<小时/>

df = spark.read.format("com.databricks.spark.avro").load("wasb://<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="9cfff3f2e8fdf5f2f9eef2fdf1f9dcfdfdfdb2b2b2fdfdfdb2fef0f3feb2fff3eef9b2ebf5f2f8f3ebefb2f2f9e8" rel="noreferrer noopener nofollow">[email protected]</a>/...")

如何解决这个问题？我似乎需要安装该软件包，但如何在 HDInsight 上安装该软件包？

Answer 1

您只需按照以下文章操作即可

https://learn.microsoft.com/en-in/azure/hdinsight/spark/apache-spark-jupyter-notebook-use-external-packages

对于 HDInsight 3.3 和 HDInsight 3.4

您将在笔记本中添加以下单元格

%%configure 
{ "packages":["com.databricks:spark-avro_2.10:0.1"] }

对于 HDInsight 3.5

您将在笔记本中添加以下单元格

%%configure
{ "conf": {"spark.jars.packages": "com.databricks:spark-avro_2.10:0.1" }}

对于 HDInsight 3.6

您将在笔记本中添加以下单元格

%%configure
{ "conf": {"spark.jars.packages": "com.databricks:spark-avro_2.11:4.0.0" }}