scala - 使用databricks dbutils-api 的问题

Question

我们如何尝试使用它？

• 将依赖项“com.databricks”%%“dbutils-api”%“0.0.1”添加到build.sbt

• 用代码编写:

  导入 com.databricks.dbutils_v1.DBUtilsHolder.dbutils
类作业{
//商业逻辑 ..
val jobResult: 字符串 = ???
dbutils.notebook.exit(jobResult)
}

当我们将代码打包到 jar 中时，我们得到:

[error] missing or invalid dependency detected while loading class file 'NotebookUtils.class'.
[error] Could not access term common in package com.databricks.backend,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'NotebookUtils.class' was compiled against an incompatible version of com.databricks.backend.
[error] one error found
[error] (compile:compileIncremental) Compilation failed

Answer 1

原因甚至是最新版本的dbutils-api目前(0.0.3)依赖于 com.databricks.backend.common (它既不是 dbutils-api 的一部分，也不是任何其他公共(public)数据 block 库的一部分)。

trait NotebookUtils ... {
...
  def getContext() : com.databricks.backend.common.rpc.CommandContext
  def setContext(ctx : com.databricks.backend.common.rpc.CommandContext) : scala.Unit
}

同时:

jar tvf dbutils-api_2.11-0.0.3.jar | grep backend
   439 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/Default.class
  1337 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/FileInfo$$anonfun$hashCode$1.class
  1265 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/FileInfo$$anonfun$hashCode$2.class
  4109 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/FileInfo.class
  1256 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/GrantInfo.class
  1480 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/MountInfo.class
   452 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/PackageGroup.class
     0 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/
     0 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/
     0 Tue Jan 01 11:00:00 MSK 1980 com/databricks/backend/daemon/dbutils/

最可能的原因是com.databricks.backend.common在 dbutils 定义文件中标有提供的范围(但我不能 100% 确定，因为这个库在开源中不可用)。我宁愿将其视为 Databricks 库中的错误。