johnsnowlabs-spark-nlp - Spark-NLP 预训练管道仅适用于 linux 系统吗？

Question

我正在尝试设置一个简单的代码，在其中传递数据帧并使用 johnSnowLabs Spark-NLP 库提供的预训练解释管道对其进行测试。
我正在使用 anaconda 的 jupyter 笔记本，并使用 apache toree 进行了 spark scala kernet 设置。每次我运行应该加载预训练管道的步骤时，它都会抛出一个 tensorflow 错误。有没有办法可以在本地 Windows 上运行它？

I was trying this in a maven project earlier and the same error had happened. Another colleague tried it on a linux system and it worked. Below is the code I have tried and the error that it gave.


import org.apache.spark.ml.PipelineModel
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
import org.apache.spark.sql.SparkSession

val spark: SparkSession = SparkSession
    .builder()
    .appName("test")
    .master("local[*]")
    .config("spark.driver.memory", "4G")
    .config("spark.kryoserializer.buffer.max", "200M")
    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
    .getOrCreate()

val testData = spark.createDataFrame(Seq(
    (1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
    (2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States"))).toDF("id", "text")
val pipeline = PretrainedPipeline("explain_document_dl", lang = "en") //this is where it gives error
val annotation = pipeline.transform(testData)

  annotation.show()

  annotation.select("entities.result").show(false)

出现以下错误:

Name: java.lang.UnsupportedOperationException Message: Spark NLP tried to load a Tensorflow Graph using Contrib module, but failed to load it on this system. If you are on Windows, this operation is not supported. Please try a noncontrib model. If not the case, please report this issue. Original error message:

Op type not registered 'BlockLSTM' in binary running on 'MyMachine'. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. StackTrace: Op type not registered 'BlockLSTM' in binary running on 'MyMachine'. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) tf.contrib.resampler should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.readGraph(TensorflowWrapper.scala:163) at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.read(TensorflowWrapper.scala:202) at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel$class.readTensorflowModel(TensorflowSerializeModel.scala:73) at com.johnsnowlabs.nlp.annotators.ner.dl.NerDLModel$.readTensorflowModel(NerDLModel.scala:134) at com.johnsnowlabs.nlp.annotators.ner.dl.ReadsNERGraph$class.readNerGraph(NerDLModel.scala:112) at com.johnsnowlabs.nlp.annotators.ner.dl.NerDLModel$.readNerGraph(NerDLModel.scala:134) at com.johnsnowlabs.nlp.annotators.ner.dl.ReadsNERGraph$$anonfun$2.apply(NerDLModel.scala:116) at com.johnsnowlabs.nlp.annotators.ner.dl.ReadsNERGraph$$anonfun$2.apply(NerDLModel.scala:116) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead$1.apply(ParamsAndFeaturesReadable.scala:31) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead$1.apply(ParamsAndFeaturesReadable.scala:30) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$class.com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead(ParamsAndFeaturesReadable.scala:30) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read$1.apply(ParamsAndFeaturesReadable.scala:41) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read$1.apply(ParamsAndFeaturesReadable.scala:41) at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:19) at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:8) at org.apache.spark.ml.util.DefaultParamsReader$.loadParamsInstance(ReadWrite.scala:652) at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$4.apply(Pipeline.scala:274) at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$4.apply(Pipeline.scala:272) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at org.apache.spark.ml.Pipeline$SharedReadWrite$.load(Pipeline.scala:272) at org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:348) at org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:342) at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:135) at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:129) at com.johnsnowlabs.nlp.pretrained.PretrainedPipelinenter code
heree.(PretrainedPipeline.scala:14)

Answer 1

我检查过，该管道内有一个 NER 模型。该 NER 模型是使用 TensorFlow 训练的，它有一些 contrib里面的代码只兼容基于 Unix 的操作系统，比如 Linux 和 macOS。这里有一个悬而未决的问题:

https://github.com/tensorflow/tensorflow/issues/26468

为此，他们发布了一些 兼容 Windows 名为 noncontrib 的管道.您可以将管道的名称更改为以下内容:

val pipeline = PretrainedPipeline("explain_document_dl_noncontrib", lang = "en")

所有预训练管道的来源:
https://nlp.johnsnowlabs.com/docs/en/pipelines

全面公开 :我是 Spark NLP 库的贡献者之一。

更新 : 自 Spark NLP 发布 2.4.0 ，所有模型和管道现在都是跨平台的: https://github.com/JohnSnowLabs/spark-nlp-models

如果您使用的是 Spark NLP 2.4.0 版本，这应该适用于 Linux、macOS 和 Windows:

val pipeline = PretrainedPipeline("explain_document_dl", lang = "en")