python - Dask分布式计算反序列化错误

Question

我有一个在 Linux 计算机上运行的 dask-scheduler 和工作程序，我尝试从 Windows 10 笔记本电脑发送网格搜索任务。

from dask.distributed import Client, progress
client = Client('10.133.20.17:8786',processes=False,threads_per_worker=4,n_workers=1, memory_limit='2GB')
from sklearn.datasets import make_classification
from sklearn.svm import SVC
from sklearn.externals import joblib
from sklearn.model_selection import GridSearchCV
import pandas as pd

param_grid = {"C": [0.001, 0.01, 0.1, 0.5, 1.0, 2.0, 5.0, 10.0],
          "kernel": ['rbf', 'poly', 'sigmoid'],
          "shrinking": [True, False]}

grid_search = GridSearchCV(SVC(gamma='auto', random_state=0, probability=True),
                       param_grid=param_grid,
                       return_train_score=False,
                       iid=True,
                       cv=3,
                       n_jobs=-1)

with joblib.parallel_backend('dask'):
    grid_search.fit(X, y)

我收到此错误distributed.protocol.core - 严重 - 无法反序列化

Traceback (most recent call last):
   File "C:\Users\kselvam\SOFTS\anaconda3\lib\site-packages\distributed\protocol\core.py", line 132, in loads
value = _deserialize(head, fs, deserializers=deserializers)
   File "C:\Users\kselvam\SOFTS\anaconda3\lib\site-packages\distributed\protocol\serialize.py", line 183, in deserialize
dumps, loads, wants_context = families[name]
KeyError: None
tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x0000027207EB7950>, <Future finished exception=CancelledError(['_fit_and_score-batch-a4f6fd8cd0354973bb6b49d7f1530390'])>)

我不明白为什么它无法反序列化。当我作为本地集群(Windows)启动调度程序时，一切正常。操作系统的更改是否会影响序列化过程？我在linux和windows上分别有相同版本的sklearn(0.21.3)，tornado(5.1.1)和dask(1.0.0)。

Answer 1

我的第一个猜测是您的软件在整个集群中并不统一。您可能需要使用以下内容验证您的软件环境:

client.get_versions(check=True)