python - 无法在 Spark worker 上导入 Pandas

Question

我已经能够设置一个 virtualenv 并在我的 Spark 集群的所有工作节点中安装我需要的库，但我仍然无法导入 pandas:

Traceback (most recent call last):
  File "/scratch/nas/2/larissa/ExperimentsML/app/experimenter/sklearn/sklearn-spark-tests.py", line 209, in <module>
    main()
  File "/scratch/nas/2/larissa/ExperimentsML/app/experimenter/sklearn/sklearn-spark-tests.py", line 202, in main
    print sc.parallelize(experiments).map(lambda experiment: run_experiment(df, input_dict, experiment)).collect()
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/rdd.py", line 808, in collect
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 79 in stage 0.0 failed 4 times, most recent failure: Lost task 79.3 in stage 0.0 (TID 82, 172.18.8.2, executor 0): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 174, in main
    process()
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/worker.py", line 169, in process
    serializer.dump_stream(func(split_index, iterator), outfile)
  File "/scratch/1/larissa/spark-2.1.1-bin-hadoop2.7/python/lib/pyspark.zip/pyspark/serializers.py", line 268, in dump_stream
    vs = list(itertools.islice(iterator, batch))
  File "/scratch/nas/2/larissa/ExperimentsML/app/experimenter/sklearn/sklearn-spark-tests.py", line 202, in <lambda>
    print sc.parallelize(experiments).map(lambda experiment: run_experiment(df, input_dict, experiment)).collect()
  File "/scratch/nas/2/larissa/ExperimentsML/app/experimenter/sklearn/sklearn-spark-tests.py", line 61, in run_experiment
    import pandas
ImportError: No module named pandas

我有这个代码:

sc.parallelize(experiments).map(lambda 实验: run_experiment(df, input_dict, experiment)).collect()

在 run_experiment 中，我有以下导入:

def run_experiment(df, input_dict, experiment):
    from sklearn.pipeline import Pipeline
    from sklearn.svm import SVC
    from sklearn import datasets, linear_model, tree
    from sklearn.preprocessing import *
    from sklearn.tree import DecisionTreeClassifier
    from sklearn.linear_model import LinearRegression, LogisticRegression
    from sklearn.neighbors import KNeighborsClassifier
    from sklearn.naive_bayes import GaussianNB
    from sklearn.ensemble import RandomForestClassifier
    from sklearn import metrics
    from sklearn.metrics import *

    import pandas

    ...

我只得到 pandas 的错误，所以 scikit-learn 安装成功。

在我的 install_virtualenv.sh 脚本中，我有以下库，并且根据日志，它们都已正确安装。

pip install --upgrade pip
pip install urllib3[secure]
pip install pyopenssl ndg-httpsclient pyasn1
pip install cloudpickle==0.2.2
pip install distributed==1.16.1
pip install joblib==0.11
pip install scipy==0.18.1
pip install numpy==1.12.0
#pip install pandas==0.19.2
#pip install pandas --upgrade
pip install pickleshare==0.7.4
pip install py4j==0.10.4
pip install python-dateutil==2.6.0
pip install --upgrade requests
pip install scikit-learn==0.18.1
pip install sklearn==0.0
pip install sklearn-pandas==1.3.0
pip install spark-sklearn==0.2.0
easy_install pandas

使用 pip list 我得到:

asn1crypto (0.22.0)
backports-abc (0.5)
backports.ssl-match-hostname (3.5.0.1)
certifi (2017.4.17)
cffi (1.10.0)
chardet (3.0.4)
click (6.7)
cloudpickle (0.2.2)
cryptography (1.9)
dask (0.14.1)
dask-searchcv (0.0.2)
distributed (1.16.1)
enum34 (1.1.6)
futures (3.1.1)
HeapDict (1.0.0)
idna (2.5)
ipaddress (1.0.18)
joblib (0.11)
msgpack-python (0.4.8)
ndg-httpsclient (0.4.2)
numpy (1.12.0)
pandas (0.19.2)
pathlib2 (2.2.1)
pickleshare (0.7.4)
pip (9.0.1)
psutil (5.2.2)
py4j (0.10.4)
pyasn1 (0.2.3)
pycparser (2.17)
pyOpenSSL (17.0.0)
python-dateutil (2.6.0)
pytz (2017.2)
requests (2.17.3)
scandir (1.5)
scikit-learn (0.18.1)
scipy (0.18.1)
setuptools (2.2)
singledispatch (3.4.0.3)
six (1.10.0)
sklearn (0.0)
sklearn-pandas (1.3.0)
sortedcollections (0.5.3)
sortedcontainers (1.5.7)
spark-sklearn (0.2.0)
tblib (1.3.2)
toolz (0.8.2)
tornado (4.5.1)
urllib3 (1.21.1)
zict (0.1.2)

如您所见，我什至尝试使用 easy_install，但仍然无法导入 pandas。知道为什么会这样吗？

谢谢!

Answer 1

听起来您的 Spark worker 指向 python 的默认/系统安装(例如 /usr/bin/python)而不是您的 virtualenv。您可以通过设置 PYSPARK_PYTHON 环境变量告诉 Spark 使用您的 virtualenv，例如export PYSPARK_PYTHON=/path/to/my/python。