mysql - 名称错误 : name '_mysql' is not defined -- On airflow start in MacOSX

Question

关于标题问题的文章很多，但没有一篇对我有用。详细错误如下:

Traceback (most recent call last):
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/MySQLdb/__init__.py", line 18, in <module>
    from . import _mysql
ImportError: dlopen(/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/MySQLdb/_mysql.cpython-37m-darwin.so, 2): Symbol not found: _mysql_affected_rows
  Referenced from: /Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/MySQLdb/_mysql.cpython-37m-darwin.so
  Expected in: flat namespace
 in /Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/MySQLdb/_mysql.cpython-37m-darwin.so

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/hiteshagarwal/Documents/venv/bin/airflow", line 25, in <module>
    from airflow.configuration import conf
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/airflow/__init__.py", line 47, in <module>
    settings.initialize()
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/airflow/settings.py", line 403, in initialize
    configure_adapters()
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/airflow/settings.py", line 326, in configure_adapters
    import MySQLdb.converters
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/MySQLdb/__init__.py", line 24, in <module>
    version_info, _mysql.version_info, _mysql.__file__
NameError: name '_mysql' is not defined

如果我使用 pip 从 python 环境中卸载 mysqlclient 然后我得到这样的错误然后我的 airflow 网络服务器和调度程序运行但不是 airflow worker 而在上述情况下任何 airflow 命令都不起作用。

 -------------- celery@Hiteshs-MBP v4.4.7 (cliffs)
--- ***** ----- 
-- ******* ---- Darwin-20.3.0-x86_64-i386-64bit 2021-05-09 16:11:23
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         airflow.executors.celery_executor:0x7fe6603a9668
- ** ---------- .> transport:   sqla+mysql://airflow:**@localhost:3306/airflow
- ** ---------- .> results:     mysql://airflow:**@localhost:3306/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> default          exchange=default(direct) key=default
                

[tasks]
  . airflow.executors.celery_executor.execute_command

[2021-05-09 16:11:23,308: INFO/MainProcess] Connected to sqla+mysql://airflow:**@localhost:3306/airflow
[2021-05-09 16:11:23,318: CRITICAL/MainProcess] Unrecoverable error: ModuleNotFoundError("No module named 'MySQLdb'")
Traceback (most recent call last):
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/celery/worker/consumer/tasks.py", line 41, in start
    c.connection, on_decode_error=c.on_decode_error,
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/celery/app/amqp.py", line 304, in TaskConsumer
    **kw
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/messaging.py", line 386, in __init__
    self.revive(self.channel)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/messaging.py", line 408, in revive
    self.declare()
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/messaging.py", line 421, in declare
    queue.declare()
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/entity.py", line 611, in declare
    self._create_queue(nowait=nowait, channel=channel)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/entity.py", line 620, in _create_queue
    self.queue_declare(nowait=nowait, passive=False, channel=channel)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/entity.py", line 655, in queue_declare
    nowait=nowait,
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 531, in queue_declare
    self._new_queue(queue, **kwargs)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 101, in _new_queue
    self._get_or_create(queue)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 80, in _get_or_create
    obj = self.session.query(self.queue_cls) \
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 75, in session
    _, Session = self._open()
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 65, in _open
    engine = self._engine_from_config()
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 54, in _engine_from_config
    return create_engine(conninfo.hostname, **transport_options)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/sqlalchemy/engine/__init__.py", line 520, in create_engine
    return strategy.create(*args, **kwargs)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/Users/hiteshagarwal/Documents/venv/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 118, in dbapi
    return __import__("MySQLdb")
ModuleNotFoundError: No module named 'MySQLdb'
Starting flask
 * Serving Flask app "airflow.bin.cli" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
[2021-05-09 16:11:23,815] {_internal.py:122} INFO -  * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
(venv) hiteshagarwal@Hiteshs-MBP Documents % 

我在 airflow.cfg 中使用的数据库配置 --

executor = CeleryExecutor

sql_alchemy_conn = postgresql+psycopg2://hiteshagarwal:<password>@127.0.0.1:5432/airflow  --postgres

broker_url = sqla+mysql+pymysql://airflow:<password>@localhost:3306/airflow --mysql

celery_result_backend = db+mysql+pymysql://hiteshagarwal:<password>@localhost/airflow   --mysql

---

更新

安装pip install apache-airflow-backport-providers-mysql后，出现以下错误:

(airflow-env) hiteshagarwal@Hiteshs-MBP Documents % airflow worker                                     
 
 -------------- celery@Hiteshs-MBP v4.4.7 (cliffs)
--- ***** ----- 
-- ******* ---- Darwin-20.3.0-x86_64-i386-64bit 2021-05-09 17:02:30
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         airflow.executors.celery_executor:0x7fd1fafa79b0
- ** ---------- .> transport:   sqla+mysql://airflow:**@localhost:3306/airflow
- ** ---------- .> results:     mysql://airflow:**@localhost:3306/airflow
- *** --- * --- .> concurrency: 16 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> default          exchange=default(direct) key=default
                

[tasks]
  . airflow.executors.celery_executor.execute_command

[2021-05-09 17:02:31,159: INFO/MainProcess] Connected to sqla+mysql://airflow:**@localhost:3306/airflow
[2021-05-09 17:02:31,178: CRITICAL/MainProcess] Unrecoverable error: ImportError('dlopen(/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/_mysql.cpython-37m-darwin.so, 2): Symbol not found: _mysql_affected_rows\n  Referenced from: /Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/_mysql.cpython-37m-darwin.so\n  Expected in: flat namespace\n in /Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/_mysql.cpython-37m-darwin.so')
Traceback (most recent call last):
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/celery/worker/worker.py", line 208, in start
    self.blueprint.start(self)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/celery/bootsteps.py", line 369, in start
    return self.obj.start()
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/celery/worker/consumer/consumer.py", line 318, in start
    blueprint.start(self)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/celery/bootsteps.py", line 119, in start
    step.start(parent)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/celery/worker/consumer/tasks.py", line 41, in start
    c.connection, on_decode_error=c.on_decode_error,
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/celery/app/amqp.py", line 304, in TaskConsumer
    **kw
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/messaging.py", line 386, in __init__
    self.revive(self.channel)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/messaging.py", line 408, in revive
    self.declare()
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/messaging.py", line 421, in declare
    queue.declare()
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/entity.py", line 611, in declare
    self._create_queue(nowait=nowait, channel=channel)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/entity.py", line 620, in _create_queue
    self.queue_declare(nowait=nowait, passive=False, channel=channel)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/entity.py", line 655, in queue_declare
    nowait=nowait,
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/transport/virtual/base.py", line 531, in queue_declare
    self._new_queue(queue, **kwargs)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 101, in _new_queue
    self._get_or_create(queue)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 80, in _get_or_create
    obj = self.session.query(self.queue_cls) \
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 75, in session
    _, Session = self._open()
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 65, in _open
    engine = self._engine_from_config()
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/kombu/transport/sqlalchemy/__init__.py", line 54, in _engine_from_config
    return create_engine(conninfo.hostname, **transport_options)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/sqlalchemy/engine/__init__.py", line 500, in create_engine
    return strategy.create(*args, **kwargs)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line 118, in dbapi
    return __import__("MySQLdb")
  File "/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/MySQLdb/__init__.py", line 18, in <module>
    import _mysql
ImportError: dlopen(/Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/_mysql.cpython-37m-darwin.so, 2): Symbol not found: _mysql_affected_rows
  Referenced from: /Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/_mysql.cpython-37m-darwin.so
  Expected in: flat namespace
 in /Users/hiteshagarwal/Documents/WMall/airflow-env/lib/python3.7/site-packages/_mysql.cpython-37m-darwin.so
Starting flask
 * Serving Flask app "airflow.bin.cli" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
[2021-05-09 17:02:32,079] {_internal.py:122} INFO -  * Running on http://0.0.0.0:8793/ (Press CTRL+C to quit)
(airflow-env) hiteshagarwal@Hiteshs-MBP Documents % 

我尝试了网上提供的许多解决方案，但没有一个有效。下面是我在我的 venv 中安装的库列表:

alabaster==0.7.12
alembic==1.6.2
amqp==2.6.1
apache-airflow==1.10.12
apispec==1.3.3
appdirs==1.4.4
argcomplete==1.12.3
astor==0.8.1
attrs==19.3.0
autoflake==1.4
Babel==2.9.1
bandit==1.7.0
billiard==3.6.4.0
black==20.8b1
cached-property==1.5.2
cattrs==1.0.0
celery==4.4.7
certifi==2020.12.5
chardet==4.0.0
click==7.1.2
colorama==0.4.4
colorlog==4.0.2
configparser==3.5.3
coverage==5.5
croniter==0.3.37
darglint==1.8.0
defusedxml==0.7.1
dill==0.3.3
distlib==0.3.1
dnspython==2.1.0
doc8==0.8.1
docutils==0.17.1
email-validator==1.1.2
eradicate==1.0
filelock==3.0.12
flake8==3.9.2
flake8-2020==1.6.0
flake8-bandit==2.1.2
flake8-broken-line==0.2.1
flake8-bugbear==19.8.0
flake8-builtins==1.5.3
flake8-commas==2.0.0
flake8-comprehensions==3.4.0
flake8-debugger==3.2.1
flake8-docstrings==1.6.0
flake8-eradicate==0.4.0
flake8-fixme==1.1.1
flake8-isort==3.0.1
flake8-logging-format==0.6.0
flake8-mutable==1.2.0
flake8-polyfill==1.0.2
flake8-quotes==2.1.2
flake8-rst-docstrings==0.0.12
flake8-string-format==0.2.3
flake8-variables-names==0.0.3
Flask==1.1.2
Flask-Admin==1.5.4
Flask-AppBuilder==2.3.4
Flask-Babel==1.0.0
Flask-Caching==1.3.3
Flask-JWT-Extended==3.25.1
Flask-Login==0.4.1
Flask-OpenID==1.2.5
Flask-SQLAlchemy==2.4.4
flask-swagger==0.2.14
Flask-WTF==0.14.3
funcsigs==1.0.2
future==0.18.2
gitdb==4.0.7
GitPython==3.1.15
graphviz==0.16
greenlet==1.1.0
gunicorn==20.1.0
idna==2.10
imagesize==1.2.0
importlib-metadata==4.0.1
iniconfig==1.1.1
iso8601==0.1.14
isort==4.3.21
itsdangerous==1.1.0
Jinja2==2.11.3
json-merge-patch==0.2
jsonschema==3.2.0
kombu==4.6.11
lazy-object-proxy==1.6.0
lockfile==0.12.2
Mako==1.1.4
Markdown==2.6.11
MarkupSafe==1.1.1
marshmallow==2.21.0
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.25.0
mccabe==0.6.1
mypy-extensions==0.4.3
natsort==7.1.1
numpy==1.20.2
packaging==20.9
pandas==1.1.5
pathspec==0.8.1
pbr==5.6.0
pendulum==1.4.4
pep8-naming==0.9.1
pluggy==0.13.1
prison==0.1.3
psutil==5.8.0
psycopg2-binary==2.8.6
py==1.10.0
pycodestyle==2.7.0
pydocstyle==5.1.1
pyflakes==2.3.1
Pygments==2.9.0
PyJWT==2.1.0
PyMySQL==1.0.2
pyparsing==2.4.7
pyrsistent==0.17.3
pytest==6.2.4
pytest-cov==2.11.1
python-daemon==2.3.0
python-dateutil==2.8.1
python-editor==1.0.4
python-nvd3==0.15.0
python-slugify==4.0.1
python3-openid==3.2.0
pytz==2021.1
pytzdata==2020.1
pyupgrade==2.15.0
PyYAML==5.4.1
regex==2021.4.4
requests==2.25.1
restructuredtext-lint==1.3.2
setproctitle==1.2.2
six==1.16.0
smmap==4.0.0
snowballstemmer==2.1.0
Sphinx==2.4.4
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==1.0.3
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.4
SQLAlchemy==1.3.23
SQLAlchemy-JSONField==0.9.0
SQLAlchemy-Utils==0.37.2
stevedore==3.3.0
tabulate==0.8.9
tenacity==4.12.0
testfixtures==6.17.1
text-unidecode==1.3
thrift==0.13.0
tokenize-rt==4.1.0
toml==0.10.2
tox==3.23.1
tox-travis==0.12
typed-ast==1.4.3
typing-extensions==3.10.0.0
tzlocal==1.5.1
unicodecsv==0.14.1
urllib3==1.26.4
vine==1.3.0
virtualenv==20.4.6
wemake-python-styleguide==0.14.1
Werkzeug==0.16.1
WTForms==2.3.3
zipp==3.4.1
zope.deprecation==4.4.0

Answer 1

https://newbedev.com/nameerror-name-mysql-is-not-defined-after-setting-change-to-mysql

上面的链接对我有帮助

pip install mysql-client
export DYLD_LIBRARY_PATH="/usr/local/mysql/lib:$PATH"