- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
我编写了一个将市场数据导入 MariaDB 数据库的 python 脚本。为了加快导入速度,我决定使用模块线程。因此,首先,一个函数会使用 url 填充一个队列,从中下载数据并将其导入我的数据库。如果脚本第一次运行,无论给定多少线程,我都没有问题,因此第一次导入可用数据。再次运行时,将检查存储的数据并在必要时进行更新。否则将被忽略。
奇怪的是,当脚本以超过 1 个线程运行并且第二次涉及更新机制时,我收到一条错误消息。
我在这个问题上苦苦挣扎了一个多星期,我可以帮助你给我一个提示可能是什么问题。
#!/usr/local/bin/python3
import requests
import queue
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry
import ujson
from datetime import datetime
import mysql.connector as mariadb
from mysql.connector import Error
from mysql.connector import errorcode
from threading import Thread
import time
num_threads = 4
threads = []
urls = queue.Queue()
def create_url():
try:
mariadb_connection = mariadb.connect(dbstuff)
cursor = mariadb_connection.cursor()
cursor.execute('SELECT type_id from tbl_items')
item_list = cursor.fetchall()
for row in item_list:
url = 'https://esi.evetech.net/latest/markets/10000002/orders/?datasource=tranquility&order_type=all&page=1&type_id=' + \
str(row[0])
urls.put(url)
return urls
except mariadb.Error as error:
mariadb_connection.rollback() # rollback if any exception occured
print("Failed retrieving itemtypes from tbl_items table {}".format(error))
finally:
if mariadb_connection.is_connected():
cursor.close()
mariadb_connection.close()
def import_mo_jita(i, urls):
station_id = 60003760
print("worker:", i)
try:
mariadb_connection = mariadb.connect(dbstuff)
cursor = mariadb_connection.cursor()
while (True):
url = urls.get()
print("Worker %s processes %s queue# %s" % (i, url, urls.qsize()))
if url == None:
break
s = requests.Session()
retries = Retry(total=5, backoff_factor=1, status_forcelist=[502, 503, 504])
s.mount('https://', HTTPAdapter(max_retries=retries))
jsonraw = s.get(url)
jsondata = ujson.loads(jsonraw.text)
for row in jsondata:
if (row['location_id'] == station_id):
cursor.execute('INSERT INTO tbl_mo_jita_esi_tmp (order_id) VALUES (%s)', (row['order_id'], ))
mariadb_connection.commit()
cursor.execute('SELECT order_id, price, volume FROM tbl_mo_jita WHERE order_id = %s', (row['order_id'], ))
db_data = cursor.fetchall()
#print (db_data)
if len(db_data) != 0:
for x in db_data:
db_order_id = x[0]
db_price = x[1]
db_volume = x[2]
if len(db_data) != 0:
if db_price == row['price'] and db_volume == row['volume_remain']:
continue
else:
print("updating order#", row['order_id'])
cursor.execute('UPDATE tbl_mo_jita SET volume = %s, price = %s WHERE order_id = %s', (row['volume_remain'], row['price'], row['order_id'], ))
mariadb_connection.commit()
else:
print("newly inserting order#", row['order_id'])
cursor.execute('INSERT INTO tbl_mo_jita (type_id, order_id, ordertype,volume, price) VALUES (%s,%s,%s,%s,%s)', (row['type_id'], row['order_id'], row['is_buy_order'], row['volume_remain'], row['price'], ))
mariadb_connection.commit()
else:
continue
urls.task_done()
except mariadb.Error as error:
mariadb_connection.rollback() # rollback if any exception occured
print("Failed retrieving itemtypes from tbl_items table {}".format(error))
finally:
if mariadb_connection.is_connected():
cursor.close()
mariadb_connection.close()
def cleanup_mo():
try:
mariadb_connection = mariadb.connect(dbstuff)
cursor = mariadb_connection.cursor()
cursor.execute('SELECT order_id FROM tbl_mo_jita')
list_mo_sql = cursor.fetchall()
cursor.execute('SELECT order_id FROM tbl_mo_jita_esi_tmp')
list_mo_esi = cursor.fetchall()
list_mo_purge = list(set(list_mo_sql)-set(list_mo_esi))
print(len(list_mo_purge))
for row in list_mo_purge:
cursor.execute('DELETE FROM tbl_mo_jita WHERE order_id = %s', ((row[0]), ))
cursor.execute('TRUNCATE tbl_mo_jita_esi_tmp')
mariadb_connection.commit()
except mariadb.Error as error:
mariadb_connection.rollback() # rollback if any exception occured
print("Failed retrieving itemtypes from tbl_items table {}".format(error))
finally:
if mariadb_connection.is_connected():
cursor.close()
mariadb_connection.close()
create_url()
for i in range(num_threads):
urls.put(None)
for i in range(num_threads):
worker = Thread(target=import_mo_jita, args=(i, urls,))
worker.setDaemon(True)
threads.append(worker)
worker.start()
for worker in threads:
worker.join()
cleanup_mo()
Fatal Python error: Segmentation fault
Thread 0x00007f21a5079700 (most recent call first):
File "/home/gregadmin/.local/lib/python3.6/site-packages/mysql/connector/connection_cext.py", line 329 in commit
File "import_mo_jita.py", line 93 in import_mo_jita
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Current thread 0x00007f21a587a700 (most recent call first):
File "import_mo_jita.py", line 99 in import_mo_jita
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f21a607b700 (most recent call first):
File "import_mo_jita.py", line 87 in import_mo_jita
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f21a687c700 (most recent call first):
File "/usr/lib/python3.6/ssl.py", line 631 in read
File "/usr/lib/python3.6/ssl.py", line 874 in read
File "/usr/lib/python3.6/ssl.py", line 1012 in recv_into
File "/usr/lib/python3.6/socket.py", line 586 in readinto
File "/usr/lib/python3.6/http/client.py", line 258 in _read_status
File "/usr/lib/python3.6/http/client.py", line 297 in begin
File "/usr/lib/python3.6/http/client.py", line 1331 in getresponse
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383 in _make_request
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601 in urlopen
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440 in send
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630 in send
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520 in request
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533 in get
File "import_mo_jita.py", line 87 in import_mo_jita
File "/usr/lib/python3.6/threading.py", line 864 in run
File "/usr/lib/python3.6/threading.py", line 916 in _bootstrap_inner
File "/usr/lib/python3.6/threading.py", line 884 in _bootstrap
Thread 0x00007f21ad64d740 (most recent call first):
File "/usr/lib/python3.6/threading.py", line 1072 in _wait_for_tstate_lock
File "/usr/lib/python3.6/threading.py", line 1056 in join
File "import_mo_jita.py", line 187 in <module>
Segmentation fault (core dumped)
最佳答案
很可能您的数据库客户端有问题。
use_pure
driver option .也试着玩一下... 关于python - 多线程应用程序中的 _wait_for_tstate_lock 错误,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/55779500/
我已经使用 vue-cli 两个星期了,直到今天一切正常。我在本地建立这个项目。 https://drive.google.com/open?id=0BwGw1zyyKjW7S3RYWXRaX24tQ
您好,我正在尝试使用 python 库 pytesseract 从图像中提取文本。请找到代码: from PIL import Image from pytesseract import image_
我的错误 /usr/bin/ld: errno: TLS definition in /lib/libc.so.6 section .tbss mismatches non-TLS reference
我已经训练了一个模型,我正在尝试使用 predict函数但它返回以下错误。 Error in contrasts<-(*tmp*, value = contr.funs[1 + isOF[nn]])
根据Microsoft DataConnectors的信息我想通过 this ODBC driver 创建一个从 PowerBi 到 PostgreSQL 的连接器使用直接查询。我重用了 Micros
我已经为 SoundManagement 创建了一个包,其中有一个扩展 MediaPlayer 的类。我希望全局控制这个变量。这是我的代码: package soundmanagement; impo
我在Heroku上部署了一个应用程序。我正在使用免费服务。 我经常收到以下错误消息。 PG::Error: ERROR: out of memory 如果刷新浏览器,就可以了。但是随后,它又随机发生
我正在运行 LAMP 服务器,这个 .htaccess 给我一个 500 错误。其作用是过滤关键字并重定向到相应的域名。 Options +FollowSymLinks RewriteEngine
我有两个驱动器 A 和 B。使用 python 脚本,我在“A”驱动器中创建一些文件,并运行 powerscript,该脚本以 1 秒的间隔将驱动器 A 中的所有文件复制到驱动器 B。 我在 powe
下面的函数一直返回这个错误信息。我认为可能是 double_precision 字段类型导致了这种情况,我尝试使用 CAST,但要么不是这样,要么我没有做对...帮助? 这是错误: ERROR: i
这个问题已经有答案了: Syntax error due to using a reserved word as a table or column name in MySQL (1 个回答) 已关闭
我的数据库有这个小问题。 我创建了一个表“articoli”,其中包含商品的品牌、型号和价格。 每篇文章都由一个 id (ID_ARTICOLO)` 定义,它是一个自动递增字段。 好吧,现在当我尝试插
我是新来的。我目前正在 DeVry 在线学习中级 C++ 编程。我们正在使用 C++ Primer Plus 这本书,到目前为止我一直做得很好。我的老师最近向我们扔了一个曲线球。我目前的任务是这样的:
这个问题在这里已经有了答案: What is an undefined reference/unresolved external symbol error and how do I fix it?
我的网站中有一段代码有问题;此错误仅发生在 Internet Explorer 7 中。 我没有在这里发布我所有的 HTML/CSS 标记,而是发布了网站的一个版本 here . 如您所见,我在列中有
如果尝试在 USB 设备上构建 node.js 应用程序时在我的树莓派上使用 npm 时遇到一些问题。 package.json 看起来像这样: { "name" : "node-todo",
在 Python 中,您有 None单例,在某些情况下表现得很奇怪: >>> a = None >>> type(a) >>> isinstance(a,None) Traceback (most
这是我的 build.gradle (Module:app) 文件: apply plugin: 'com.android.application' android { compileSdkV
我是 android 的新手,我的项目刚才编译和运行正常,但在我尝试实现抽屉导航后,它给了我这个错误 FAILURE: Build failed with an exception. What wen
谁能解释一下?我想我正在做一些非常愚蠢的事情,并且急切地等待着启蒙。 我得到这个输出: phpversion() == 7.2.25-1+0~20191128.32+debian8~1.gbp108
我是一名优秀的程序员,十分优秀!