gpt4 book ai didi

python - 如何使用带有线程的MYSQL select查询而不重复select查询结果?

转载 作者:行者123 更新时间:2023-11-29 15:17:21 25 4
gpt4 key购买 nike

简短上下文:我正在使用 mysql 表来选择一个值,通过使用 API+值我获取结果并将结果保存到同一个表中。

问题:如何同时处理多行?每当我使用线程启动该函数时,它都会为每个线程选择相同的值(即光标为每个线程返回相同的值)。我需要每个线程处理不同的值。这样我会减少一些时间。

我的程序是

import requests
import os
import json
import pymysql
import threading

conn = pymysql.connect(host='localhost', user=USER, passwd=PASSWORD, db='sampledb',charset='utf8mb4',autocommit=True)

url = "http://www.someapi.com/somelink/"

cur = conn.cursor()

def main():
cur.execute("select asset_id from getprocessid where status =%s LIMIT 1",("uploaded",))
idofassets = cur.fetchone()[0]
req = requests.Session()
resp = req.get(url+str(idofassets))
resp_json = json.loads(resp.text)
actual = resp_json['getResponse']
cur.execute("update getprocessid set class = %s ,status =%s where asset_id = %s",(str(actual),"completed",str(idofasset),))

while True:

# For threading purpose i added

thread1 = threading.Thread(target=main)
thread2 = threading.Thread(target=main)
thread3 = threading.Thread(target=main)

thread1.start()
thread2.start()
thread3.start()

thread1.join()
thread2.join()
thread3.join()

最佳答案

您的问题似乎分为两个主要不同的任务:

1 - 从 getprocessid MySQL 表中获取结果

2 - 处理结果并更新同一个表(但不同的字段)

因此,优化代码的一种方法是让一个线程(可能是主线程)执行步骤 1,然后将步骤 2 中的问题分配给 3 个线程:

import requests
import os
import json
import pymysql
import threading
#you can create these dynamically if you
#want more (or less) threads
batches = [[], [], []]

conn = pymysql.connect(host='localhost', user=USER,
passwd=PASSWORD,
db='sampledb',charset='utf8mb4',autocommit=True)

url = "http://www.someapi.com/somelink/"

cur = conn.cursor()

def fetch_and_split():
cur.execute("select asset_id from getprocessid
where status =%s LIMIT 1",("uploaded",))
results = cur.fetchall()
count = 0
#this populates the lists to be processed with the ids
while count < size(results):
cur_batch = batches[size(batches) % count ]
cur_batch.append(results[count][0])
count++

def process_and_update(batch):
#each thread receives its own list
for idofassets in batch:
req = requests.Session()
resp = req.get(url+str(idofassets))
resp_json = json.loads(resp.text)
actual = resp_json['getResponse']
cur.execute("update getprocessid set class = %s
,status =%s where asset_id = %s",
(str(actual),"completed",str(idofasset),))


while True:

# For threading purpose i added
# The main thread splits the results
fetch_and_split()
# The other threads process the
# results and update the values
thread1 = threading.Thread(target=process_and_update, args=(batches[0],))
thread2 = threading.Thread(target=process_and_update, args=(batches[1],))
thread3 = threading.Thread(target=process_and_update, args=(batches[2],))

thread1.start()
thread2.start()
thread3.start()

thread1.join()
thread2.join()
thread3.join()

关于python - 如何使用带有线程的MYSQL select查询而不重复select查询结果?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/59612671/

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com