python - 并行化 pandas pyodbc SQL 数据库调用

Question

我目前正在通过 pandas.io.sql.read_sql() 命令将数据查询到数据框中。我想并行化类似于这些人所提倡的调用:( Embarrassingly parallel database calls with Python (PyData Paris 2015 ) )

类似(非常笼统):

pools = [ThreadedConnectionPool(1,20,dsn=d) for d in dsns]
connections = [pool.getconn() for pool in pools]
parallel_connection = ParallelConnection(connections)
pandas_cursor = parallel_connection.cursor()
pandas_cursor.execute(my_query)

这样的事情可能吗？

Answer 1

是的，这应该有效，但需要注意的是您需要在您站点的那个演讲中更改 parallel_connection.py。在该代码中有一个 fetchall 函数，它并行执行每个游标，然后组合结果。这是您要更改的核心内容:

旧代码:

def fetchall(self):
    results = [None] * len(self.cursors)
    def do_work(index, cursor):
        results[index] = cursor.fetchall()
    self._do_parallel(do_work)
    return list(chain(*[rs for rs in results]))

新代码:

def fetchall(self):
    results = [None] * len(self.sql_connections)
    def do_work(index, sql_connection):
        sql, conn = sql_connection  #  Store tuple of sql/conn instead of cursor
        results[index] = pd.read_sql(sql, conn)
    self._do_parallel(do_work)
    return pd.DataFrame().append([rs for rs in results])

repo :https://github.com/godatadriven/ParallelConnection