- html - 出于某种原因,IE8 对我的 Sass 文件中继承的 html5 CSS 不友好?
- JMeter 在响应断言中使用 span 标签的问题
- html - 在 :hover and :active? 上具有不同效果的 CSS 动画
- html - 相对于居中的 html 内容固定的 CSS 重复背景?
请不要因为长篇大论而气馁。我尽量提供尽可能多的数据,我真的需要帮助解决这个问题:S。如果有新的提示或想法,我会每天更新
问题:
我尝试在并行进程的帮助下在两核机器上并行运行 Python 代码(以避免 GIL),但存在代码显着减慢的问题。例如,在单核机器上运行每个工作负载需要 600 秒,但在双核机器上运行需要 1600 秒(每个工作负载 800 秒)。
我已经尝试过的:
result = [None]*psutil.cpu_count()
e = futures.ProcessPoolExecutor( max_workers=psutil.cpu_count() )
for i in range(psutil.cpu_count()):
result[i] = e.submit(process_function, ...)
from math import floor
from math import ceil
import numpy
import MySQLdb
import time
db = MySQLdb.connect(...)
cursor = db.cursor()
query = "SELECT ...."
cursor.execute(query)
[...] #save db results into the variable db_matrix (30 columns, 5.000 rows)
[...] #save db results into the variable bp_vector (3 columns, 500 rows)
[...] #save db results into the variable option_vector( 3 columns, 4000 rows)
cursor.close()
db.close()
counter = 0
for i in range(4000):
for j in range(500):
helper[:] = (1-bp_vector[j,0]-bp_vector[j,1]-bp_vector[j,2])*db_matrix[:,0]
+ db_matrix[:,option_vector[i,0]] * bp_vector[j,0]
+ db_matrix[:,option_vector[i,1]] * bp_vector[j,1]
+ db_matrix[:,option_vector[i,2]] * bp_vector[j,2]
result[counter,0] = (helper < -7.55).sum()
counter = counter + 1
return result
1-core machine, started 1 process:
time: 225sec , CPU utilization: ~100%
1-core machine, started 2 process:
time: 557sec , CPU utilization: ~100%
1-core machine, started 1 process, limited max. CPU-utilization to 50%:
time: 488sec , CPU utilization: ~50%
2-core machine, started 2 process:
time: 665sec , CPU-1 utilization: ~100% , CPU-2 utilization: ~100%
the process did not jumped between the cores, each used 1 core
(at least htop displayed these results with the “Process” column)
2-core machine, started 1 process:
time: 222sec , CPU-1 utilization: ~100% (0%) , CPU-2 utilization: ~0% (100%)
however, the process jumped sometimes between the cores
2-core machine, started 1 process, limited max. CPU-utilization to 50%:
time: 493sec , CPU-1 utilization: ~50% (0%) , CPU-2 utilization: ~0% (100%)
however, the process jumped extremely often between the cores
python -m cProfile -s cumtime fun_name.py
623158 function calls (622735 primitive calls) in 229.286 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.371 0.371 229.287 229.287 20_with_multiprocessing.py:1(<module>)
3 0.000 0.000 225.082 75.027 threading.py:309(wait)
1 0.000 0.000 225.082 225.082 _base.py:378(result)
25 225.082 9.003 225.082 9.003 {method 'acquire' of 'thread.lock' objects}
1 0.598 0.598 3.081 3.081 get_BP_Verteilung_Vektoren.py:1(get_BP_Verteilung_Vektoren)
3 0.000 0.000 2.877 0.959 cursors.py:164(execute)
3 0.000 0.000 2.877 0.959 cursors.py:353(_query)
3 0.000 0.000 1.958 0.653 cursors.py:315(_do_query)
3 0.000 0.000 1.943 0.648 cursors.py:142(_do_get_result)
3 0.000 0.000 1.943 0.648 cursors.py:351(_get_result)
3 1.943 0.648 1.943 0.648 {method 'store_result' of '_mysql.connection' objects}
3 0.001 0.000 0.919 0.306 cursors.py:358(_post_get_result)
3 0.000 0.000 0.917 0.306 cursors.py:324(_fetch_row)
3 0.917 0.306 0.917 0.306 {built-in method fetch_row}
591314 0.161 0.000 0.161 0.000 {range}
626052 function calls (625616 primitive calls) in 578.086 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.310 0.310 578.087 578.087 20_with_multiprocessing.py:1(<module>)
30 574.310 19.144 574.310 19.144 {method 'acquire' of 'thread.lock' objects}
2 0.000 0.000 574.310 287.155 _base.py:378(result)
3 0.000 0.000 574.310 191.437 threading.py:309(wait)
1 0.544 0.544 2.854 2.854 get_BP_Verteilung_Vektoren.py:1(get_BP_Verteilung_Vektoren)
3 0.000 0.000 2.563 0.854 cursors.py:164(execute)
3 0.000 0.000 2.563 0.854 cursors.py:353(_query)
3 0.000 0.000 1.715 0.572 cursors.py:315(_do_query)
3 0.000 0.000 1.701 0.567 cursors.py:142(_do_get_result)
3 0.000 0.000 1.701 0.567 cursors.py:351(_get_result)
3 1.701 0.567 1.701 0.567 {method 'store_result' of '_mysql.connection' objects}
3 0.001 0.000 0.848 0.283 cursors.py:358(_post_get_result)
3 0.000 0.000 0.847 0.282 cursors.py:324(_fetch_row)
3 0.847 0.282 0.847 0.282 {built-in method fetch_row}
591343 0.152 0.000 0.152 0.000 {range}
623164 function calls (622741 primitive calls) in 235.954 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.246 0.246 235.955 235.955 20_with_multiprocessing.py:1(<module>)
3 0.000 0.000 232.003 77.334 threading.py:309(wait)
25 232.003 9.280 232.003 9.280 {method 'acquire' of 'thread.lock' objects}
1 0.000 0.000 232.003 232.003 _base.py:378(result)
1 0.593 0.593 3.104 3.104 get_BP_Verteilung_Vektoren.py:1(get_BP_Verteilung_Vektoren)
3 0.000 0.000 2.774 0.925 cursors.py:164(execute)
3 0.000 0.000 2.774 0.925 cursors.py:353(_query)
3 0.000 0.000 1.981 0.660 cursors.py:315(_do_query)
3 0.000 0.000 1.970 0.657 cursors.py:142(_do_get_result)
3 0.000 0.000 1.969 0.656 cursors.py:351(_get_result)
3 1.969 0.656 1.969 0.656 {method 'store_result' of '_mysql.connection' objects}
3 0.001 0.000 0.794 0.265 cursors.py:358(_post_get_result)
3 0.000 0.000 0.792 0.264 cursors.py:324(_fetch_row)
3 0.792 0.264 0.792 0.264 {built-in method fetch_row}
591314 0.144 0.000 0.144 0.000 {range}
626072 function calls (625636 primitive calls) in 682.460 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.334 0.334 682.461 682.461 20_with_multiprocessing.py:1(<module>)
4 0.000 0.000 678.231 169.558 threading.py:309(wait)
33 678.230 20.552 678.230 20.552 {method 'acquire' of 'thread.lock' objects}
2 0.000 0.000 678.230 339.115 _base.py:378(result)
1 0.527 0.527 2.974 2.974 get_BP_Verteilung_Vektoren.py:1(get_BP_Verteilung_Vektoren)
3 0.000 0.000 2.723 0.908 cursors.py:164(execute)
3 0.000 0.000 2.723 0.908 cursors.py:353(_query)
3 0.000 0.000 1.749 0.583 cursors.py:315(_do_query)
3 0.000 0.000 1.736 0.579 cursors.py:142(_do_get_result)
3 0.000 0.000 1.736 0.579 cursors.py:351(_get_result)
3 1.736 0.579 1.736 0.579 {method 'store_result' of '_mysql.connection' objects}
3 0.001 0.000 0.975 0.325 cursors.py:358(_post_get_result)
3 0.000 0.000 0.973 0.324 cursors.py:324(_fetch_row)
3 0.973 0.324 0.973 0.324 {built-in method fetch_row}
5 0.093 0.019 0.304 0.061 __init__.py:1(<module>)
1 0.017 0.017 0.275 0.275 __init__.py:106(<module>)
1 0.005 0.005 0.198 0.198 add_newdocs.py:10(<module>)
591343 0.148 0.000 0.148 0.000 {range}
Database access took 2.53 seconds
Matrix manipulation took 236.71 seconds
1842384 function calls (1841974 primitive calls) in 241.114 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 219.036 219.036 241.115 241.115 20_with_multiprocessing.py:1(<module>)
406000 0.873 0.000 18.097 0.000 {method 'sum' of 'numpy.ndarray' objects}
406000 0.502 0.000 17.224 0.000 _methods.py:31(_sum)
406001 16.722 0.000 16.722 0.000 {method 'reduce' of 'numpy.ufunc' objects}
1 0.587 0.587 3.222 3.222 get_BP_Verteilung_Vektoren.py:1(get_BP_Verteilung_Vektoren)
3 0.000 0.000 2.964 0.988 cursors.py:164(execute)
3 0.000 0.000 2.964 0.988 cursors.py:353(_query)
3 0.000 0.000 1.958 0.653 cursors.py:315(_do_query)
3 0.000 0.000 1.944 0.648 cursors.py:142(_do_get_result)
3 0.000 0.000 1.944 0.648 cursors.py:351(_get_result)
3 1.944 0.648 1.944 0.648 {method 'store_result' of '_mysql.connection' objects}
3 0.001 0.000 1.006 0.335 cursors.py:358(_post_get_result)
3 0.000 0.000 1.005 0.335 cursors.py:324(_fetch_row)
3 1.005 0.335 1.005 0.335 {built-in method fetch_row}
591285 0.158 0.000 0.158 0.000 {range}
Database access took 2.32 seconds
Matrix manipulation took 242.45 seconds
1842390 function calls (1841980 primitive calls) in 246.535 seconds
Ordered by: cumulative time
ncalls tottime percall cumtime percall filename:lineno(function)
1 224.705 224.705 246.536 246.536 20_with_multiprocessing.py:1(<module>)
406000 0.911 0.000 17.971 0.000 {method 'sum' of 'numpy.ndarray' objects}
406000 0.526 0.000 17.060 0.000 _methods.py:31(_sum)
406001 16.534 0.000 16.534 0.000 {method 'reduce' of 'numpy.ufunc' objects}
1 0.617 0.617 3.113 3.113 get_BP_Verteilung_Vektoren.py:1(get_BP_Verteilung_Vektoren)
3 0.000 0.000 2.789 0.930 cursors.py:164(execute)
3 0.000 0.000 2.789 0.930 cursors.py:353(_query)
3 0.000 0.000 1.938 0.646 cursors.py:315(_do_query)
3 0.000 0.000 1.920 0.640 cursors.py:142(_do_get_result)
3 0.000 0.000 1.920 0.640 cursors.py:351(_get_result)
3 1.920 0.640 1.920 0.640 {method 'store_result' of '_mysql.connection' objects}
3 0.001 0.000 0.851 0.284 cursors.py:358(_post_get_result)
3 0.000 0.000 0.849 0.283 cursors.py:324(_fetch_row)
3 0.849 0.283 0.849 0.283 {built-in method fetch_row}
591285 0.160 0.000 0.160 0.000 {range}
最佳答案
您的程序似乎大部分时间都花在获取锁上。这似乎表明,在您的情况下,多处理弊大于利。
删除所有多处理的东西并开始测量没有它需要多长时间。例如。像这样。
from math import floor
from math import ceil
import numpy
import MySQLdb
import time
start = time.clock()
db = MySQLdb.connect(...)
cursor = db.cursor()
query = "SELECT ...."
cursor.execute(query)
stop = time.clock()
print "Database access took {:.2f} seconds".format(stop - start)
start = time.clock()
[...] #save db results into the variable db_matrix (30 columns, 5.000 rows)
[...] #save db results into the variable bp_vector (3 columns, 500 rows)
[...] #save db results into the variable option_vector( 3 columns, 4000 rows)
stop = time.clock()
print "Creating matrices took {:.2f} seconds".format(stop - start)
cursor.close()
db.close()
counter = 0
start = time.clock()
for i in range(4000):
for j in range(500):
helper[:] = (1-bp_vector[j,0]-bp_vector[j,1]-bp_vector[j,2])*db_matrix[:,0]
+ db_matrix[:,option_vector[i,0]] * bp_vector[j,0]
+ db_matrix[:,option_vector[i,1]] * bp_vector[j,1]
+ db_matrix[:,option_vector[i,2]] * bp_vector[j,2]
result[counter,0] = (helper < -7.55).sum()
counter = counter + 1
stop = time.clock()
print "Matrix manipulation took {:.2f} seconds".format(stop - start)
multiprocessing
正如你现在所做的那样,对你的表现非常不利。在双核机器上,具有多处理功能的程序比没有它的程序花费的时间要长得多!
helper
的计算, 由 4 个部分组成。在单独的过程中执行每个部分,并在最后将结果相加。这确实会产生一些开销;每个进程都必须从数据库中检索所有数据,并且必须将部分结果传输回主进程(也可能通过数据库?)。
pypy
而不是
Cpython
.它可以明显更快。
关于python - Python 中的多处理 : Numpy + Vector Summation -> Huge Slowdown,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/36383962/
请不要因为长篇大论而气馁。我尽量提供尽可能多的数据,我真的需要帮助解决这个问题:S。如果有新的提示或想法,我会每天更新 问题: 我尝试在并行进程的帮助下在两核机器上并行运行 Python 代码(以避免
看到this question之后关于在 Pandas 中复制类似 SQL 选择语句的行为,我添加了 this answer显示可以缩短 the accepted answer 中给出的冗长语法的两种
我已经开发了一个应用程序,该应用程序经常使用扩展和写入来读取和写入网络设备。手术通常需要大约7分钟才能完成。。然而,当我同时在两个不同的设备上执行这个应用程序的两个单独的实例时,完成时间延长到15分钟
当我启用 data-search-pseudo-elements 时,它会导致网格内带有图标的行渲染性能降低 100 倍以上。奇怪的是,网格虚拟了行,所以实际上只有 100 行可见。如果没有此设置,渲
我是一名优秀的程序员,十分优秀!