gpt4 book ai didi

Maximum limit on number of threads in python(对python线程数量的最大限制)

转载 作者:bug小助手 更新时间:2023-10-25 17:22:21 25 4
gpt4 key购买 nike



I am using Threading module in python. How to know how many max threads I can have on my system?

我使用的是PYTHON中的线程模块。如何知道我的系统上可以有多少个最大线程?


更多回答

Possible duplicate of Maximum number of threads per process in Linux?

在Linux中,每个进程的最大线程数可能重复吗?

This depends on your system and type of work you are conducting in your threads. You divide your CPU time over different threads if they are small tasks you can run more threads than with large tasks. Please also refer to stackoverflow.com/questions/481970/how-many-threads-is-too-many

这取决于您的系统和您在线程中执行的工作类型。您可以将您的CPU时间分配给不同的线程,如果它们是小任务,您可以运行比大型任务更多的线程。另请参阅stackoverflow.com/questions/481970/how-many-threads-is-too-many

@liliscent Not really, the question ask for a way to get the number from the OS with Python.

@liliscent并非如此,这个问题问的是一种从使用Python的操作系统获取数字的方法。

(although having too many threads isn't good anyway)

(尽管线程太多无论如何都不好)

But still knowing max limit can help write optimum code.

但知道最大限制仍然可以帮助编写最佳代码。

优秀答案推荐


I am using Threading module in python. How to know how many max
threads I can have on my system?



There doesn't seem to be a hard-coded or configurable MAX value that I've ever found, but there is definitely a limit. Run the following program:

据我所知,似乎没有硬编码或可配置的最大值,但肯定有一个限制。运行以下程序:


import threading
import time


def mythread():
time.sleep(1000)

def main():
threads = 0 #thread counter
y = 1000000 #a MILLION of 'em!
for i in range(y):
try:
x = threading.Thread(target=mythread, daemon=True)
threads += 1 #thread counter
x.start() #start each thread
except RuntimeError: #too many throws a RuntimeError
break
print("{} threads created.\n".format(threads))

if __name__ == "__main__":
main()

I suppose I should mention that this is using Python 3.

我想我应该提一下,这是在使用Python3。


The first function, mythread(), is the function which will be executed as a thread. All it does is sleep for 1000 seconds then terminate.

第一个函数my线程()是将作为线程执行的函数。它所做的就是休眠1000秒,然后终止。


The main() function is a for-loop which tries to start one million threads. The daemon property is set to True simply so that we don't have to clean up all the threads manually.

Main()函数是一个for循环,它尝试启动一百万个线程。守护程序属性被简单地设置为True,这样我们就不必手动清理所有线程。


If a thread cannot be created Python throws a RuntimeError. We catch that to break out of the for-loop and display the number of threads which were successfully created.

如果无法创建线程,则Python会抛出一个RounmeError。我们捕捉它以跳出for循环,并显示成功创建的线程数。


Because daemon is set True, all threads terminate when the program ends.

因为守护进程设置为True,所以所有线程都会在程序结束时终止。


If you run it a few times in a row you're likely to see that a different number of threads will be created each time. On the machine from which I'm posting this reply, I had a minimum 18,835 during one run, and a maximum of 18,863 during another run. And the more you fiddle with the code, as in, the more code you add to this in order to experiment or find more information, you'll find the fewer threads can/will be created.

如果连续运行几次,您可能会看到每次都会创建不同数量的线程。在我发布这篇回复的机器上,我在一次运行中最少有18,835人,在另一次运行中最多有18,863人。您对代码摆弄得越多,为了试验或找到更多信息而向其中添加的代码越多,您就会发现可以/将创建的线程就越少。


So, how to apply this to real world.

那么,如何将这一点应用于现实世界呢?


Well, a server may need the ability to start a triple-digit number of threads, but in most other cases you should re-evaluate your game plan if you think you're going to be generating a large number of threads.

服务器可能需要启动三位数线程的能力,但在大多数其他情况下,如果您认为将生成大量线程,则应该重新评估您的游戏计划。


One thing you need to consider if you're using Python: if you're using a standard distribution of Python, your system will only execute one Python thread at a time, including the main thread of your program, so adding more threads to your program or more cores to your system doesn't really get you anything when using the threading module in Python. You can research all of the pedantic details and ultracrepidarian opinions regarding the GIL / Global Interpreter Lock for more info on that.

如果您使用的是Python,需要考虑的一件事是:如果您使用的是标准的Python发行版,您的系统一次只会执行一个Python线程,包括程序的主线程,所以在使用Python线程模块时,向程序添加更多的线程或向系统添加更多的内核并不能真正得到任何结果。你可以研究所有关于GIL/Global Interpreter Lock的迂腐细节和极端爬行动物的观点,以获得更多信息。


What that means is that cpu-bound (computationally-intensive) code doesn't benefit greatly from factoring it into threads.

这意味着受CPU限制(计算密集型)的代码不会从将其分解到线程中获得很大好处。


I/O-bound (waiting for file read/write, network read, or user I/O) code, however, benefits greatly from multithreading! So, start a thread for each network connection to your Python-based server.

然而,I/O受限(等待文件读/写、网络读取或用户I/O)代码从多线程中获益良多!因此,为您的基于Python的服务器的每个网络连接启动一个线程。


Threads can also be great for triggering/throwing/raising signals at set periods, or simply to block out the processing sections of your code more logically.

线程还可以很好地在设定的时间段触发/抛出/引发信号,或者只是更符合逻辑地阻止代码的处理部分。



On linux it's limited globally, based on the size of your RAM, to get the current number of thread limit ask via:

在Linux上,根据您的RAM大小,获取当前线程限制的数量是全局受限的,请通过:


cat /proc/sys/kernel/threads-max

For me it's 61821 for 8GB RAM.
Of course if you run out of memory sooner then you won't be able to create that many.

对我来说,8 GB内存的价格是61821。当然,如果您很快耗尽内存,那么您将无法创建那么多内存。



I could see at Max 4096 threads are creating when I run the following code:

当我运行以下代码时,我可以看到最多创建了4096个线程:


import threading
import time


class NumberPrinter(threading.Thread):
def __init__(self, args):
self.counter = args[0]

def run(self) -> None:
time.sleep(5)
print(f"Thread Name: {threading.current_thread().name}, Counter: {self.counter}")


if __name__ == '__main__':
for i in range(10000000):
number_printer = NumberPrinter(args=(i+1,))
number_printer.start()

更多回答

So, are you saying Python's multiprocessing would be better than threading for that first use-case? docs.python.org/3/library/multiprocessing.html

那么,您是说在第一个用例中,Python的多处理会比线程更好吗?Docs.python.org/3/library/multiprocessing.html

Good point, and in retrospect I should have mentioned that. Yes, the multiprocessing module would be what you would choose for computer-intensive tasks.

说得好,回想起来,我应该提到这一点。是的,对于计算机密集型任务,多处理模块将是您的选择。

25 4 0
Copyright 2021 - 2024 cfsdn All Rights Reserved 蜀ICP备2022000587号
广告合作:1813099741@qq.com 6ren.com