multithreading - 多个wget -r同时访问一个站点？

Question

任何带有选项的命令/wget？

对于多线程下载，是否需要递归同时进行网站？

Answer 1

我找到了一个不错的解决方案。

在http://www.linuxquestions.org/questions/linux-networking-3/wget-multi-threaded-downloading-457375/阅读原件

wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url] &

copied as many times as you deem fitting to have as much processes downloading. This isn't as elegant as a properly multithreaded app, but it will get the job done with only a slight amount of over head. the key here being the "-N" switch. This means transfer the file only if it is newer than what's on the disk. This will (mostly) prevent each process from downloading the same file a different process already downloaded, but skip the file and download what some other process hasn't downloaded. It uses the time stamp as a means of doing this, hence the slight overhead.

It works great for me and saves a lot of time. Don't have too many processes as this may saturate the web site's connection and tick off the owner. Keep it around a max of 4 or so. However, the number is only limited by CPU and network bandwidth on both ends.