bash - 并行运行两个连续的 bash 脚本 4 次

Question

我有一个配置文件列表:

cfg1.cfg
cfg2.cfg
cfg3.cfg
cfg4.cfg
cfg5.cfg
cfg6.cfg
cfg7.cfg
...

作为两个脚本的输入:

script1.sh
script2.sh

我按如下顺序运行:

script1.sh cfgX.cfg && script2.sh cfgX.cfg

其中 X=1, 2, 3, ...

这些脚本没有并行化，需要很长时间才能运行。我怎样才能并行启动它们，比如说同时启动 4 个，这样我就不会杀死运行它们的服务器？

对于一个脚本，我尝试了一种类似于以下的蛮力方法:

export COUNTER_LIMIT=4

export COUNTER=1

for each in $(ls *.cfg)
do

INSTRUCTION="./script1.sh $each "

if (($COUNTER >= $COUNTER_LIMIT)) ; 
then
$INSTRUCTION   &&
export COUNTER=$(($COUNTER-$COUNTER_LIMIT));
echo
sleep 600s
else
$INSTRUCTION  &
sleep 5s
fi

echo $COUNTER
export COUNTER=$(($COUNTER+1));

done

( sleep 是因为某些原因脚本不能同时启动...)

所以，我可以这样做吗？

script1.sh cfgX.cfg && script2.sh cfgX.cfg

不阻止暴力并行化吗？

我也接受更好更简单的方法；)

干杯豪尔赫

更新

我应该提到的是，配置文件不一定按顺序命名，可以有任何名称，我只是将它们设为这样，以使示例尽可能简单。

Answer 1

parallel --jobs 4  \
         --load 50% \
         --bar       \
         --eta "( echo 1st-for-{}; echo 2nd-for-{} )" < aListOfAdHocArguments.txt
0% 0:5=0s
1st-for-Abraca
2nd-for-Abraca
20% 1:4=0s                                                                                                                                                                                                 
1st-for-Dabra
2nd-for-Dabra
40% 2:3=0s                                                                                                                                                                                                 
1st-for-Hergot
2nd-for-Hergot
60% 3:2=0s                                                                                                                                                                                                 
1st-for-Fagot
2nd-for-Fagot
80% 4:1=0s                                                                                                                                                                                                 

100% 5:0=0s

Q : How can I launch them in parallel, let's say 4 at the time, so I do not kill the server where I run them?

GNU 的一项可爱任务 parallel 。

首先让我们检查一下 localhost 生态系统(exosystems，执行 parallel -jobs 超过 ssh -连接的远程主机可能，但超出了本文的范围):

parallel --number-of-cpus
parallel --number-of-cores
parallel --show-limits

--jobs 4 之外的更多配置细节，可能是 --memfree或 --noswap , --load <max-load>或 --keep-order和 --results <aFile>或 --output-as-files :

man parallel

---

parallel --jobs 4 \
         --bar     \
         --eta "( script1.sh cfg{}.cfg; script2.sh cfg{}.cfg )" ::: {1..123}

这里，
由一对串联 echo 模拟-s 用于向下计数的索引，因此进度条是不可见的，并且 E 估计 -Time-of-Arival --eta迹象几乎是即时的......:

parallel --jobs 4  \
         --load 50% \
         --bar       \
         --eta "( echo 1st-for-cfg-{}; echo 2nd-for-cfg-{} )" ::: {10..0}
0% 0:11=0s 7                                                                                                                                                                                               
1st-for-cfg-10
2nd-for-cfg-10
9% 1:10=0s 6                                                                                                                                                                                               
1st-for-cfg-9
2nd-for-cfg-9
18% 2:9=0s 5                                                                                                                                                                                               
1st-for-cfg-8
2nd-for-cfg-8
27% 3:8=0s 4                                                                                                                                                                                               
1st-for-cfg-7
2nd-for-cfg-7
36% 4:7=0s 3                                                                                                                                                                                               
1st-for-cfg-6
2nd-for-cfg-6
45% 5:6=0s 2                                                                                                                                                                                               
1st-for-cfg-5
2nd-for-cfg-5
54% 6:5=0s 1                                                                                                                                                                                               
1st-for-cfg-4
2nd-for-cfg-4
63% 7:4=0s 0                                                                                                                                                                                               
1st-for-cfg-3
2nd-for-cfg-3
72% 8:3=0s 0                                                                                                                                                                                               
1st-for-cfg-2
2nd-for-cfg-2
81% 9:2=0s 0                                                                                                                                                                                               
1st-for-cfg-1
2nd-for-cfg-1
90% 10:1=0s 0                                                                                                                                                                                              
1st-for-cfg-0
2nd-for-cfg-0

更新

你添加了:

I should have mentioned that the config files are not necessarily sequentially named and can have any name, I just made them like this to make the example as simple as possible.

< list_of_arguments 解决了这个事后更改的问题定义:

parallel [options] [command [arguments]] < list_of_arguments