linux - 使用管道重新格式化 Git 克隆进度

Question

我正在使用脚本来部署 git 代码。我有一个用于格式化进度的函数:

log ()
{
    d=$(date "+%Y%m%d %H%M%S.%4N")
    echo "[$d] $1"
}

我试图了解有关管道 git clone 的其他文章/帖子，但只看到解决管道到文件的帖子。

这是我的:

git clone --progress $api_repository 2>&1 | log

显然不起作用，因为它没有将值作为参数传递给日志(输出仅显示一个日期 - 而且只有一个)。

感谢任何帮助。谢谢。

编辑(示例 git 克隆输出):

Cloning into 'apicode'...
remote: Counting objects: 515, done.
remote: Compressing objects: 100% (414/414), done.
remote: Total 515 (delta 275), reused 234 (delta 76)
Receiving objects: 100% (515/515), 70.82 KiB, done.
Resolving deltas: 100% (275/275), done.

我希望实现的是获得这样的输出:

[20170511 161944.1041] Cloning into 'apicode'...
[20170511 161944.1241] remote: Counting objects: 515, done.
[20170511 161944.1341] remote: Compressing objects: 100% (414/414), done.
[20170511 161944.1441] remote: Total 515 (delta 275), reused 234 (delta 76)
[20170511 161944.1541] Receiving objects: 100% (515/515), 70.82 KiB, done.
[20170511 161944.1641] Resolving deltas: 100% (275/275), done.

(基本上是给每一行加一个时间戳)

Answer 1

Here is what I have:

git clone --progress $api_repository 2>&1 | log

Clearly not working as it does not pass the value as a parameter to log (output only shows a date - and only one).

首先，这是行不通的，因为 log 函数不是从标准输入中获取字符串，而是它的命令行参数 $1 — 在这种情况下它是空的.所以它只在没有输入的情况下运行一次，因此只显示日期。

将 log 应用于另一个程序输出的每一行的通常方法是用 xargs 包装它。为了使 xargs 能够调用定义为 shell 函数的内容，您需要首先从中创建一个脚本文件(比如 ./log.sh):

#!/bin/bash
d=$(date "+%Y%m%d %H%M%S.%4N")
echo "[$d] $1"

然后，您可以这样调用 xargs:

git clone --progress https://github.com/torvalds/linux.git 2>&1 |
    xargs -n1 -d'\n' ./log.sh

......唉，这也没有按预期工作。

---

这是您要使用的 git clone 选项:

--progress
       Progress status is reported on the standard error stream by default
       when it is attached to a terminal, unless -q is specified. This flag
       forces progress status even if the standard error stream is not
       directed to a terminal.

当标准错误未附加到终端时未报告进度状态的原因，也是因为解决问题的简单方法失败了，是这样的:

0000000: 436c 6f6e 696e 6720 696e 746f 2027 6c69  Cloning into 'li
0000010: 6e75 7827 2e2e 2e0a 7265 6d6f 7465 3a20  nux'....remote: 
0000020: 436f 756e 7469 6e67 206f 626a 6563 7473  Counting objects
0000030: 3a20 3533 3739 3436 3020 2020 2020 2020  : 5379460       
0000040: 2020 2020 0d72 656d 6f74 653a 2043 6f75      .remote: Cou
0000050: 6e74 696e 6720 6f62 6a65 6374 733a 2035  nting objects: 5
0000060: 3338 3838 3534 2c20 646f 6e65 2e20 2020  388854, done.   
0000070: 2020 2020 200a 7265 6d6f 7465 3a20 436f       .remote: Co
0000080: 6d70 7265 7373 696e 6720 6f62 6a65 6374  mpressing object
0000090: 733a 2020 2030 2520 2831 2f37 3931 3429  s:   0% (1/7914)
00000a0: 2020 2020 2020 2020 2020 200d 7265 6d6f             .remo
00000b0: 7465 3a20 436f 6d70 7265 7373 696e 6720  te: Compressing 
00000c0: 6f62 6a65 6374 733a 2020 2031 2520 2838  objects:   1% (8
00000d0: 302f 3739 3134 2920 2020 2020 2020 2020  0/7914)         
00000e0: 2020 0d72 656d 6f74 653a 2043 6f6d 7072    .remote: Compr
00000f0: 6573 7369 6e67 206f 626a 6563 7473 3a20  essing objects: 
20000100: 2020 3225 2028 3135 392f 3739 3134 2920    2% (159/7914)

这是管道另一端的程序所看到的。 git 使用 carriage return 字符(0d 或 \r)更新进度输出中的计数器。当由终端解释时，回车将光标移动到一行的开头，有效地覆盖 log 试图插入的前缀。但更糟糕的是，如果您要重定向到一个文件，您的日志将包含所有中间内容，这将占用比最终输出多数十至数千倍的空间。

所以这实际上意味着如果您只对最终输出感兴趣，则必须过滤掉所有中间进度字符串:

git clone --progress https://github.com/torvalds/linux.git 2>&1 |
    sed 's:.*\r::' |
    xargs -n1 -d'\n' ./log.sh

这将在管道传输到 log 之前删除以回车符结尾的子字符串。这是输出:

[20170512 002119.3233] Cloning into 'linux'...
[20170512 002119.3264] remote: Counting objects: 277, done.        
[20170512 002119.3289] remote: Total 277 (delta 0), reused 0 (delta 0), pack-reused 274        
[20170512 002119.3310] Receiving objects: 100% (277/277), 56.33 KiB | 0 bytes/s, done.
[20170512 002119.3331] Resolving deltas: 100% (155/155), done.

对于更具交互性(和详细)的版本，请将 \r 替换为 \n。以下使用 tr :

git clone --progress https://github.com/torvalds/linux.git 2>&1 |
    tr '\r' '\n' |
    xargs -n1 -d'\n' ./log.sh

结果如下:

[20170511 234551.3756] Cloning into 'linux'...
[20170511 234551.3806] remote: Counting objects: 5379460           
[20170511 234551.3854] remote: Counting objects: 5388854, done.        
[20170511 234551.3898] remote: Compressing objects:   0% (1/7914)           
[20170511 234551.3941] remote: Compressing objects:   1% (80/7914)           
[20170511 234551.3979] remote: Compressing objects:   2% (159/7914)           
[20170511 234551.4019] remote: Compressing objects:   3% (238/7914)           
[20170511 234551.4066] remote: Compressing objects:   4% (317/7914)           
[20170511 234551.4108] remote: Compressing objects:   5% (396/7914)           
[20170511 234551.4132] remote: Compressing objects:   6% (475/7914)           
[20170511 234551.4155] remote: Compressing objects:   7% (554/7914)           
[20170511 234551.4183] remote: Compressing objects:   8% (634/7914)           
[20170511 234551.4208] remote: Compressing objects:   9% (713/7914)           
[20170511 234551.4231] remote: Compressing objects:  10% (792/7914)           
[20170511 234551.4254] remote: Compressing objects:  11% (871/7914)           
[20170511 234551.4277] remote: Compressing objects:  12% (950/7914)           
[20170511 234551.4300] remote: Compressing objects:  13% (1029/7914)           
[20170511 234551.4323] remote: Compressing objects:  14% (1108/7914)           
[20170511 234551.4346] remote: Compressing objects:  15% (1188/7914)           
[20170511 234551.4381] remote: Compressing objects:  16% (1267/7914)           
[20170511 234551.4408] remote: Compressing objects:  17% (1346/7914)           
[20170511 234551.4432] remote: Compressing objects:  18% (1425/7914)           
[20170511 234551.4456] remote: Compressing objects:  19% (1504/7914)           
[20170511 234551.4480] remote: Compressing objects:  20% (1583/7914)