bash:jq 选择中的日期比较和结果导向

Question

我正在使用以下命令从 Debian 上的 bash 文本日志文件中选择一些数据:

cat /mnt/WD1003/logs/sn1.log | grep 1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE | \
grep -E "GET_AUDIT" | jq -R '. | split("\t") | (.[4] | fromjson) as $body |
{SatelliteID: $body."Satellite ID", ($body."Piece ID"): {(.[0]): .[3]}}' | \
jq -s 'reduce .[] as $item ({}; . * $item)'

文本日志文件的示例“原始”输出:

2022-07-26T15:03:10.670Z    INFO    piecestore  download started    {"Process": "storagenode", "Piece ID": "DZ4HPUJE7IFLPABM47L3B5MXFIV3L5P2IBM32CXUJBYUOMJBNJBQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2022-07-26T15:03:10.893Z    INFO    piecestore  downloaded  {"Process": "storagenode", "Piece ID": "DZ4HPUJE7IFLPABM47L3B5MXFIV3L5P2IBM32CXUJBYUOMJBNJBQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2022-07-26T15:47:12.285Z    INFO    piecestore  download started    {"Process": "storagenode", "Piece ID": "CDTQRMUZITFKKCKWTUHGNCVWE2LYZZEPVELC6ADPMREMD5SURZVQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2022-07-26T15:47:12.493Z    INFO    piecestore  downloaded  {"Process": "storagenode", "Piece ID": "CDTQRMUZITFKKCKWTUHGNCVWE2LYZZEPVELC6ADPMREMD5SURZVQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}

当前结果如下:

{
  "SatelliteID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE",
  "NG6KAUMU7TP22DNGROKBU2MRRNV675QYEOJC3X2BXH4OCML6BPNQ": {
    "2022-06-28T21:24:39.646Z": "download started",
    "2022-06-28T21:24:40.002Z": "downloaded"
  },
  "IADTQX62PCZQEJRRYPCKNWX3QSPG7A3U53IBWPQRSX6ZMH6I45UQ": {
    "2022-06-28T21:32:40.597Z": "download started",
    "2022-06-28T21:32:40.893Z": "downloaded",
    "2022-07-09T20:00:10.698Z": "download started",
    "2022-07-09T24:00:10.995Z": "downloaded"
  },
  "MZEPH4JSGSAJZ72QQV4YOYYVGLER7KOQPBUB2VEANL4MPNSZDBTA": {
    "2022-06-28T21:58:56.184Z": "download started",
    "2022-06-28T22:01:26.454Z": "downloaded"
  },
  "GFATHGO2WFBZNAOQJKXYNHTFKH2T5T4OXK3BEL7U62FNK5ZRR6OQ": {
    "2022-06-28T22:08:49.765Z": "download started",
    "2022-06-28T22:08:50.089Z": "downloaded"
  },
...
}

如果“已开始”和“已下载”(或“下载失败”或“下载已取消”)之间的差异大于 3 分钟，我只需要获得上述 jq 查询的结果。如果没有，则结果为空。

所以目标结果应该是这样的:

{
  "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE": "2",
  "anotherSatelliteID": "1"
}

其中数字表示每个 satelliteID 的两个时间戳之间“时间滞后”大于 3 分钟的计数。

有几颗卫星，因此在上面的示例结果中，我们有 2 颗卫星有问题，分别有 5 个 2 个时滞警报。

补充说明:该命令也应在 MacOS 上运行。

请帮助建议我该怎么做。

更新 #1:

我从多次下载中找到了另一个示例。我不希望第二次下载可以在第一次下载完成之前开始 - 但在这种情况下，应该跳过 PieceID。

$ cat /mnt/WD1003/logs/sn1.log | grep CDTQRMUZITFKKCKWTUHGNCVWE2LYZZEPVELC6ADPMREMD5SURZVQ
2022-07-24T02:37:47.570Z    INFO    piecestore  download started    {"Process": "storagenode", "Piece ID": "CDTQRMUZITFKKCKWTUHGNCVWE2LYZZEPVELC6ADPMREMD5SURZVQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2022-07-24T02:37:47.815Z    INFO    piecestore  downloaded  {"Process": "storagenode", "Piece ID": "CDTQRMUZITFKKCKWTUHGNCVWE2LYZZEPVELC6ADPMREMD5SURZVQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2022-07-26T15:47:12.285Z    INFO    piecestore  download started    {"Process": "storagenode", "Piece ID": "CDTQRMUZITFKKCKWTUHGNCVWE2LYZZEPVELC6ADPMREMD5SURZVQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}
2022-07-26T15:47:12.493Z    INFO    piecestore  downloaded  {"Process": "storagenode", "Piece ID": "CDTQRMUZITFKKCKWTUHGNCVWE2LYZZEPVELC6ADPMREMD5SURZVQ", "Satellite ID": "1wFTAgs9DP5RSnCqKV1eLf6N9wtk4EAtmN5DpSxcs8EjT69tGE", "Action": "GET_AUDIT"}

Answer 1

假设 PieceID 是唯一的，这里有一个 jq 解决方案:

jq -Rn '
    reduce (
        inputs / "\t" |
        try ( .[4] |= fromjson ) catch empty |
        select(.[4].Action == "GET_AUDIT") |
        [
            ( .[0] | sub("\\.\\d+Z$"; "Z") | strptime("%Y-%m-%dT%H:%M:%SZ") | mktime ),
            .[3],
            .[4]."Satellite ID",
            .[4]."Piece ID"
        ]
    ) as [ $time, $event, $satelliteID, $pieceID ] (
        [{},{}];
        if $event == "download started"
        then
            .[1][$pieceID] = $time
        else
            if (.[1] | has($pieceID) | not) or ($time - .[1][$pieceID]) / 60 > 3
            then
                .[0][$satelliteID] += 1
            else
                .
            end
        end
    ) |
    if .[0] != {} then .[0] else empty end
' /mnt/WD1003/logs/sn1.log

鉴于以下日志文​​件，我将其 ID 截断为 8 个字符，删除了不相关的 JSON 键，并用 - 标记了未使用的字段:

2022-07-24T02:37:47.570Z    -   -   download started    {"Piece ID": "CDTQRMUZ", "Satellite ID": "1wFTAgs9", "Action": "GET_AUDIT"}
2022-07-24T02:37:47.770Z    -   -   download started    {"Piece ID": "CDTQRMUZ", "Satellite ID": "1wFTAgs9", "Action": "GET_AUDIT"}
2022-07-24T02:37:47.815Z    -   -   downloaded  {"Piece ID": "CDTQRMUZ", "Satellite ID": "1wFTAgs9", "Action": "GET_AUDIT"}
2022-07-24T02:37:48.107Z    -   -   downloaded  {"Piece ID": "CDTQRMUZ", "Satellite ID": "1wFTAgs9", "Action": "GET_AUDIT"}
2022-07-26T15:47:12.285Z    -   -   download started    {"Piece ID": "CDTQRMUZ", "Satellite ID": "IADTQX62", "Action": "GET_AUDIT"}
2022-07-26T15:48:13.362Z    -   -   download started    {"Piece ID": "GFATHGO2", "Satellite ID": "4EXtmN5f", "Action": "GET_AUDIT"}
2022-07-26T15:48:13.693Z    -   -   downloaded  {"Piece ID": "GFATHGO2", "Satellite ID": "4EXtmN5f", "Action": "GET_AUDIT"}
2022-07-26T15:51:23.789Z    -   -   downloaded  {"Piece ID": "CDTQRMUZ", "Satellite ID": "IADTQX62", "Action": "GET_AUDIT"}
2022-07-26T16:11:30.625Z    -   -   downloaded  {"Piece ID": "MREMD5SU", "Satellite ID": "2LYZZEPV", "Action": "GET_AUDIT"}
2023-01-01T00:00:00.000Z - - "missing TABs" {"Action":"GET_AUDIT"}
2022-01-01T00:00:00.000Z    -   -   "broken JSON"   {
2023-01-01T00:00:00.000Z    -   -   "not a JSON object" "string"
2023-01-01T00:00:00.000Z    -   -   "no 'Action' key"   {}

^{备注:您应该以这种简化形式发布您的输入数据:-P}

reduce 的第一个 (...) 会产生如下内容:

[ 1658633867, "download started", "1wFTAgs9", "CDTQRMUZ" ]
[ 1658633867, "download started", "1wFTAgs9", "CDTQRMUZ" ]
[ 1658633867, "downloaded",       "1wFTAgs9", "CDTQRMUZ" ]
[ 1658633868, "downloaded",       "1wFTAgs9", "CDTQRMUZ" ]
[ 1658854032, "download started", "IADTQX62", "CDTQRMUZ" ]
[ 1658854093, "download started", "4EXtmN5f", "GFATHGO2" ]
[ 1658854093, "downloaded",       "4EXtmN5f", "GFATHGO2" ]
[ 1658854283, "downloaded",       "IADTQX62", "CDTQRMUZ" ]
[ 1658854800, "downloaded",       "2LYZZEPV", "MREMD5SU" ]

^{备注: "missing TABs" 和"broken JSON" 记录被try catch<丢弃 过滤器，而 “不是 JSON 对象” 和 “没有‘Action’键” 记录被 select 过滤器<丢弃/sup>}

根据当前逻辑，最终结果将是:

{
  "IADTQX62": 1,
  "2LYZZEPV": 1
}