shell - 查找包含字符串的行并使用 shell 脚本将该值回显到新行

Question

我想在 bash 脚本中找到解决方案:我有原始输出日志。每行都以日期开头，例如4 月 10 日 11:17:35

我想遍历每个日志项，找到包含字符串 coderbyte heroku/router 的行。对于其中的每一个，将 request_id 值回显到新行，如果 fwd 键的值为 MASKED，则在行尾添加一个 [M]，并在其前面留一个空格

输出日志

Apr 10 11:17:35 coderbyte app/web.3: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?shareLinkId=69dff0hba0nv HTTP/1.1" 200 148 "https://coderbyte.com" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0
Apr 10 11:17:35 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?key=s2fwad2Es2" host=coderbyte.com request_id=b19a87a1-1bbb-4e67-b207-bd9f23d46afa fwd="108.31.000.000" dyno=web.3 connect=0ms service=92ms status=200 bytes=3194 protocol=https

Apr 10 11:17:35 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q" host=coderbyte.com request_id=910b07d1-3f71-4347-a1a7-bfa20384ef65 fwd="108.31.000.000" dyno=web.2 connect=1ms service=17ms status=200 bytes=4435 protocol=https

Apr 10 11:17:35 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q" host=coderbyte.com request_id=097bf65e-e189-4f9f-9dfb-4758cff411b2 fwd="108.31.000.000" dyno=web.3 connect=1ms service=10ms status=200 bytes=4435 protocol=https

Apr 10 11:17:35 coderbyte app/web.2: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?key=s2fwad2Es2 HTTP/1.1" 200 4263 "https://coderbyte.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Apr 10 11:17:35 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=4eiramcmayu0" host=coderbyte.com request_id=d48278c2-5731-464e-be38-ab9ad84ac4a8 fwd="108.31.000.000" dyno=web.4 connect=1ms service=7ms status=200 bytes=3194 protocol=https

Apr 10 11:17:35 coderbyte app/web.3: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q HTTP/1.1" 200 4263 "https://coderbyte.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Apr 10 11:17:35 coderbyte app/web.3: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q HTTP/1.1" 200 4263 "https://coderbyte.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Apr 10 11:17:36 coderbyte app/web.4: IP_MASKED - - [10/Apr/2020:18:17:35 +0000] "GET /backend/requests/editor/placeholder?shareLinkId=4eiramcmayu0 HTTP/1.1" 200 3023 "https://coderbyte.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36

Apr 10 11:17:36 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q" host=coderbyte.com request_id=8bb2413c-3c67-4180-8091-000313b8d9ca fwd="MASKED" dyno=web.3 connect=1ms service=32ms status=200 bytes=4435 protocol=https

Apr 10 11:17:36 coderbyte heroku/router: at=info method=GET path="/backend/requests/editor/placeholder?shareLinkId=tosrve4v8q8q" host=coderbyte.com request_id=10f93da3-2753-48a3-9485-857a93d8a88a fwd="MASKED" dyno=web.3 connect=1ms service=37ms status=200 bytes=4435 protocol=https

这是我的代码

#!/bin/bash
curl -s https://coderbyte.com/api/challenges/logs/web-logs-raw -O > /dev/null
#cat web-logs-raw

grep -F "coderbyte heroku/router" web-logs-raw >> test
cat test

到目前为止，这可以过滤日志并找到包含字符串 coderbyte heroku/router 的行。但是我如何将 request_id 值回显到新行，如果 fwd 键的值为 MASKED，则在行尾添加一个 [M] 并在其前面留一个空格。

输出应该是这样的

b19a87a1-1bbb-000-00000
b19a87a1-1bbb-000-11111
8bb2413c-3c67-4180-22222 [M]
10f93da3-2753-48a3-33333 [M]

Answer 1

一个类轮:

 awk '/coderbyte heroku\/router/ { split($10,map,"=");id=map[2];split($11,map1,"\"");print map1[2]=="MASKED"?id" [M]":id }' web-logs-raw

解释:

awk '/coderbyte heroku\/router/ { # Search for lines with required text
          split($10,map,"="); # Split the 10th space delimited field into the array map using "=" as the field separator
          id=map[2]; # Set the variable id to the the second index of the map array
          split($11,map1,"\""); # Split the 11th field into the array map1 using " as the field separator (this is the masked variable)
          print map1[2]=="MASKED"?id" [M]":id # If the masked entry is MASKED, print "[M]" and then the id otherwise just print the id
     }' web-logs-raw  

输出:

b19a87a1-1bbb-4e67-b207-bd9f23d46afa
910b07d1-3f71-4347-a1a7-bfa20384ef65
097bf65e-e189-4f9f-9dfb-4758cff411b2
d48278c2-5731-464e-be38-ab9ad84ac4a8
8bb2413c-3c67-4180-8091-000313b8d9ca [M]
10f93da3-2753-48a3-9485-857a93d8a88a [M]