Linux读取日志文件并过滤以仅获取一次相同类型的日志消息

Question

在我的日志文件中，我有三种类型的日志消息:信息、警告和错误。我只想抓取错误消息，但是由于错误消息的类型不同，并且日志文件中可能会多次出现相同的错误消息，所以我只想抓取每种类型的错误
只有一次。我可以在 Ubuntu 终端中使用什么命令？我努力了:

grep -E 'level=error' server.log | sort --unique

但这也给了我“信息”和“警告”信息。

然后我使用了这个命令，但我仍然得到了所有三种类型的消息，而不仅仅是错误消息。

grep 'error' server.log | uniq -f 1

参数 -f 1 是跳过时间戳字段，因为它始终是唯一的。

例如，我的日志消息是:

.
.
.
11-03-2020 11:53:32" level=info msg="Starting up" file="etc/load/startwith.txt"
11-03-2020 11:53:33" level=info msg="Started" file="etc/load/startwith.txt"
11-03-2020 11:54:29" level=warning msg="Some fields missing" file="etc/load/startwith.php" 
11-03-2020 11:54:47" level=info msg="Started the process" file="etc/load/startwith.php" 
11-03-2020 11:54:51" level=info msg="Connecting to database" file="etc/db/dbinfo.php" 
11-03-2020 11:54:53" level=error msg="Database connection failed" file="etc/db/dbinfo.php"  
11-03-2020 13:26:22" level=info msg="Started back-up process" file="etc/load/startwith.php" 
11-03-2020 13:26:23" level=info msg="Starting up" file="etc/load/startwith.txt"
11-03-2020 13:26:26" level=error msg="Start up failed" file="etc/db/startwith.php" 
11-03-2020 13:26:27" level=info msg="Starting up" file="etc/load/startwith.txt"
11-03-2020 13:26:31" level=error msg="Start up failed" file="etc/db/startwith.php"
11-03-2020 13:26:32" level=info msg="Starting up" file="etc/load/startwith.txt"
11-03-2020 13:26:35" level=warning msg="Duplicate fields found" file="etc/load/startwith.php" 
11-03-2020 13:26:36" level=info msg="Started the process" file="etc/load/startwith.php" 
11-03-2020 13:26:37" level=info msg="Connecting to database" file="etc/db/dbinfo.php"
11-03-2020 13:26:38" level=info msg="Success. Connected to the database" file="etc/db/db-success.php"
11-03-2020 13:26:38" level=info msg="Inserting data to database" file="etc/db/dboperation.php"
11-03-2020 13:26:39" level=warning msg="Null fields found" file="etc/db/dboperation.php"
11-03-2020 13:26:39" level=info msg="Data inserted" file="etc/db/dboperation.php"
11-03-2020 13:26:39" level=info msg="Disconnected" file="etc/db/dboperation.php"
11-03-2020 13:26:43" level=info msg="Inserting data to database" file="etc/db/dboperation.php"
11-03-2020 13:26:43" level=error msg="Required data missing" file="etc/db/dboperation.php"
11-03-2020 13:26:44" level=info msg="Inserting data to database" file="etc/db/dboperation.php"
11-03-2020 13:26:44" level=error msg="Required data missing" file="etc/db/dboperation.php"
.
.
.

上述日志中错误的预期输出(3 种不同类型的错误，不是总的错误发生)将是:

11-03-2020 11:54:53" level=error msg="Database connection failed" file="etc/db/dbinfo.php" 
11-03-2020 13:26:31" level=error msg="Start up failed" file="etc/db/startwith.php"
11-03-2020 13:26:44" level=error msg="Required data missing" file="etc/db/dboperation.php"

所以基本上我需要过滤日志文件以获取错误消息，并且每种类型只有一个错误。

Answer 1

只是:

awk '/error/ && !seen[$4]++'

或者使用引号作为分隔符来包含完整的 msg="this text"像这样的消息:

awk -F'"' '/error/ && !seen[$3]++'

不用 awk 也可以做到 - grep error然后 nl数行然后 sort -u使用 msg= 在字段中排序唯一, 然后对行号重新排序并使用 cut 删除行号.或者 msg="part"可以用 sed 提取缓解 sort标记化。像这样:

grep error | sed 's/.* msg="\([^"]*\)"/\1\t&/' | nl -w1 |
sort -t $'\t' -u -k2,2 | sort -n -k1 | cut -f3-