linux - 如何使用 awk、perl 或 sed 从 LiveHTTPHeaders 输出中删除响应？

Question

假设我有这样的东西(这只是一个例子，实际请求会有所不同:我加载了 StackOverflow 并启用了 LiveHTTPHeaders 以处理一些示例):

http://stackoverflow.com/GET / HTTP/1.1Host: stackoverflow.comUser-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.2) Gecko/20070220 Firefox/2.0.0.2Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5Accept-Language: en-us,en;q=0.5Accept-Encoding: gzip,deflateAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7Keep-Alive: 300Connection: keep-aliveHTTP/1.x 200 OKCache-Control: privateContent-Type: text/html; charset=utf-8Content-Encoding: gzipExpires: Sat, 28 Nov 2009 16:04:24 GMTVary: Accept-EncodingServer: Microsoft-IIS/7.0Date: Sat, 28 Nov 2009 16:04:23 GMTContent-Length: 19015----------------------------------------------------------...

Full log of requests and responses is available on pastebin

And I want to remove all responses (HTTP/1.x 200 OK and everything in that response, for example) and all one liners showing page address. I would like to only have all requests left in text file with saved LiveHTTPHeaders output.

So, the output would be:

GET / HTTP/1.1Host: stackoverflow.comUser-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.2) Gecko/20070220 Firefox/2.0.0.2Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5Accept-Language: en-us,en;q=0.5Accept-Encoding: gzip,deflateAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7Keep-Alive: 300Connection: keep-aliveGET /so/all.css?v=5290 HTTP/1.1Host: sstatic.netUser-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.2) Gecko/20070220 Firefox/2.0.0.2Accept: text/css,*/*;q=0.1Accept-Language: en-us,en;q=0.5Accept-Encoding: gzip,deflateAccept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7Keep-Alive: 300Connection: keep-aliveReferer: http://stackoverflow.com/...

Again, the full text of what I want to keep is available on pastebin.

If I save LiveHTTPHeaders captured session to text file and I would like to get result like from second 'code' in this question, how do I do this? Maybe with awk, sed or perl? Or something else? I'm on Linux.

---

Edit:I'm trying to run Sinan's script. Script is this:

#!/usr/bin/perl
local $/ = "\n\n";
while (<>) {
    print if /^GET|POST/; # Add more request types as needed
}

我试过这样运行它:

./cleanup-headers.pl livehttp.txt > filtered.txt

这样:

perl cleanup-headers.pl < livehttp.txt > filtered.txt

...文件 filtered.txt 已创建，但它完全是空的。

有人在我粘贴到 pastebin 的完整 header 上试过吗？有效果吗？

Full headers

Answer 1

在 Perl 中:

local $/ = "\n\n";
while (<>) {
    print if /^(?:GET|POST)/; # Add more request types as needed
}

注意: 查看 LiveHTTPHeaders 生成的输出，条目很清楚地被两个换行符分隔，所以我认为设置 $/= "\n\n" 是比设置 $/= '' 更合适。我相信您的问题是由于您的输入文件中的行实际上是缩进的。

我最初是从 pastebin 下载文件并使用完整文件来测试我的脚本。我不相信您在计算机上用来测试的文件与您放在 pastebin 中的文件相同。

如果您想稳健地处理可能出现的缩进行，同时与 LiveHTTPHeaders 的输出格式保持一致，您应该使用如下内容:

#!/usr/bin/perl

use strict; use warnings;

local $/ = "\n\n";
while (<>) {
    next unless /^\s*(?:GET|POST)/;
    s!^\s+!!gm;
    print;
}

我认为在同一管道中使用 sed 和 perl 有点令人厌恶。