logstash - 如何获得总结的 Varnishlog

Question

我想将 Varnishlog 与 Logstash、Elasticsearch 和 Kibana 结合使用，以大致了解“我的服务器响应请求需要多长时间”等重要事项。我的问题是，Varnish 将每个请求记录在多行中。

例如:

   14 BackendOpen  b default 127.0.0.1 42697 127.0.0.1 2727
   14 BackendXID   b 722117072
   14 TxRequest    b GET
   14 TxURL        b /images/maske.png
   14 TxProtocol   b HTTP/1.1
   14 TxHeader     b Host: ecomxenvm01:7272
   14 TxHeader     b User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
   14 TxHeader     b Accept: image/png,image/*;q=0.8,*/*;q=0.5
   14 TxHeader     b Accept-Language: de,en-US;q=0.7,en;q=0.3
   14 TxHeader     b Referer: http://ecomxenvm01:7272/format.css
   14 TxHeader     b X-Varnish: 722117072
   14 TxHeader     b Accept-Encoding: gzip
   14 RxProtocol   b HTTP/1.1
   14 RxStatus     b 200
   14 RxResponse   b OK
   14 RxHeader     b Date: Thu, 02 Jul 2015 08:27:38 GMT
   14 RxHeader     b Server: Apache/2.2.15 (CentOS)
   14 RxHeader     b Last-Modified: Wed, 24 Jun 2015 08:08:38 GMT
   14 RxHeader     b ETag: "60010-111f8-5193f039fed80"
   14 RxHeader     b Accept-Ranges: bytes
   14 RxHeader     b Content-Length: 70136
   14 RxHeader     b Connection: close
   14 RxHeader     b Content-Type: image/png
   14 Fetch_Body   b 4(length) cls 0 mklen 1
   14 Length       b 70136
   14 BackendClose b default
   12 ReqStart     c 172.16.145.134 57776 722117072
   12 RxRequest    c GET
   12 RxURL        c /images/maske.png
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: ecomxenvm01:7272
   12 RxHeader     c User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
   12 RxHeader     c Accept: image/png,image/*;q=0.8,*/*;q=0.5
   12 RxHeader     c Accept-Language: de,en-US;q=0.7,en;q=0.3
   12 RxHeader     c Accept-Encoding: gzip, deflate
   12 RxHeader     c Referer: http://ecomxenvm01:7272/format.css
   12 RxHeader     c Connection: keep-alive
   12 VCL_call     c recv lookup
   12 VCL_call     c hash
   12 Hash         c /images/maske.png
   12 Hash         c ecomxenvm01:7272
   12 VCL_return   c hash
   12 VCL_call     c miss fetch
   12 Backend      c 14 default default
   12 TTL          c 722117072 RFC 120 -1 -1 1435825659 0 1435825658 0 0
   12 VCL_call     c fetch deliver
   12 ObjProtocol  c HTTP/1.1
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Thu, 02 Jul 2015 08:27:38 GMT
   12 ObjHeader    c Server: Apache/2.2.15 (CentOS)
   12 ObjHeader    c Last-Modified: Wed, 24 Jun 2015 08:08:38 GMT
   12 ObjHeader    c ETag: "60010-111f8-5193f039fed80"
   12 ObjHeader    c Content-Type: image/png
   12 VCL_call     c deliver deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache/2.2.15 (CentOS)
   12 TxHeader     c Last-Modified: Wed, 24 Jun 2015 08:08:38 GMT
   12 TxHeader     c ETag: "60010-111f8-5193f039fed80"
   12 TxHeader     c Content-Type: image/png
   12 TxHeader     c Content-Length: 70136
   12 TxHeader     c Accept-Ranges: bytes
   12 TxHeader     c Date: Thu, 02 Jul 2015 08:27:38 GMT
   12 TxHeader     c X-Varnish: 722117072
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 Length       c 70136
   12 ReqEnd       c 722117072 1435825658.602955103 1435825658.603794336 0.012183905 0.000789165 0.000050068
   14 BackendOpen  b default 127.0.0.1 42698 127.0.0.1 2727
   14 BackendXID   b 722117073
   14 TxRequest    b GET
   14 TxURL        b /images/arrow.png
   14 TxProtocol   b HTTP/1.1
   14 TxHeader     b Host: ecomxenvm01:7272
   14 TxHeader     b User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
   14 TxHeader     b Accept: image/png,image/*;q=0.8,*/*;q=0.5
   14 TxHeader     b Accept-Language: de,en-US;q=0.7,en;q=0.3
   14 TxHeader     b Referer: http://ecomxenvm01:7272/format.css
   14 TxHeader     b X-Varnish: 722117073
   14 TxHeader     b Accept-Encoding: gzip
   14 RxProtocol   b HTTP/1.1
   14 RxStatus     b 200
   14 RxResponse   b OK
   14 RxHeader     b Date: Thu, 02 Jul 2015 08:27:38 GMT
   14 RxHeader     b Server: Apache/2.2.15 (CentOS)
   14 RxHeader     b Last-Modified: Wed, 24 Jun 2015 08:08:37 GMT
   14 RxHeader     b ETag: "60009-af-5193f0390ab40"
   14 RxHeader     b Accept-Ranges: bytes
   14 RxHeader     b Content-Length: 175
   14 RxHeader     b Connection: close
   14 RxHeader     b Content-Type: image/png
   14 Fetch_Body   b 4(length) cls 0 mklen 1
   14 Length       b 175
   14 BackendClose b default
   12 ReqStart     c 172.16.145.134 57776 722117073
   12 RxRequest    c GET
   12 RxURL        c /images/arrow.png
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: ecomxenvm01:7272
   12 RxHeader     c User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
   12 RxHeader     c Accept: image/png,image/*;q=0.8,*/*;q=0.5
   12 RxHeader     c Accept-Language: de,en-US;q=0.7,en;q=0.3
   12 RxHeader     c Accept-Encoding: gzip, deflate
   12 RxHeader     c Referer: http://ecomxenvm01:7272/format.css
   12 RxHeader     c Connection: keep-alive
   12 VCL_call     c recv lookup
   12 VCL_call     c hash
   12 Hash         c /images/arrow.png
   12 Hash         c ecomxenvm01:7272
   12 VCL_return   c hash
   12 VCL_call     c miss fetch
   12 Backend      c 14 default default
   12 TTL          c 722117073 RFC 120 -1 -1 1435825659 0 1435825658 0 0
   12 VCL_call     c fetch deliver
   12 ObjProtocol  c HTTP/1.1
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Thu, 02 Jul 2015 08:27:38 GMT
   12 ObjHeader    c Server: Apache/2.2.15 (CentOS)
   12 ObjHeader    c Last-Modified: Wed, 24 Jun 2015 08:08:37 GMT
   12 ObjHeader    c ETag: "60009-af-5193f0390ab40"
   12 ObjHeader    c Content-Type: image/png
   12 VCL_call     c deliver deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache/2.2.15 (CentOS)
   12 TxHeader     c Last-Modified: Wed, 24 Jun 2015 08:08:37 GMT
   12 TxHeader     c ETag: "60009-af-5193f0390ab40"
   12 TxHeader     c Content-Type: image/png
   12 TxHeader     c Content-Length: 175
   12 TxHeader     c Accept-Ranges: bytes
   12 TxHeader     c Date: Thu, 02 Jul 2015 08:27:38 GMT
   12 TxHeader     c X-Varnish: 722117073
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 Length       c 175
   12 ReqEnd       c 722117073 1435825658.607635498 1435825658.608191013 0.003841162 0.000532389 0.000023127`

现在我的问题是:如何将这些行合并为一行？或者我如何总结这个请求？

我是否必须更改配置文件中的某些内容？或者有什么工具可以帮助我吗？

谢谢厘米

配置文件:

    DAEMON_OPTS="$DAEMON_OPTS -i RxStatus"

    # Configuration file for varnish
    #
    # /etc/init.d/varnish expects the variable $DAEMON_OPTS to be set from this
    # shell script fragment.
    #

    # Maximum number of open files (for ulimit -n)
    NFILES=131072

    # Locked shared memory (for ulimit -l)
    # Default log size is 82MB + header
    MEMLOCK=82000

    # Maximum number of threads (for ulimit -u)
    NPROCS="unlimited"

    # Maximum size of corefile (for ulimit -c). Default in Fedora is 0
    # DAEMON_COREFILE_LIMIT="unlimited"

    # Set this to 1 to make init script reload try to switch vcl without restart.
    # To make this work, you need to set the following variables
    # explicit: VARNISH_VCL_CONF, VARNISH_ADMIN_LISTEN_ADDRESS,
    # VARNISH_ADMIN_LISTEN_PORT, VARNISH_SECRET_FILE, or in short,
    # use Alternative 3, Advanced configuration, below
    RELOAD_VCL=1

    # This file contains 4 alternatives, please use only one.

    ## Alternative 1, Minimal configuration, no VCL
    #
    # Listen on port 7272, administration on localhost:6082, and forward to
    # content server on localhost:2727.  Use a fixed-size cache file.
    #
    #DAEMON_OPTS="-a :6081 \
    #             -T localhost:6082 \
    #             -b localhost:8080 \
    #             -u varnish -g varnish \
    #             -s file,/var/lib/varnish/varnish_storage.bin,1G"


    ## Alternative 2, Configuration with VCL
    #
    # Listen on port 6081, administration on localhost:6082, and forward to
    # one content server selected by the vcl file, based on the request.  Use a
    # fixed-size cache file.
    #
    #DAEMON_OPTS="-a :6081 \
    #             -T localhost:6082 \
    #             -f /etc/varnish/default.vcl \
    #             -u varnish -g varnish \
    #             -S /etc/varnish/secret \
    #             -s file,/var/lib/varnish/varnish_storage.bin,1G"


    ## Alternative 3, Advanced configuration
    #
    # See varnishd(1) for more information.
    #
    # # Main configuration file. You probably want to change it :)
    VARNISH_VCL_CONF=/etc/varnish/default.vcl
    #
    # # Default address and port to bind to
    # # Blank address means all IPv4 and IPv6 interfaces, otherwise specify
    # # a host name, an IPv4 dotted quad, or an IPv6 address in brackets.
    # VARNISH_LISTEN_ADDRESS=
    VARNISH_LISTEN_PORT=7272
    #
    # # Telnet admin interface listen address and port
    VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
    VARNISH_ADMIN_LISTEN_PORT=6082
    #
    # # Shared secret file for admin interface
    VARNISH_SECRET_FILE=/etc/varnish/secret
    #
    # # The minimum number of worker threads to start
    VARNISH_MIN_THREADS=50
    #
    # # The Maximum number of worker threads to start
    VARNISH_MAX_THREADS=1000
    #
    # # Idle timeout for worker threads
    VARNISH_THREAD_TIMEOUT=120
    #
    # # Cache file location
    VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
    #
    # # Cache file size: in bytes, optionally using k / M / G / T suffix,
    # # or in percentage of available disk space using the % suffix.
    VARNISH_STORAGE_SIZE=1G
    #
    # # Backend storage specification
    VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
    #
    # # Default TTL used when the backend does not specify one
    VARNISH_TTL=120
    #
    # # DAEMON_OPTS is used by the init script.  If you add or remove options, make
    # # sure you update this section, too.
    DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
                 -f ${VARNISH_VCL_CONF} \
                 -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
                 -t ${VARNISH_TTL} \
                 -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
                 -u varnish -g varnish \
                 -S ${VARNISH_SECRET_FILE} \
                 -s ${VARNISH_STORAGE}"
    #


    ## Alternative 4, Do It Yourself. See varnishd(1) for more information.
    #
    # DAEMON_OPTS=""

当我找到 Varnishlog 时:

[elk@ecomxenvm01 ~]$ sudo locate varnishlog
/etc/rc.d/init.d/varnishlog
/etc/rc.d/rc0.d/K10varnishlog
/etc/rc.d/rc1.d/K10varnishlog
/etc/rc.d/rc2.d/K10varnishlog
/etc/rc.d/rc3.d/K10varnishlog
/etc/rc.d/rc4.d/K10varnishlog
/etc/rc.d/rc5.d/K10varnishlog
/etc/rc.d/rc6.d/K10varnishlog
/usr/bin/varnishlog
/usr/share/man/man1/varnishlog.1.gz
/usr/share/man/man8/varnishlog_selinux.8.gz

Answer 1

你应该看看this .特别是选项 -I 和 -i。

带有标签的示例，仅显示 RxStatus 日志条目:

varnishlog -i RxStatus

使用正则表达式的示例，仅显示 ReqStart 和 ReqEnd 条目:

varnishlog -I "Req[Start|End]"