regex - Q : log4net file parse with Regex + NXLog

Question

我正在尝试将 log4net 文件解析为 json。

这是我的示例 log4net:

2015-01-27 01:06:18,859 [7] ERROR Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider [(null)] - Get taxonomy Type Failed for Tools
2015-01-27 06:34:31,051 [26] ERROR www.Status404 [(null)] - ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20
    UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36
    HostAddress: 192.168.10.2
    RequestUrl: /ErrorPages/404.aspx
    MachineName: QA01
    Raw Url:/undefined/
    Referrer: http://qa1.www.something.com/toolset.aspx

2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled

有了这个，我使用 xm_multiline 来捕获每个日志条目。

<Extension multiline>
    Module        xm_multiline
    HeaderLine    /^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
    EndLine       /\r?\n\r?\n^\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}/
</Extension>

我使用正则表达式捕获时间戳作为标题然后我使用正则表达式捕获两次换行然后下一个时间戳作为结尾。然而，它仍然将第二个和最后一个条目视为一个日志条目。

这是输出:

{  
   "EventReceivedTime":"2015-01-27 01:06:35",
   "SourceModuleName":"log4net",
   "SourceModuleType":"im_file",
   "time":"2015-01-27 01:06:18,859",
   "thread":"7",
   "level":"ERROR",
   "logger":"Web.Cms.Content.Base.Taxonomy.TaxonomyDetectionProvider",
   "ndc":"(null)",
   "message":"Get taxonomy Type Failed for Tools"
}{  
   "EventReceivedTime":"2015-01-27 06:34:35",
   "SourceModuleName":"log4net",
   "SourceModuleType":"im_file",
   "time":"2015-01-27 06:34:31,051",
   "thread":"26",
   "level":"ERROR",
   "logger":"www.Status404",
   "ndc":"(null)",
   "message":"  ErrorId: 20150127_102b01c6-3208-48c5-8c8b-ae4f92cf2b20\r\n  UserAgent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36\r\n  HostAddress: 192.168.10.2\r\n  RequestUrl: /ErrorPages/404.aspx\r\n  MachineName: QA01\r\n  Raw Url:/undefined/\r\n  Referrer: http://qa1.www.something.com/toolset.aspx\r\n\r\n2015-01-27 06:34:33,270 [26] DEBUG Web.Caching.Core.CacheManagerBase [(null)] - Custom CacheProvider:Web.Caching.Core.AppFabricCacheManager,Web.Caching.Core Disabled"
}

我用它来产生那个输出:

Exec    if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- (.*)/s \
        { \
            $time = $1; \
            $thread = $2; \
            $level = $3; \
            $logger = $4; \
            $ndc = $5; \
            $message = $6; \
            to_json(); \
        } \
        else \
        { \
            drop(); \
        }

我还尝试通过使用它来调整它以避免将最后两个条目合并为一个。但是我无法再获得最后一个条目。

Exec    if $raw_event =~ /^(\d{4}\-\d{2}\-\d{2} \d{2}\:\d{2}\:\d{2},\d{3}) \[(\S+)\] (\S+) (\S+) \[(\S+)\] \- ([\s\S]*?)(\r?\n\r?\n|$)/ \
        { \
            $time = $1; \
            $thread = $2; \
            $level = $3; \
            $logger = $4; \
            $ndc = $5; \
            $message = $6; \
            to_json(); \
        } \
        else \
        { \
            drop(); \
        }

Answer 1

我不会费心尝试将您的日志解析为 JSON。相反，您应该直接生成 JSON。您可以直接使用一些附加程序来执行此操作，例如 log4net.ext.json :

Extend log4net facility with simple configuration options to create JSON log entries. This is especially handy to pass semantic information to other utilities, such as nxlog, LogStash, GrayLogs2 and similar.

(强调我的)

如果您需要人类可读版本的日志，您可以创建两个记录器，每个记录器输出一种格式，但我猜您无论如何都会使用 nxlog。

在我看来，正则表达式不是将自由格式日志推回结构化日志的好方法，因此您最好直接构建它。