python - Windows事件日志消息字段传递给elasticsearch

Question

我有一些从Powershell导出到CSV的事件日志。如何获取和扩展消息字段，以便可以在Elasticsearch中使用其中的字段？我目前正在使用 Pandas 将数据发送到elasticsearch。

当前的拆分方法如下:

sec_events['action'] = sec_events.join(sec_events['message'].str.split('\\s\\s\\s\\s', 1, expand=True).apply(pd.Series))[0]
sec_events['message'] = sec_events.join(sec_events['message'].str.split('\\s\\s\\s\\s', 1, expand=True).apply(pd.Series))[1]

这输出我的方法，如:

0,mycompname.domm.f.f.a,4688,successaudit,microsoft-windows-security-auditing,3/7/2017 10:38:16 am,3/7/2017 10:38:16 am,NONE,"subject:   security id:  s-1-5-18   account name:  mycompname$   account domain:  domm   logon id:  0x3e7    process information:   new process id:  0x1a54   new process name: c:\windows\system32\ipconfig.exe   token elevation type: %%1936   creator process id: 0x1b38   process command line:     token elevation type indicates the type of token that was assigned to the new process in accordance with user account control policy.    type 1 is a full token with no privileges removed or groups disabled.  a full token is only used if user account control is disabled or if the user is the built-in administrator account or a service account.    type 2 is an elevated token with no privileges removed or groups disabled.  an elevated token is used when user account control is enabled and the user chooses to start the program using run as administrator.  an elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the administrators group.    type 3 is a limited token with administrative privileges removed and administrative groups disabled.  the limited token is used when user account control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using run as administrator.",a new process has been created.
1,mycompname.domm.f.f.f,4656,failureaudit,microsoft-windows-security-auditing,3/7/2017 10:38:05 am,3/7/2017 10:38:05 am,NONE,subject:   security id:  s-1-5-18   account name:  mycompname$   account domain:  f   logon id:  0x3e7    object:   object server:  security   object type:  key   object name:  \registry\machine\system\controlset001\services\policyagent\parameters\cache   handle id:  0x0    process information:   process id:  0x1b14   process name:  c:\windows\system32\reg.exe    access request information:   transaction id:  {00000000-0000-0000-0000-000000000000}   accesses:  %%4432      %%4435         access reasons:  -   access mask:  0x9   privileges used for access check: -   restricted sid count: 0,a handle to an object was requested.

如何从导出的csv将Windows事件日志的消息字段转换为elasticsearch / json样式格式的字段？

我正在寻找类似以下内容的json格式，以便可以将其纳入logstash:

{message: {
    subject:  ,
    security id: s-1-5-18,
    account name: mycompname$,
    account domain:  domm,
    logon id:  0x3e7,
    process information:  ,
    new process id:  0x1a54,
    new process name: c:\windows\system32\ipconfig.exe,
    token elevation type: %%1936,
    creator process id: 0x1b38,
    process command line:  ,
    process command line:  ,
    string1: token elevation type indicates the type of token that was assigned to the new process in accordance with user account control policy.
             type 1 is a full token with no privileges removed or groups disabled.
             a full token is only used if user account control is disabled or if the user is the built-in administrator account or a service account.
             type 2 is an elevated token with no privileges removed or groups disabled.
             an elevated token is used when user account control is enabled and the user chooses to start the program using run as administrator.
             an elevated token is also used when an application is configured to always require administrative privilege or to always require maximum privilege, and the user is a member of the administrators group.
             type 3 is a limited token with administrative privileges removed and administrative groups disabled.
             the limited token is used when user account control is enabled, the application does not require administrative privilege, and the user does not choose to start the program using run as administrator.


    action: a new process has been created.

    }
}

我已经尝试了以下一些项目:

sec_events['message'] = sec_events.message.replace([':\s+,', '\[', '\]', ':\s+', ',\s+','\s\s\s'],['":"none","', '{"', '"}', '":"', '","','","'], regex=True)

我似乎无法正常工作。任何帮助将是惊人的谢谢。

我注意到以下情况，

3 exact white spaces = transition from field to field 
6 exact white spaces = break between the accesses field 
2 exact white spaces = transition from field name to data if the data is there. 

Answer 1

问题最终是比赛之间的比赛没有拿起逗号。

例:

,Token elevation type: %%1936,
creator process id: 0x1b38,
process command line:  C:\windows\system32\cmd.exe,
process command line:  ,

匹配只会是

 Key                    Value
 Token elevation type   %%1936
 process command line   C:\windows\system32\cmd.exe