php - rsyslog 性能优化

Question

应该如何设置 rsyslog以获得最佳性能？

• 我们可以允许一些元素在服务器崩溃时丢失或直接丢失。
• 我们要将日志保存到 MySQL 数据库。
• 我们希望能够每秒处理至少 100 次日志写入，延迟为 0.001 - 0.005 秒。
• 我们正在从 PHP 应用程序编写日志。

谢谢你的帮助。

Answer 1

我们刚刚使用 MongoDB 作为数据库进行了类似的练习，因此我将记录我们所做的并希望它对您有所帮助。

这是我们第一次使用 rsyslog，所以花了一些努力才找到正确的文档并将所有内容拼凑在一起。最后，我们的测试驱动程序(我们使用的是 SoapUI)能够通过使用 rsyslog 编写事务摘要记录的 php 网络服务获得 1000 TPS。

我们发现以下文章让我们开始了:

• http://www.rsyslog.com/doc/rsyslog_high_database_rate.html
• http://www.rsyslog.com/tag/mongodb/

概述是当守护进程的内存队列已满时，您将启用 rsyslog 的队列基础结构以将传入消息写入磁盘。在我们的例子中，我们启用了 $ActionQueueSaveOnShutdown，这听起来好像您不需要。然后，您将配置 rsyslog 规则集以解析传入消息并将它们传递给 MySQL 的输出处理程序。最后，您的 php 脚本将使用 openlog() 和 syslog() 来写入您想要记录的任何数据。哦，我们还必须从源代码编译 rsyslog，以启用 json/mongo 插件，这本身就是一个练习。我们在 Ubuntu 12.04 上使用 rsyslog 7.4.5。

我当然不是 rsyslog 方面的专家，但是可以为您提供我们的配置文件和代码作为起点。同样，它们适用于 MongoDB，希望它能让您了解要做什么以及在何处更改您的实现。

祝你好运!

/etc/rsyslog.conf:

$ModLoad imuxsock # provides support for local system logging
$ModLoad imklog   # provides kernel logging support (previously done by rklogd)

# Load modules for MongoDB integration: json parser and MongoDB output driver
module(load="mmjsonparse")
module(load="ommongodb")

# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

# Set the default permissions for all log files.
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

# Where to place spool files
$WorkDirectory /var/spool/rsyslog

# use queue to decouple the db writes from default message handling
# From http://www.rsyslog.com/doc/rsyslog_high_database_rate.html
$MainMsgQueueFileName mainq     # set file name for main queue, also enables disk mode
$ActionQueueType LinkedList     # use asynchronous processing
$ActionQueueFileName mongodbq   # set file name for mongo db queue, enables disk mode
$ActionResumeRetryCount -1      # infinite retries on insert failure
$ActionQueueSaveOnShutdown on   # write all queue data to disk when rsyslogd is
                                #   terminated (default is off)

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

/etc/rsyslog.d/10-mongo.conf:

input(type="imuxsock" socket="/dev/log")

template(name="mongodblocal" type="subtree" subtree="$!")

# use json parser for all "local0" facility messages, 
# if parsed successfully run the template to load the
# message into the MongoDB database.
if $syslogfacility-text == 'local0' then {
        action(type="mmjsonparse")
        if $parsesuccess == "OK" then {
                # set some local vars that are appended onto the 
                # document that's written to MongoDB
                set $!time = $timestamp;
                set $!sys = $hostname;
                set $!procid = $syslogtag;
                set $!syslog_fac = $syslogfacility;
                set $!syslog_sever = $syslogpriority;
                set $!pid = $procid;
                action(type="ommongodb" server="127.0.0.1" db="test" collection="syslog" template="mongodblocal")
        }
}

/etc/rsyslog.d/50-default.conf:注意:这会禁用默认处理的“local0”消息。

# First some standard log files.  Log by facility.
auth,authpriv.*         /var/log/auth.log

# don't write "local0" messages to syslog, 
# as they're processed using ommongodb (see 10-mongo.conf)
*.*;local0,auth,authpriv.none   -/var/log/syslog

kern.*              -/var/log/kern.log
mail.*              -/var/log/mail.log

# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
mail.err            /var/log/mail.err

# Logging for INN news system.
news.crit           /var/log/news/news.crit
news.err            /var/log/news/news.err
news.notice         -/var/log/news/news.notice

# Emergencies are sent to everybody logged in.
*.emerg                                :omusrmsg:*

php 网络服务相关调用:

// open syslog, include the process ID and open the connection to the logger 
// immediately, and use a user defined logging mechanism Local0
openlog($SCRIPT_NAME, LOG_PID | LOG_NDELAY, LOG_LOCAL0);
// note: calling closelog() is optional, and we don't use it

...
// construct $doc, which is what will be logged, change this as appropriate
// for your implementation; here $ary_headers is the request's HTTP headers,
// and $request/$response are what was posted/returned
$doc = array("headers" => $ary_headers
            ,"request" => $request
            ,"response" => $response
            );
...

// write the log entry to syslog, where it queues it and writes it to MongoDB
// NOTE: need the '@cee: ' prefix so the rsyslog json parser will process it
// See:  http://www.rsyslog.com/doc/rsyslog_conf_modules.html/mmjsonparse.html

// JSON_BIGINT_AS_STRING = Encodes large integers as their original string value.
// JSON_NUMERIC_CHECK = Encodes numeric strings as numbers.
// JSON_UNESCAPED_SLASHES = Don't escape "/".

syslog(LOG_INFO, '@cee: ' . json_encode($doc, JSON_BIGINT_AS_STRING | JSON_NUMERIC_CHECK | JSON_UNESCAPED_SLASHES));