巧用rsyslog收集多套日志并做单套日志的过滤分离

237次阅读

共计 3958 个字符，预计需要花费 10 分钟才能阅读完成。

日志是 supervisor 打出来的 python 日志，且把不同格式的日志打印到了同一批文件里，需求是把带 post 和 ERROR 关键字的日志分离，并进入两个不同 kafka 的 topic 队列，目前的情况是 rsyslog 已经收集了 nginx 的访问日志，不能相互影响，就是说不能直接用 if 判断做分离，因为可能会日志混掉。

要收集的日志格式如下：

123 ERROR:root:requeue {“withRefresh”: false, “localPath”: “/data1/ms/cache/file_store_location/n.fdaimg.cn/translate/20170219/oobE-fyarref6029227.jpg?43”, “remotePath”: “translate/20170219/oobE-fyarref6029227.jpg?43”}
INFO:root:2017-02-22T11:53:11.395165, {“withRefresh”: false, “localPath”: “/data1/ms/cache/file_store_location/n.adfaimg.cn/w/20170222/aue–fyarref6523250.jpeg”, “remotePath”: “w/20170222/aue–fyarref6523250.jpeg”}
INFO:root:post /data1/ms/cache/file_store_location/n.fsdaimg.cn/w/20170222/aue–fyarref6523250.jpeg to w/20170222/aue–fyarref6523250.jpeg took 112.954854965 ms…

操作做之前配置的 rsyslog 的规则如下：

module(load=”imfile”)
module(load=”omkafka”)
$PreserveFQDN on
main_queue(
queue.workerthreads=”10″ # threads to work on the queue
queue.dequeueBatchSize=”1000″ # max number of messages to process at once
queue.size=”50000″ # max queue size
)
######################### nginx access #####################
$template nginxlog,”xd172\.16\.11\.44`%msg%”
ruleset(name=”nginxlog”) {
action(
broker=[“10.13.88.190:9092″,”10.13.88.191:9092″,”10.13.88.192:9092″,”10.13.88.193:9092”]
type=”omkafka”
topic=”cms-nimg-s3″
template=”nginxlog”
partitions.auto=”on”
)
}
input(type=”imfile”
File=”/data1/ms/comos/logs/access_s3.log”
Tag=””
ruleset=”nginxlog”
freshStartTail=”on”
reopenOnTruncate=”on”
)

当时想直接用 if 判断做分离，后来发现所有的日志都会进 if 判断，完全可能把日志混淆，后来测试发现,ruleset 里竟然可以嵌套 if 判断，神奇的 rsyslog，解决了一个大问题，配置如下：

module(load=”imfile”)
module(load=”omkafka”)
$PreserveFQDN on
main_queue(
queue.workerthreads=”10″ # threads to work on the queue
queue.dequeueBatchSize=”1000″ # max number of messages to process at once
queue.size=”50000″ # max queue size
)
######################### nginx access #####################
$template nginxlog,”xd172\.16\.11\.44`%msg%”
ruleset(name=”nginxlog”) {
action(
broker=[“10.13.88.190:9092″,”10.13.88.191:9092″,”10.13.88.192:9092″,”10.13.88.193:9092”]
type=”omkafka”
topic=”cms-nimg-s3″
template=”nginxlog”
partitions.auto=”on”
)
}
input(type=”imfile”
File=”/data1/ms/comos/logs/access_s3.log”
Tag=””
ruleset=”nginxlog”
freshStartTail=”on”
reopenOnTruncate=”on”
)
####################### python s3 post error################################
$template s3post,”xd172\.16\.11\.43 %msg%”
ruleset(name=”s3post”) {
if ($msg contains “post”) then {
action(
broker=[“10.13.88.190:9092″,”10.13.88.191:9092″,”10.13.88.192:9092″,”10.13.88.193:9092”]
type=”omkafka”
topic=”cms-nimg-s3-post”
template=”s3post”
partitions.auto=”on”
)
}
if ($msg contains “ERROR”) then {
action(
broker=[“10.13.88.190:9092″,”10.13.88.191:9092″,”10.13.88.192:9092″,”10.13.88.193:9092”]
type=”omkafka”
topic=”cms-nimg-s3-post-error”
template=”s3post”
partitions.auto=”on”
)
}
}
input(type=”imfile”
File=”/data1/ms/comos/logs/s3q_daemon_0.err”
Tag=””
ruleset=”s3post”
freshStartTail=”on”
reopenOnTruncate=”on”
)
input(type=”imfile”
File=”/data1/ms/comos/logs/s3q_daemon_1.err”
Tag=””
ruleset=”s3post”
freshStartTail=”on”
reopenOnTruncate=”on”
)
input(type=”imfile”
File=”/data1/ms/comos/logs/s3q_daemon_2.err”
Tag=””
ruleset=”s3post”
freshStartTail=”on”
reopenOnTruncate=”on”
)