Today, I worked on splitting out the emerg, crit, and alert messages so they may be acted upon more rapidly.  I also got the max file size with rotation set up for all the files that will be read by SEC.  The rotation was incredibly simple with rsyslog.

After today’s work (and some new filesystem creation) the config file for rsyslog looks like this:

$ModLoad immark
$ModLoad ommail
$MarkMessagePeriod 1200
$ModLoad imuxsock
$ModLoad imudp
$UDPServerAddress *
$UDPServerRun 514
$ModLoad imtcp

$template HostDirs,"/logs/archive/%HOSTNAME%/%$year%/%$month%/%$day%/%syslogfacility-text%-%syslogseverity-text%.log"

$outchannel all_log, /logs/current/all.log, 52428800, /logs/current/rotate.all.log.sh
$outchannel emerg_log, /logs/current/emerg.log, 52428800, /logs/current/rotate.emerg.log.sh
$outchannel crit_log, /logs/current/crit.log, 52428800, /logs/current/rotate.crit.log.sh
$outchannel alert_log, /logs/current/alert.log, 52428800, /logs/current/rotate.alert.log.sh

*.* ?HostDirs
& $all_log

*.=emerg $emerg_log

*.=alert $alert_log

*.=crit $crit_log

Unless something unforeseen comes up, I think I’m probably done with the rsyslog config file.  All the heavy lifting will be done by SEC.

First, I’ll write SEC config files that parse through the emerg, alert, and crit logs in the same way a Perl script does today.  It shouldn’t be too hard to port a Perl script in to SEC config language.  The hard part will be the handling of everything that goes in to all.log.

This morning, rsyslog 4.2.0 was released with a tag of STABLE.  Given the stable tag, I’ll probably update my package to 4.2.0 and start using it in a few days.

For long term storage, I want to store the data in a directory structure like: /var/log/<hostname>/<year>/<month>/<day>/<facility>-<severity>.log

I plan to use SEC to handle the filtering of the log messages and acting on them.  To make it easy to get log entries into  SEC, I’m going to send all log entries to a single log file.

$ModLoad immark
$ModLoad ommail
$MarkMessagePeriod 1200
$ModLoad imuxsock
$ModLoad imudp
$UDPServerAddress *
$UDPServerRun 514
$ModLoad imtcp
$template HostDirs,"/var/log/%HOSTNAME%/%$year%/%$month%/%$day%/%syslogfacility-text%-%syslogseverity-text%.log"
*.* ?HostDirs
& /var/log/test.log

Tomorrow, I’m going to use an output template to limit the size of test.log file.

I started down the path of using Syslog-ng.  Solaris is our platform of choice for infrastructure systems.  I’ve spent several weeks (part time) trying to get Syslog-ng to compile on Solaris 10.  I’ve had different problems on x86, sun4u and sun4v.  I’ve tried 3.x, 2.x, and 1.6.x.

After a particularly frustrating afternoon of build errors with Syslog-ng, I decided to give Rsyslog a shot.  I first downloaded the version dubbed ’stable’, 3.22.0.  It had a couple of build problems right out of the box that were Solaris specific.  They showed up clearly on the mailing lists.  The build problems were addressed in the 4.x branch, so I downloaded 4.1.7.  It compiled and installed without incident.

At this point, I think I’m going with Rsyslog.  It has all the features I need and doesn’t look like it will consume much of my time.

Tomorrow I’ll try to create a package for Solaris with SMF and default config file.

Currently, we have a FreeBSD based central syslog server.  It receives about 10 million syslog messages per day.  They messages come from a mix of Solaris, Cisco, Windows, VMware, and NetApp servers.

The only automation is a simple perl scripts that generates an email when an event is alert or higher severity.  I would like to change the way we react to syslog messages.  Instead of reacting solely on the basis of severity, I would like to process all the well known, frequently hit cases.  Over time we could reduce the number of unknown log messages that appear in the system.

In the end, we will have fewer alerts and a deeper understanding of what is going on in our systems.