sysadmin

So, I got the package built and installed yesterday.  Today I worked on an initial configuration file.

For long term storage, I want to store the data in a directory structure like: /var/log/<hostname>/<year>/<month>/<day>/<facility>-<severity>.log

I plan to use SEC to handle the filtering of the log messages and acting on them.  To make it easy to get log entries into  SEC, I’m going to send all log entries to a single log file.

$ModLoad immark
$ModLoad ommail
$MarkMessagePeriod 1200
$ModLoad imuxsock
$ModLoad imudp
$UDPServerAddress *
$UDPServerRun 514
$ModLoad imtcp
$template HostDirs,"/var/log/%HOSTNAME%/%$year%/%$month%/%$day%/%syslogfacility-text%-%syslogseverity-text%.log"
*.* ?HostDirs
& /var/log/test.log

Tomorrow, I’m going to use an output template to limit the size of test.log file.

First choice to make in my syslog project is which daemon to use.  Syslog-ng seems to be the best supported by the community.  Rsyslog is also another contender.

I started down the path of using Syslog-ng.  Solaris is our platform of choice for infrastructure systems.  I’ve spent several weeks (part time) trying to get Syslog-ng to compile on Solaris 10.  I’ve had different problems on x86, sun4u and sun4v.  I’ve tried 3.x, 2.x, and 1.6.x.

After a particularly frustrating afternoon of build errors with Syslog-ng, I decided to give Rsyslog a shot.  I first downloaded the version dubbed ’stable’, 3.22.0.  It had a couple of build problems right out of the box that were Solaris specific.  They showed up clearly on the mailing lists.  The build problems were addressed in the 4.x branch, so I downloaded 4.1.7.  It compiled and installed without incident.

At this point, I think I’m going with Rsyslog.  It has all the features I need and doesn’t look like it will consume much of my time.

Tomorrow I’ll try to create a package for Solaris with SMF and default config file.

I’m starting a new project.  I need to refresh our aging syslog infrastructure.

Currently, we have a FreeBSD based central syslog server.  It receives about 10 million syslog messages per day.  They messages come from a mix of Solaris, Cisco, Windows, VMware, and NetApp servers.

The only automation is a simple perl scripts that generates an email when an event is alert or higher severity.  I would like to change the way we react to syslog messages.  Instead of reacting solely on the basis of severity, I would like to process all the well known, frequently hit cases.  Over time we could reduce the number of unknown log messages that appear in the system.

In the end, we will have fewer alerts and a deeper understanding of what is going on in our systems.