Today, I worked on splitting out the emerg, crit, and alert messages so they may be acted upon more rapidly.  I also got the max file size with rotation set up for all the files that will be read by SEC.  The rotation was incredibly simple with rsyslog.

After today’s work (and some new filesystem creation) the config file for rsyslog looks like this:

$ModLoad immark
$ModLoad ommail
$MarkMessagePeriod 1200
$ModLoad imuxsock
$ModLoad imudp
$UDPServerAddress *
$UDPServerRun 514
$ModLoad imtcp

$template HostDirs,"/logs/archive/%HOSTNAME%/%$year%/%$month%/%$day%/%syslogfacility-text%-%syslogseverity-text%.log"

$outchannel all_log, /logs/current/all.log, 52428800, /logs/current/rotate.all.log.sh
$outchannel emerg_log, /logs/current/emerg.log, 52428800, /logs/current/rotate.emerg.log.sh
$outchannel crit_log, /logs/current/crit.log, 52428800, /logs/current/rotate.crit.log.sh
$outchannel alert_log, /logs/current/alert.log, 52428800, /logs/current/rotate.alert.log.sh

*.* ?HostDirs
& $all_log

*.=emerg $emerg_log

*.=alert $alert_log

*.=crit $crit_log

Unless something unforeseen comes up, I think I’m probably done with the rsyslog config file.  All the heavy lifting will be done by SEC.

First, I’ll write SEC config files that parse through the emerg, alert, and crit logs in the same way a Perl script does today.  It shouldn’t be too hard to port a Perl script in to SEC config language.  The hard part will be the handling of everything that goes in to all.log.

This morning, rsyslog 4.2.0 was released with a tag of STABLE.  Given the stable tag, I’ll probably update my package to 4.2.0 and start using it in a few days.

For long term storage, I want to store the data in a directory structure like: /var/log/<hostname>/<year>/<month>/<day>/<facility>-<severity>.log

I plan to use SEC to handle the filtering of the log messages and acting on them.  To make it easy to get log entries into  SEC, I’m going to send all log entries to a single log file.

$ModLoad immark
$ModLoad ommail
$MarkMessagePeriod 1200
$ModLoad imuxsock
$ModLoad imudp
$UDPServerAddress *
$UDPServerRun 514
$ModLoad imtcp
$template HostDirs,"/var/log/%HOSTNAME%/%$year%/%$month%/%$day%/%syslogfacility-text%-%syslogseverity-text%.log"
*.* ?HostDirs
& /var/log/test.log

Tomorrow, I’m going to use an output template to limit the size of test.log file.

I started down the path of using Syslog-ng.  Solaris is our platform of choice for infrastructure systems.  I’ve spent several weeks (part time) trying to get Syslog-ng to compile on Solaris 10.  I’ve had different problems on x86, sun4u and sun4v.  I’ve tried 3.x, 2.x, and 1.6.x.

After a particularly frustrating afternoon of build errors with Syslog-ng, I decided to give Rsyslog a shot.  I first downloaded the version dubbed ’stable’, 3.22.0.  It had a couple of build problems right out of the box that were Solaris specific.  They showed up clearly on the mailing lists.  The build problems were addressed in the 4.x branch, so I downloaded 4.1.7.  It compiled and installed without incident.

At this point, I think I’m going with Rsyslog.  It has all the features I need and doesn’t look like it will consume much of my time.

Tomorrow I’ll try to create a package for Solaris with SMF and default config file.

Currently, we have a FreeBSD based central syslog server.  It receives about 10 million syslog messages per day.  They messages come from a mix of Solaris, Cisco, Windows, VMware, and NetApp servers.

The only automation is a simple perl scripts that generates an email when an event is alert or higher severity.  I would like to change the way we react to syslog messages.  Instead of reacting solely on the basis of severity, I would like to process all the well known, frequently hit cases.  Over time we could reduce the number of unknown log messages that appear in the system.

In the end, we will have fewer alerts and a deeper understanding of what is going on in our systems.


$ pkg set-authority -O http://pkg.opensolaris.org/dev dev
$ BUILD=`uname -v | sed -e "s/snv_//" -e "s/[a-z]//"`
$ pfexec pkg refresh
$ pfexec pkg install entire@0.5.11-0.${BUILD}
$ pfexec pkg install SUNWipkg@0.5.11-0.${BUILD}
$ pfexec pkg install SUNWinstall-libs
$ pfexec pkg image-update


This was pulled from:

• http://opensolaris.org/os/project/indiana/resources/relnotes/200805/image-update/
• http://sites.google.com/site/solarium/how-to-update-opensolaris

Unable to copy the source file
/usr/lib/vmware-tools/configurator/autostart-vmware-user.sh
to the destination file
/usr/dt/config/Xsession.d/9999.autostart-vmware-user.sh.

All I had to do to fix this problem was: pfexec mkdir -p /usr/dt/config/Xsession.d

My initial thought was to mount the CIFS share on the NetApp from the SUSE box and use rsync to do the data migration. I figured that if I only moved the data across the network once I would get the best performance. I had a ton of trouble with this. I’m guessing the mount_smb in this old SUSE box just wasn’t up to snuff. I got all kinds of permissions problems and timeouts on the writes. For the data that actually did get copied, I only got about 100G-200G per day throughput. I know the NetApp is capable of significantly higher throughput, so I looked for another way to move the data.

Since my normal tool of choice is Solaris, I looked for a Solaris solution. Solaris 10 update 6 is the current version we use in my shop. I couldn’t find an elegant solution on Solaris 10. What I did come up with was mount_smbfs on OpenSolaris.

I downloaded a copy of OpenSolaris 2009.06 B108 and installed it in a VM. I mounted the old and new locations via mount_smbfs and I’m using rsync to do the copies. I don’t have the performance information yet, but after about four hours there have been no errors and the speed looks *very* good. Even though I added another network hop, I was able to remove the weakest link from the data flow, mount_smb on SUSE 9.x.

If I can remember, I’ll update with some performance numbers when I have them.