arthurguru: Building an enterprise wide rsyslog system

Building an enterprise wide rsyslog system

Intro

Good system logging is an asset. If however you are running important systems then good system logging is essential. Most sysadmins are quite content to switch on logging but leave it at the local file system level for each host. This approach is rather old fashioned as it only lets you trouble shoot the logs on a host by host basis. Modern applications are often comprised of many hosts working in symphony with each other which makes trouble shooting much more complicated, although not impossible.
This is where a tool called rsyslog comes in very handy. Setting up a central system logging repository allows you to capture everything in the one place, and using an analyser like the free LogAnalyzer tool you can either sit happily watching the log screen for any real time alerts in your entire organisation, or run filters on the log repository to trace a problem across targeted hosts.
Indeed rsyslog proved immensely useful to me when I noticed a misconfiguration in dovecot with authentication for an embedded mail client I had set up in a web service of an extremely busy web site. I noticed a timeout message pop up in rsyslog followed by a successful authentication. I was able to filter the problem down to the problem host where I discovered that dovecot was configured to authenticate via password files before LDAP causing a minor delay for the user. The fix was easy, as too was finding the cause of the problem by using a well implemented rsyslog system.
In this document I will show you how to set up a rsyslog system for a typical enterprise.

Unix: Rsyslog stuff to know up front

I'm not a real fan of the rsyslog configuration file as it is a bit cryptic and somewhat hard to learn, but after you get the hang of it, it becomes real easy to manage. Rsyslog is pretty flexible and powerful so I guess that's why its configuration file is what it is. The key points to remember are that outside environment directives (queue sizes, protocols, backends, etc) each system message will enter the top of the configuration file and trickle down through all the rules along the way to the bottom unless it is discarded enroute. If the system message finds a rule that it matches then that rule will get executed, and if that rule doesn't discard the system message after execution, then that system message keeps going along its journey towards the bottom of the configuration file where it then falls off the edge of the world. Got it.
To TCP or to UDP, that is network congestion. You will need to consider whether to use TCP or UDP packets for your rsyslog client configuration. This decision depends on a lot of things. For example, if you have a true SOA environment and a NIC dedicated to a network for alerts then UDP packets may be the way to go. If on the other hand you run virtualised or bladed hosts with a fast backplane then sticking with TCP packets can keep things simple. You may even want to run a mix of both.
You need to speak the language and learn what log levels and facilities are, which is easy: <facility>.<level>
The enterprise wide rsyslog system introduces a few extra concepts in configuration, namely roles. These are the same bits of rsyslog software but use a custom configuration file for each role instead.
Rsyslog roles

Clients

Relays

Collectors

Rsyslog is very flexible with its configuration and often you will configure multiple roles for the important rsyslog hosts, that is, a rsyslog relay is also configured with a client role, and a rsyslog collector can also be configured to have a relay and client roles.

Unix: Prepare

You can't have both syslog and rsyslog installed so ensure syslog is uninstalled before installing rsyslog.
yum install rsyslog
Depending on the version of rsyslog you use you may have to edit /etc/sysconfig/rsyslog and set a few start up options.
SYSLOGD_OPTIONS="-c 5"
Install logrotate. Don't be a Wally and let your log files grow to such large sizes on your hosts that performance, efficiency and stability are affected. Manage them properly with logrotate. Even though you will have a central logging repository, it's also handy for each host to have its own logging area (usually /var/log) just in case the central repository can't be reached. Also note that not all software uses the syslog protocol, some write directly to the log files themselves. I usually configure logrotate to rotate the logs daily and only keep 30 days worth of local logs.
yum install logrotate
Unix: Configuring the rsyslog client host

Try as you may but you won't be able to use the same configuration file for all your rsyslog client hosts. A mail server uses different rsyslog facilities to a load balancer which use different rsyslog facilities to an LDAP server which use different rsyslog facilities to a high availability configuration and on it keeps going. You will find that at best you can get about 80% of your hosts using the same rsyslog client configuration file with the rest requiring tailored customisations.
Client configuration summary:

Because I was using a farm of virtual hosts I chose to use the TCP protocol for the client configuration.

At the very start I have a set of rules that discard a bunch of unimportant syslog messages that match a facility name.

Next I have a set of rules that try to preserve standard syslog message behaviour of the local host by writing the local syslog messages to their respective local log files. The idea here is that if rsyslog was not installed then the local system logging will look exactly the same for this host. I apply the same idea to both the rsyslog relay and collector hosts as well.

Next I have a rather noisy syslog message (from a Microsoft SCOM implementation) that is logged locally, however I don't want to send it to the rsyslog relay to clutter the network and eventually the database, so I discard it.

Next I now set up a message queue buffer of 5Mb in size that should start discarding syslog messages when it fills up.
CAUTION: The rsyslog queue should never fill up under normal operations, however if it does fill up and you don't start discarding syslog messages then some processes may hang when their syslog function is blocked, and you may end up in a situation where you will not be able to log into your host because rsyslog blocked the login process when it tried to write a syslog message to /var/log/secure.

Finally any syslog message is sent to the rsyslog relay host. I have configured the rsyslog client to send its syslog messages to a rsyslog relay host that lives on the same network. I didn't specify a FQDN for the rsyslog relay host name in the configuration file so that I can use the same file across different networks (this assumes that your rsyslog relay host FQDN names all begin as "logs.<domain.name>").

Typical client configuration of rsyslog.conf (usually /etc/rsyslog.conf):
#########################################
# Rsyslog conf file for CentOS CLIENT role
#
# This setup:
# - logs to local files
# - logs to relay host
########################################

# NOTE: Please include the following option to /etc/sysconfig/rsyslog
# SYSLOGD_OPTIONS="-c 5"

# Turn on basic functionality
$ModLoad immark.so      # provides --MARK-- message capability
$MarkMessagePeriod 3600 # mark messages appear every 60 Minutes
$ModLoad imuxsock.so    # Unix sockets
$ModLoad imklog.so      # kernel logging

# Use TCP for network logging
$ModLoad imtcp

# Define the default template
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Note messages are applied to all rules sequentially unless explicitly discarded

# Messages to discard
:syslogfacility-text, isequal, "news" ~
:syslogfacility-text, isequal, "lpr"  ~
:syslogfacility-text, isequal, "uucp" ~
:syslogfacility-text, isequal, "crit" ~

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                        /dev/console

# Log to local files
$ActionQueueType              Direct      # Log directly to file
mail.*                                        /var/log/maillog
cron.*                                        /var/log/cron
authpriv,auth.*                               /var/log/secure
local7.*                                      /var/log/boot.log
*.info;mail,authpriv,auth,cron.none           /var/log/messages

# Prevent scheduled SCOM entries from cluttering the central log stores.
:msg, contains, "opt/microsoft/scx" ~

# Log to the central logging server. Note @@=TCP, @=UDP
$ActionQueueType              FixedArray  # Use a fixed block of memory.
$ActionQueuesize              10000       # Reserve 5Mb memory, each queue element is 512b
$ActionQueueDiscardMark       9500        # If the queue looks like filling, start discarding to not block ssh/login/etc.
$ActionQueueDiscardSeverity   0           # When in discarding mode discard everything.
$ActionQueueTimeoutEnqueue    0           # When in discarding mode do not enable throttling.
*.* @@logs:10514

# The end
Unix: Configuring the rsyslog relay host

The rsyslog relay is a dedicated host of usually one per network, whose disk space is primarily used as an emergency buffer. Its function is to collect all rsyslog messages on the local network and forward them on to the rsyslog collector.
Relay configuration summary:

At the very start I allocate a huge 3Gb disk buffer which should only kick in when memory runs low.

The rsyslog relay should be configured to listen on both the TCP and UDP protocols. Most network hardware like routers and printers tend to use the UDP protocol to send syslog messages, so if you want to capture and log those messages, and you should, then enable the UDP protocol.

Next I have a set of rules that discard a bunch of unimportant syslog messages that match a facility name.

Next I relay all syslog messages to the rsyslog collector host.

Next I have a set of rules that try to preserve standard syslog message behaviour of the local host by writing the local syslog messages to their respective local log files. The idea here is that if rsyslog was not installed then the local system logging will look exactly the same for this host.

Typical relay configuration of rsyslog.conf (usually /etc/rsyslog.conf):
#########################################
# Rsyslog conf file for CentOS RELAY role
#
# This setup:
# - logs localhost messages to local files
# - logs to collector host
########################################

# Turn on basic functionality
$ModLoad immark.so      # provides --MARK-- message capability
$ModLoad imuxsock.so    # Unix sockets
$ModLoad imklog.so      # kernel logging

# Buffer for when upstream server is unavailable
$WorkDirectory /usr/local/etc/rsyslog/work  # default location for work (spool) files
$ActionQueueType LinkedList                 # use asynchronous processing
$ActionQueueFileName srvfwd                 # set file name, also enables disk mode
$ActionQueueMaxDiskSpace 3G                 # Set disk queue limit
$ActionResumeRetryCount -1                  # infinite retries on insert failure
$ActionQueueSaveOnShutdown on               # save in-memory data if rsyslog shuts down

# Enable the receiving logging interface for smart devices
$ModLoad imtcp          # TCP interface
$InputTCPServerRun 10514
# Enable the receiving logging interface for dumb devices
$ModLoad imudp          # UDP interface
$UDPServerRun 514

# Define the default template
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Note messages are applied to all rules sequentially unless explicitly discarded

# Messages to discard
:syslogfacility-text, isequal, "news" ~
:syslogfacility-text, isequal, "lpr"  ~
:syslogfacility-text, isequal, "uucp" ~

# Log to the central logging server. Note @@=TCP, @=UDP
*.* @@logs.corp.mycompany.com:10514

# Log to local files if local message otherwise discard
if $hostname != $$myhostname and $hostname != 'localhost' then ~
$ActionQueueType Direct                       # Log directly to file
mail.*                                        /var/log/maillog
cron.*                                        /var/log/cron
authpriv,auth.*                               /var/log/secure
local7.*                                      /var/log/boot.log
*.info;mail,authpriv,auth,cron.none           /var/log/messages

# The end
Unix: Configuring the rsyslog collector host

The rsyslog collector usually lives somewhere secure and central, like an internal corporate network. Its function is to act like a hub and collect all rsyslog messages from all the rsyslog relays and store them centrally. The rsyslog collector can also be configured as a relay to the local network it lives on, and as its own client.
Collector configuration summary:

At the very start I enable the database plugin and allocate a huge 3Gb disk buffer which should only kick in when memory runs low.

Because the rsyslog collector is also acting as a relay it should also be configured to listen on both the TCP and UDP protocols. Most network hardware like routers and printers tend to use the UDP protocol to send syslog messages, so if you want to capture and log those messages, and you should, then enable the UDP protocol.

Next I have a set of rules that discard a bunch of unimportant syslog messages that match a facility name.

Next I write just the important syslog messages to a global file area.

Next I write all syslog messages to a central database.

Next I have a set of rules that try to preserve standard syslog message behaviour of the local host by writing the local syslog messages to their respective local log files. The idea here is that if rsyslog was not installed then the local system logging will look exactly the same for this host.

As I will be storing the rsyslog messages in a database I need to install the rsyslog database plugin.
yum install rsyslog-mysql
Because I will also be writing the important rsyslog messages to a global file repository (which is a mounted filesystem), I need to configure logrotate to also manage these log files too which grow rapidly. Although I only keep 30 days worth of local logging, I configured logrotate to preserve 3 months of global logs and rotate them daily.
Typical collector configuration of rsyslog.conf (usually /etc/rsyslog.conf):
#########################################
# Rsyslog conf file for CentOS COLLECTOR role
#
# This setup:
# - logs localhost messages to local files
# - logs to central repository (database and global files)
########################################

# Turn on basic functionality
$ModLoad immark.so      # provides --MARK-- message capability
$ModLoad imuxsock.so    # Unix sockets
$ModLoad imklog.so      # kernel logging

# Allow logging to database
$ModLoad ommysql
# Buffer DB transactions
$WorkDirectory /usr/local/etc/rsyslog/work  # default location for work (spool) files
$ActionQueueType LinkedList                 # use asynchronous processing
$ActionQueueFileName dbq                    # set file name, also enables disk mode
$ActionQueueMaxDiskSpace 3G                 # Set disk queue limit
$ActionResumeRetryCount -1                  # infinite retries on insert failure
$ActionQueueSaveOnShutdown on               # save in-memory data if rsyslog shuts down

# Enable the receiving logging interface for smart devices
$ModLoad imtcp          # TCP interface
$InputTCPServerRun 10514
# Enable the receiving logging interface for dumb devices
$ModLoad imudp          # UDP interface
$UDPServerRun 514

# Define the default template
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Note messages are applied to all rules sequentially unless explicitly discarded

# Messages to discard
:syslogfacility-text, isequal, "news" ~
:syslogfacility-text, isequal, "lpr"  ~
:syslogfacility-text, isequal, "uucp" ~

# Log to global files
mail.*                                        /var/log/global/maillog
cron.*                                        /var/log/global/cron
authpriv,auth.*                               /var/log/global/secure
*.info;mail,authpriv,auth,cron.none           /var/log/global/messages

# Log all messages to a database server,db,user,pw
*.* :ommysql:sql-server1.corp.mycompany.com,Syslog,rsyslog,some_secret_database_password;

# Log to local files if local message otherwise discard
if $hostname != $$myhostname and $hostname != 'localhost' then ~
$ActionQueueType Direct                       # Log directly to file
mail.*                                        /var/log/maillog
cron.*                                        /var/log/cron
authpriv,auth.*                               /var/log/secure
local7.*                                      /var/log/boot.log
*.info;mail,authpriv,auth,cron.none           /var/log/messages

# The end
Unix: Create the database repository

The rsyslog-mysql plugin contains instructions for creating a blank rsyslog database which is also compatible with LogAnalyzer.
To prevent the database from growing too big in size you can use the following script I wrote called mysql_trim_syslog_table.sh to keep it trimmed.
#!/bin/sh
#
# BSD License for mysql_trim_syslog_table.sh
# Copyright (c) 2013, Arthur Gouros
# All rights reserved.
# 
# Redistribution and use in source and binary forms, with or without 
# modification, are permitted provided that the following conditions are met:
# 
# - Redistributions of source code must retain the above copyright notice, 
#   this list of conditions and the following disclaimer.
# - Redistributions in binary form must reproduce the above copyright notice, 
#   this list of conditions and the following disclaimer in the documentation 
#   and/or other materials provided with the distribution.
# - Neither the name of Arthur Gouros nor the names of its contributors 
#   may be used to endorse or promote products derived from this software 
#   without specific prior written permission.
# 
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE 
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR 
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF 
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS 
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN 
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) 
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 
# POSSIBILITY OF SUCH DAMAGE.
#
#
# Trim the size of the syslog table to a specified epoch.
#
#
# Author: Arthur Gouros 13/12/2011


# Globals
DAYS_TO_KEEP=61
DB_HOST="sql-server1.corp.mycompany.com"
DB_USERNAME="rsyslog"
DB_PASSWORD="some_secret_database_password"
SYSLOG_DATABASE="Syslog"
SYSLOG_TABLE="SystemEvents"

MYSQL="/usr/bin/mysql"
MYSQL_SCRIPT="/var/tmp/mysql_syslog_script.sql"
LOGFILE="/var/tmp/mysql_syslog_script.log"

get_mysql_date_epoch()
{
  # Generate a MySQL compatible date that is in the past.
  # FreeBSD
  # date -v-${DAYS_TO_KEEP}d '+%Y-%m-%d 00:00:00'
  # Linux
  date --date="${DAYS_TO_KEEP} days ago" '+%Y-%m-%d 00:00:00'
}


trim_syslog_table()
{
  # Create SQL query
  echo "DELETE FROM SystemEvents WHERE ReceivedAt < \"${1}\"" > ${MYSQL_SCRIPT}
  # Execute SQL query
  ${MYSQL} --skip-reconnect -h ${DB_HOST} -u ${DB_USERNAME} -p${DB_PASSWORD} ${SYSLOG_DATABASE} < ${MYSQL_SCRIPT} > ${LOGFILE} 2>&1
}


###################
# Main
###################
mysql_date_epoch="`get_mysql_date_epoch`"
trim_syslog_table "${mysql_date_epoch}"

exit 0
And run the above script daily via cron as follows:
15 0 * * * root /usr/local/sbin/mysql_trim_syslog_table.sh
Unix: Install LogAnalyzer

LogAnalyzer is a simple web based tool that lets you watch logs in near real time or filter through log messages that are stored in a database. The LogAnalyzer tool is easy to install and configure, download the latest version from the official web site and follow the instructions on setting up the web server and accessing the SQL database.
Edit config.php and include something similar:
$CFG['DefaultSourceID'] = 'Source1';

$CFG['Sources']['Source1']['ID'] = 'Source1';
$CFG['Sources']['Source1']['Name'] = 'rsyslog';
$CFG['Sources']['Source1']['ViewID'] = 'SYSLOG';
$CFG['Sources']['Source1']['SourceType'] = SOURCE_PDO;
$CFG['Sources']['Source1']['DBTableType'] = 'monitorware';
$CFG['Sources']['Source1']['DBType'] = DB_MYSQL;
$CFG['Sources']['Source1']['DBServer'] = 'sql-server1.corp.mycompany.com';
$CFG['Sources']['Source1']['DBName'] = 'Syslog';
$CFG['Sources']['Source1']['DBUser'] = 'loganalyzer';
$CFG['Sources']['Source1']['DBPassword'] = 'some_secret_password_with_readonly_access_to_SystemEvents_table';
$CFG['Sources']['Source1']['DBTableName'] = 'SystemEvents';
$CFG['Sources']['Source1']['DBEnableRowCounting'] = false;