Configure Rsyslog with Any Log File

Modern linux distros ship with Rsyslog which has some nice additional functionality (imfile module) that provides the ability to convert any standard text file into a Syslog message.

A standard text file is a file consisting of printable characters with lines being delimited by LF.

The file is read line-by-line and any line read is passed to rsyslog’s rule engine. The rule engine applies filter conditions and selects which actions needs to be carried out. Empty lines are not processed, as they would result in empty syslog records. They are simply ignored.

State Files

Rsyslog must keep track of which parts of the monitored file are already processed. This is done in so-called “state files” that are created in the rsyslog working directory and are read on startup to resume monitoring after a shutdown. The location of the rsyslog working directory is configurable via the global(workDirectory) advanced format parameter.

Configuration

The configuration should be placed on top of the rsyslog.conf file.vi /etc/rsyslog.conf

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514

## added by Inam
module(load="imfile" PollingInterval="10")
# needs to be done just once. PollingInterval is a module directive and is only set once when loading the module


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on

# File to store the position in the journal
$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

## Added by Inam
# File 1
input(type="imfile" File="/tmp/File1"
Tag="tag1"
Severity="error"
Facility="local7")


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###


How it works

The configuration for using the Text File Input Module is very extensive. At the beginning of your rsyslog configuration file, you always load the modules. There you need to load the module for Text File Input as well. Like all other modules, this has to be made just once. Please note that the directive PollingInterval is a module directive which needs to be set when loading the module.

module(load="imfile" PollingInterval="10")

Next up comes the input and its parameters. We configure a input of a certain type and then set the parameters to be used by this input. This is basically the same principle for all inputs:

# File 1
input(type="imfile" File="/tmp/File1"
Tag="tag1"
Severity="error"
Facility="local7")

File specifies, the path and name of the text file that should be monitored. The file name must be absolute.

Tag will set a tag in front of each message pulled from the file. If you want a colon after the tag you must set it as well, it will not be added automatically.

Severity will give all log messages of a file the same severity. This is optional. By default all mesages will be set to "notice".

Facility gives alle log messages of a file the same facility. Again, this is optional. By default all messages will be set to "local0″.


After making modification to rsyslog.conf, you need to restart the rsyslog service

service rsyslog restart

Test the Message


[root@hdpmaster ~]# echo 'Hi 8' >> /tmp/File1

[root@hdpmaster ~]# tail -f /var/log/messages

Multiple Log Messages as Single Line Message


Rsyslog can group multiline log message into a single message via imfile module, however only following three read modes are supported:

Line based (0 default) - each line is a new message.

Paragraph (1) - there is a blank line between log messages.


Indented (2) - new log messages start at the beginning of a line. If a line starts with a space it is part of the log message before it.


Keep in mind that LF escaping is turned on by default so the above config statement (in rsyslog.conf) is equivalent to below.


# File 1
input(type="imfile" File="/tmp/File1"
Tag="tag1"
Severity="error"
escapelf="on"

readMode="2"
Facility="local7")
To turn off LF escaping , use below


# File 1
input(type="imfile" File="/tmp/File1"
Tag="tag1"
Severity="error"
escapelf="off"readMode="2"Facility="local7")