Centralised logging with custom logs

Table of Contents

  1. File Read
  2. UDP Stream
  3. Format

You may have some applications that you wish to be centrally logged (along with the MageStack core applications). You can send any type of file/stream to your central logging server using two methods.

Be aware that sending Magento system/exception logs, whilst possible, isn't always desirable if you have any "noisy" 3rd party extensions. Some developers can be overly eager to log with a high verbosity, which results in the log files quickly filling up.

We advise sending data to the centralised logging for those entries that are usually a single line. Multi-line outputs (eg. a Magento exception back-trace) is usually not advised. Although, if you have fairly "quiet" logs (eg. not frequently written to), then this can be advantageous, as you can configure alerts/rules around any entries like these.

File Read

Using your access server and rsyslog (the pre-configured syslog daemon), you can send a log file automatically to the logging server by adding a custom configuration entry.

Eg. Send the Magento system.log to the central log server

You need to enable the file load module (if it isn't already loaded),

echo '$ModLoad imfile' > /etc/rsyslog.d/imfile.conf

Create a new file

/etc/rsyslog.d/magento-example.com.conf

Then within this file, enter,

$InputFileName /microcloud/domains/example/domains/example.com/http/var/log/system.log
$InputFileTag magento-system
$InputFileStateFile e7bcf2c89ca624546dbbd0f0d9d0ce0f
$InputFileSeverity 5
$InputRunFileMonitor

if $programname == 'magento-system' then @monitor1.i:1521
& ~

The value for $InputFileStateFile is just a unique key, we usually use the md5sum of the path to the file.

echo "/microcloud/domains/example/domains/example.com/http/var/log/system.log" | md5sum - | cut -f1 -d" "

Then restart the daemon

/etc/init.d/rsyslog restart

You'll soon start seeing the entries going into Kibana. Be aware there is a slight delay with the input file method.

UDP Stream

Your central syslog server runs on your monitoring server on port 1521 (UDP). You can stream content to this from a PHP script or via command line.

PHP

There is a good example of how this can be achieved here.

Command line

Eg. Temporarily stream the system log from example.com to the monitoring server

tail -F /microcloud/domains/example/domains/example.com/http/var/log/system.log | nc -u -q 0 monitor1.i 1521

Format

The log daemon will take an input in either of the following three formats,

All four fields (application name, timestamp, vhost and message

# Eg. magento-MyModule 2014-11-18T18:05:49+00:00 example.com here is an important error message
(?<application name>[a-zA-Z0-9.-]+) %{TIMESTAMP_ISO8601:timestamp} (?<vhost>[a-zA-Z0-9.-]{3,63}) (?<msg>.+)

No vhost specified (application name, timestamp and message)

# Eg. magento-MyModule 2014-11-18T18:05:49+00:00 example.com here is an important error message
(?<application name>[a-zA-Z0-9.-]+) %{TIMESTAMP_ISO8601:timestamp} (?<msg>.+)

Message only (message)

# Eg. here is an important error message
(?<msg>.+)