Centralised logging with custom logs

You may have some applications that you wish to be centrally logged (along with the MageStack core applications). You can send any type of file/stream to your central logging server using two methods.

Be aware that sending Magento system/exception logs, whilst possible, isn't always desirable if you have any "noisy" 3rd party extensions. Some developers can be overly eager to log with a high verbosity, which results in the log files quickly filling up.

We advise sending data to the centralised logging for those entries that are usually a single line. Multi-line outputs (eg. a Magento exception back-trace) is usually not advised. Although, if you have fairly "quiet" logs (eg. not frequently written to), then this can be advantageous, as you can configure alerts/rules around any entries like these.

File Read

Using your access server and rsyslog (the pre-configured syslog daemon), you can send a log file automatically to the logging server by adding a custom configuration entry.

Eg. Send the Magento system.log to the central log server

You need to enable the file load module (if it isn't already loaded),

echo '$ModLoad imfile' > /etc/rsyslog.d/imfile.conf

Create a new file

/etc/rsyslog.d/magento-example.com.conf

Then within this file, enter,

$InputFileName /microcloud/domains/example/domains/example.com/http/var/log/system.log
$InputFileTag magento-system
$InputFileStateFile e7bcf2c89ca624546dbbd0f0d9d0ce0f
$InputFileSeverity 5
$InputRunFileMonitor

if $programname == 'magento-system' then @monitor1.i:1521
& ~

The value for $InputFileStateFile is just a unique key, we usually use the md5sum of the path to the file.

echo "/microcloud/domains/example/domains/example.com/http/var/log/system.log" | md5sum - | cut -f1 -d" "

Then restart the daemon

/etc/init.d/rsyslog restart

You'll soon start seeing the entries going into Kibana. Be aware there is a slight delay with the input file method.

UDP Stream

Your central syslog server runs on your monitoring server on port 1521 (UDP). You can stream content to this from a PHP script or via command line.

PHP

There is a good example of how this can be achieved here.

Command line

Eg. Temporarily stream the system log from example.com to the monitoring server

tail -F /microcloud/domains/example/domains/example.com/http/var/log/system.log | nc -u -q 0 monitor1.i 1521

Format

The log daemon will take an input in either of the following three formats,

All four fields (application name, timestamp, vhost and message

# Eg. magento-MyModule 2014-11-18T18:05:49+00:00 example.com here is an important error message
(?<application name>[a-zA-Z0-9.-]+) %{TIMESTAMP_ISO8601:timestamp} (?<vhost>[a-zA-Z0-9.-]{3,63}) (?<msg>.+)

No vhost specified (application name, timestamp and message)

# Eg. magento-MyModule 2014-11-18T18:05:49+00:00 example.com here is an important error message
(?<application name>[a-zA-Z0-9.-]+) %{TIMESTAMP_ISO8601:timestamp} (?<msg>.+)

Message only (message)

# Eg. here is an important error message
(?<msg>.+)