Table of Contents
You may have some applications that you wish to be centrally logged (along with the MageStack core applications). You can send any type of file/stream to your central logging server using two methods.
Be aware that sending Magento system/exception logs, whilst possible, isn't always desirable if you have any "noisy" 3rd party extensions. Some developers can be overly eager to log with a high verbosity, which results in the log files quickly filling up.
We advise sending data to the centralised logging for those entries that are usually a single line. Multi-line outputs (eg. a Magento exception back-trace) is usually not advised. Although, if you have fairly "quiet" logs (eg. not frequently written to), then this can be advantageous, as you can configure alerts/rules around any entries like these.
File Read
Using your access server and rsyslog
(the pre-configured syslog daemon), you can send a log file automatically to the logging server by adding a custom configuration entry.
Eg. Send the Magento system.log
to the central log server
You need to enable the file load module (if it isn't already loaded),
echo '$ModLoad imfile' > /etc/rsyslog.d/imfile.conf
Create a new file
/etc/rsyslog.d/magento-example.com.conf
Then within this file, enter,
$InputFileName /microcloud/domains/example/domains/example.com/http/var/log/system.log
$InputFileTag magento-system
$InputFileStateFile e7bcf2c89ca624546dbbd0f0d9d0ce0f
$InputFileSeverity 5
$InputRunFileMonitor
if $programname == 'magento-system' then @monitor1.i:1521
& ~
The value for $InputFileStateFile
is just a unique key, we usually use the md5sum of the path to the file.
echo "/microcloud/domains/example/domains/example.com/http/var/log/system.log" | md5sum - | cut -f1 -d" "
Then restart the daemon
/etc/init.d/rsyslog restart
You'll soon start seeing the entries going into Kibana. Be aware there is a slight delay with the input file method.
UDP Stream
Your central syslog server runs on your monitoring server on port 1521 (UDP). You can stream content to this from a PHP script or via command line.
PHP
There is a good example of how this can be achieved here.
Command line
Eg. Temporarily stream the system log from example.com to the monitoring server
tail -F /microcloud/domains/example/domains/example.com/http/var/log/system.log | nc -u -q 0 monitor1.i 1521
Format
The log daemon will take an input in either of the following three formats,
All four fields (application name, timestamp, vhost and message
# Eg. magento-MyModule 2014-11-18T18:05:49+00:00 example.com here is an important error message
(?<application name>[a-zA-Z0-9.-]+) %{TIMESTAMP_ISO8601:timestamp} (?<vhost>[a-zA-Z0-9.-]{3,63}) (?<msg>.+)
No vhost specified (application name, timestamp and message)
# Eg. magento-MyModule 2014-11-18T18:05:49+00:00 example.com here is an important error message
(?<application name>[a-zA-Z0-9.-]+) %{TIMESTAMP_ISO8601:timestamp} (?<msg>.+)
Message only (message)
# Eg. here is an important error message
(?<msg>.+)