Syslog is a message logging standard has been around for decades, but has renewed popularity as a method of log capture with the advent of containerization and centralized logging solutions.
Enabling an Ubutu 14.04 or 16.04 host to act as a syslog server only takes a few simple steps.
Syslog listener on Ubuntu 14.04
First, open “/etc/rsyslog.conf” and uncomment the following lines which means the syslog service will listen on port 514 for both TCP/UDP.
# provides UDP syslog reception $ModLoad imudp $UDPServerRun 514 # provides TCP syslog reception $ModLoad imtcp $InputTCPServerRun 514 # uncomment if you need to force the listener address #$UDPServerAddress 127.0.0.1 #$TCPServerAddress 127.0.0.1
Then restart the service, and check if the TCP listener on port 514 is visible:
$ sudo service rsyslog restart $ netstat -an | grep "LISTEN "
Syslog listener on Ubuntu 16.04
First, open “/etc/rsyslog.conf” and uncomment the following lines which means the syslog service will listen on port 514 for both TCP/UDP.
# provides UDP syslog reception #module(load="imudp") #input(type="imudp" port="514") # provides TCP syslog reception module(load="imtcp") input(type="imtcp" port="514") # syntax for forcing listener address # input(type="imtcp" port="514" address="127.0.0.1")
Then restart the service, and check if the TCP listener on port 514 is visible:
# restart using systemd $ sudo systemctl restart rsyslog # show syslog logs using systemd journal $ sudo journalctl -u rsyslog # check $ netstat -an | grep "LISTEN "
Smoke Test
On both both 14.04 and 16.04 you can send simple test messages using netcat which comes standard on Ubuntu hosts. These messages should show up in “/var/log/syslog”.
$ nc -w0 localhost 514 <<< "<14>TCP User Info msg from `hostname` on `date`" $ nc -w0 -u localhost 514 <<< "<14>UDP User Info msg from `hostname` on `date`" $ tail /var/log/syslog May 24 11:37:11 TCP User Info msg from xenial1 on Wed May 24 11:37:11 UTC 2017 May 24 11:37:16 UDP User Info msg from xenial1 on Wed May 24 11:37:16 UTC 2017
If the rsyslog service is not started (“ps -A | grep rsyslog”), then errors in the rsyslog configuration can be found by:
# rsyslogd -N1
Application Logs
Let’s take it one step further and have our application logs written to their own file. This works for both Ubuntu 14.04 and 16.04.
Create ‘/etc/rsyslog.d/30-mytest1.conf’ with the content below which tells syslog to write any syslog messages from the application ‘mytest1’ to their own file at “/var/log/mytest1/mytest1.log”
if $programname == 'mytest1' or $syslogtag == 'mytest1' then /var/log/mytest1/mytest1.log & stop
As a side note, the ‘programname’ check would be true if the syslog was being sent in the older BSD style, while the ‘syslogtag’ catches the message when sent in the newer RFC5424 style.
Now, create the log directory and restart the rsyslog service:
$ sudo mkdir -p /var/log/mytest1 $ sudo chown syslog:adm /var/log/mytest1 $ sudo chmod 755 /var/log/mytest1 # restart service on 14.04 $ sudo service rsyslog restart # restart service on 16.04 $ sudo systemctl restart rsyslog
Now use netcat to send BSD formatted syslog messages:
<PRIORITY><TIMESTAMP> <PROGRAM>[<PID>] MESSAGE
Using the following commands:
$ nc -w0 localhost 514 <<< "<14>`env LANG=us_US.UTF-8 date "+%b %d %H:%M:%S"` `hostname` mytest1[1234] TCP User Info msg from `hostname` on `date`" $ nc -w0 -u localhost 514 <<< "<14>`env LANG=us_US.UTF-8 date "+%b %d %H:%M:%S"` `hostname` mytest1[1234] UDP User Info msg from `hostname` on `date`" $ tail /var/log/mytest1/mytest1.log May 24 12:59:50 xenial1 mytest1[1234] TCP User Info msg from xenial1 on Wed May 24 12:59:50 UTC 2017 May 24 12:53:51 xenial1 mytest1[1234] UDP User Info msg from xenial1 on Wed May 24 12:53:51 UTC 2017
The <PRIORITY> of 14 corresponds to (facility*8 + priority). In this case facility=1 (user level messages) and severity=6 (informational).
Custom Formats
On both Ubuntu 14.04 and 16.04, the default syslog format is set to RSYSLOG_TraditionalFileFormat with its low-precision timestamps and unstructured message field.
If you wanted more detail or structure you could use one of the other built-in formats like RSYSLOG_SyslogProtocol23Format or create your own.
Below is an example of template we added to “/etc/rsyslog.conf” that outputs in json format and adds metadata fields for “type” and “host” that are added by syslog when an event is processed.
# default
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
# add the line below which provides json output
$template jsonRfc5424Template,"{\"type\":\"mytype1\",\"host\":\"%HOSTNAME%\",\"message\":\"<%PRI%>1 %TIMESTAMP:::date-rfc3339% %HOSTNAME% %APP-NAME% %PROCID% %MSGID% %STRUCTURED-DATA% %msg:::json%\"}\n"
Then if we modify our “/etc/rsyslog.d/30-mytest1.conf” as first introduced in the section above, and specify which template should be used:
if $programname == 'mytest1' or $syslogtag == 'mytest1' then /var/log/mytest1/mytest1.log;jsonRfc5424Template & stop
Now when we send a test message and look at the mytest1 application log, we see the json format being used.
$ nc -w0 localhost 514 <<< "<14>`env LANG=us_US.UTF-8 date "+%b %d %H:%M:%S"` `hostname` mytest1[1234] TCP User Info msg from `hostname` on `date`"
$ tail /var/log/mytest1/mytest1.log
{"type":"mytype1","host":"xenial1","message":"<14>1 2017-05-24T13:26:14+00:00 xenial1 mytest1 1234 - - TCP User Info msg from xenial1 on Wed May 24 13:26:14 UTC 2017"}
This format and the extra metadata can be valuable when sending to a logging solution that will extract and persist the fields.
Remote forwarding
If you need to forward an application log to a remote syslog server, that is also possible by modifying “/etc/rsyslog.d/50-default.conf”. You can either specify the program name like below:
# single at sign means UDP, two means TCP :programname, isequal, "mytest1" @@myremotehost:514
OR you can specify by facility, in this case we have told it to send everything going to facility ‘local4’. You would need to modify the netcat command described above to send a priority of <166> which means facility=local4 and severity=informational.
# receives priority of 166, local4*8+info (20*8+6) local4.* @@myremotehost:514
Then restart the rsyslog service. As long as the remote server is listening for the messages and is not blocked by the firewall (sudo ufw allow 514), then the message should be sent to the remote host.
Note that the message will be sent raw to the remote host. None of the template changes you specified will be applied before sending the data. If you want custom templates applied, that must be done on the receiving side.
REFERENCES
http://www.rsyslog.com/doc/master/configuration/modules/imudp.html
http://www.posteet.com/view/2143
https://github.com/Graylog2/graylog-guide-syslog-linux
https://techzone.ergon.ch/syslog-message-format
http://www.rsyslog.com/doc/v8-stable/configuration/templates.html
https://tools.ietf.org/html/rfc3164