Log rotation is an essential maintenance task for managed servers. The logrotate package available in the main Ubuntu repository is easily configurable and is invoked by the cron service for automated log retention.
Installation
The first step is to install the logrotate package and make sure the cron service is running.
$ sudo apt-get install logrotate -y $ sudo service cron status
Daily log rotation is now enabled per a minimal set of rules found in ‘/etc/logrotate.d’.
Configuration
The configuration file is located at ‘/etc/logrotate.conf’. By default, it is set to run weekly, save 4 weeks of data (rotate 4), and runs all rules found in the directory ‘/etc/logrotate.d’.
I prefer to keep a daily schedule (‘daily’), with persistence for 14 days (rotate 14), but this all according to your business requirements and datacenter standards.
Logrotate does not work as a daemon, and instead runs from the system cron on a daily basis; defined in the file ‘/etc/crontab’ and executing all scripts in ‘/etc/cron.daily’. You can validate that logrotate is being invoked daily using:
$ cat /etc/cron.daily/logrotate
If you want to force a manual invocation as a dry run (no changes made):
$ sudo logrotate -f /etc/logrotate.conf -d
Modifying retention policies
As a quick example of modifying the retention policy on a standard log, let’s modify a rule so that the ‘/var/log/syslog’ persists only 3 rotations and does not compress (for easier viewing).
Modify the top half of ‘/etc/logrotate.d/rsyslog’ as below:
/var/log/syslog
{
#rotate 7
rotate 3
daily
missingok
notifempty
delaycompress
#compress
nocompress
postrotate
reload rsyslog >/dev/null 2>&1 || true
endscript
}
Now we will make manual modifications to ‘/var/log/syslog’, and invoke logrotate manually to illustrate the functionality. Run the following commands 3 times in a row:
$ sudo echo `date` >> /var/log/syslog $ sudo logrotate -f /etc/logrotate.conf $ ls -l /var/log/syslog*
Now you will have 3 files created: /var/log/syslog.{1,2,3}. View the contents of these files:
$ cat /var/log/syslog.?
Then run the echo and logrotate command again, and once again you will only have 3 files, but now the contents of the oldest files are no longer available because it has been rotate out.
$ cat /var/log/syslog.?
Before moving on, you should modify ‘/etc/logrotate.d/rsyslog’ back to its original settings.
Custom Application
The same principles will apply to a custom application that generates logs.
For purpose of illustration (I don’t expect you to have this product setup), let’s take ElasticSearch’s Kibana application. It runs as a service on Ubuntu and generates two logs: ‘/var/log/kibana/kibana.stderr’ and ‘/var/log/kibana/kibana.stderr’. This application has no understanding of log rotation, so the logs will continue to grow steadily.
And unlike the rsyslog example above where we can count on syslog being polite and allowing us to create a new log file and do a reload of its config, this application would not like logrotate creating a new file while its process is holding the file handle.
Luckily logrotate can handle this by using a copy and truncate implementation. Below is the contents of ‘/etc/logrotate.d/kibana’ (root:root mode=644) that rotates both files hourly and keeps the last 12 hours.
/var/log/kibana/kibana.stderr
/var/log/kibana/kibana.stdout
{
copytruncate
rotate 12
hourly
missingok
notifempty
delaycompress
compress
create 640 root root
sharedscripts
}
Take note that the ‘su’ option in addition to ‘create <user> <group>’ is often necessary to create the rotated archives with the same permission as the original logs. The man page can help explain the various options, and this cheatsheet can come in handy as well.
So if we ran through the command below in multiple cycles, we could see the successive ‘kibana.stdout.*.gz’ files being rotated.
$ sudo echo going to kibana stdout `date` >> /var/log/kibana/kibana.stdout $ sudo logrotate -f /etc/logrotate.d/kibana
Going back to how this is invoked by automation, remember that logrotate is being invoked by the system cron ‘/etc/cron.daily/logrotate’, so our hourly rotation is not going to be checked every hour unless we have our own script. Using the existing file as a template, do a copy:
$ sudo cp /etc/cron.daily/logrotate /etc/cron.hourly/kibana-hourly $ sudo chmod 755 /etc/cron.hourly/kibana-hourly
And then edit ‘etc/cron.hourly/kibana-hourly’ by modifying the last line to use just our kibana config as below:
/usr/sbin/logrotate -f /etc/logrotate.d/kibana
Restart the cron service, add a couple of lines to the file, and within an hour you should see rotation of the file.
$ sudo service cron restart $ sudo echo going to kibana stdout `date` >> /var/log/kibana/kibana.stdout
On Ubuntu 14.04 and earlier, the system cron time is set in “/etc/crontab”. On Ubuntu 16.04 look in “/lib/systemd/system/apt-daily.timer”.
REFERENCES
https://help.ubuntu.com/community/LinuxLogFiles
http://www.linuxcommand.org/man_pages/logrotate8.html
https://www.jamescoyle.net/cheat-sheets/676-logrotate-cheat-sheet
http://stackoverflow.com/questions/9679085/understanding-syslogd
http://stackoverflow.com/questions/7271945/logrotate-compress-files-after-the-postrotate-script
https://support.rackspace.com/how-to/understanding-logrotate-utility/
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=652600
https://askubuntu.com/questions/36971/at-what-time-does-cron-execute-daily-scripts
https://askubuntu.com/questions/824718/ubuntu-16-04-unattended-upgrades-runs-at-random-times