The Go language with its simplicity, concurrency support, rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.
However, the Go language does not natively provide a reliable way to daemonize itself. In this article I will describe how to take a couple of simple Go language programs and run them using a systemd service file that starts them at boot time on Ubuntu 22.04.
Prerequisites
If you have not installed Go on Ubuntu, install using my article here.
Service Design considerations
Before we start, let’s consider the issues we must address when going from running a foreground task versus a daemon.
First, the application needs to run in the background. Because of complex interactions with the Go thread pool and forks/dropping permissions [1,2,3,4], running a simple nohup or double fork of the program is not an option – but truthfully it should not be anyway given the rich set of alternatives available today.
There are many process control systems such as Supervisor and monit, but with Ubuntu 22.04 we can use the systemd which is the default init system. Systemd will also take care of restarting the process if it dies, as well as starting automatically after reboot.
Background processes are detached from the terminal, but can still receive signals, so we would like a way to catch those so we can gracefully exit if required.
For security, we should have the daemon run as its own user so that we can control exactly what privileges and file permissions are accessed.
Then we need to ensure that logging is available. While ‘journald’ does capture the systemd logs in its binary format, what we really want is to have an independent file available in the standard “/var/log/<service>” location.
SleepService run in foreground
Let’s start with a simple Go program that goes into an infinite loop, printing “hello world” to the terminal with a random sleep delay in between. Any signal sent to the process will end the program gracefully.
package main
import (
"time"
"log"
"flag"
"math/rand"
"os"
"os/signal"
"syscall"
)
func main() {
// load command line arguments
name := flag.String("name","world","name to print")
flag.Parse()
log.Printf("Starting sleepservice for %s",*name)
// setup signal catching
sigs := make(chan os.Signal, 1)
// catch all signals since not explicitly listing
signal.Notify(sigs)
//signal.Notify(sigs,syscall.SIGQUIT)
// method invoked upon seeing signal
go func() {
for sig := range sigs {
log.Printf("RECEIVED SIGNAL: %s",sig)
switch sig {
case syscall.SIGURG:
log.Printf("ignoring sigurg")
default:
AppCleanup()
os.Exit(1)
} // switch
} // for
}()
// infinite print loop
for {
log.Printf("hello %s",*name)
// wait random number of milliseconds
Nsecs := rand.Intn(3000)
log.Printf("About to sleep %dms before looping again",Nsecs)
time.Sleep(time.Millisecond * time.Duration(Nsecs))
}
}
func AppCleanup() {
log.Println("CLEANUP APP BEFORE EXIT!!!")
}
Run it in the foreground as our current user. Below are the commands for Linux:
export GOPATH=$HOME/work export PATH=$PATH:/usr/local/go/bin # create directory for program mkdir -p $GOPATH/src/sleepservice cd $GOPATH/src/sleepservice # get source and build wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/sleepservice19/sleepservice.go go build sleepservice.go # run binary in foreground ./sleepservice
Which should produce output that looks something like below that exits when you Control-C out the execution:
2022/10/29 04:53:09 Starting sleepservice for world 2022/10/29 04:53:09 hello world 2022/10/29 04:53:09 About to sleep 2081ms before looping again 2022/10/29 04:53:11 hello world 2022/10/29 04:53:11 About to sleep 1887ms before looping again ^C2022/10/29 04:53:13 RECEIVED SIGNAL: interrupt 2022/10/29 04:53:13 CLEANUP APP BEFORE EXIT!!!
Notice that the application did not just halt abruptly. It sensed the Control-C (SIGINT signal), performed custom cleanup of the application, then exited.
If you were to start sleepservice in one terminal, then go to a different terminal and send various signals to the process with killall:
killall --signal SIGTRAP sleepservice # restart binary, then try this signal killall --signal SIGINT sleepservice
You would see the application reflect those different signals, like below where a SIGTRAP was sent:
2022/10/29 05:39:33 RECEIVED SIGNAL: trace/breakpoint trap 2022/10/29 05:39:33 CLEANUP APP BEFORE EXIT!!!
SleepService as Systemd service
Turning this into a service for systemd requires that we create a unit service file at “/lib/systemd/system/sleepservice.service” like below:
[Unit] Description=Sleep service ConditionPathExists=/home/ubuntu/work/src/sleepservice/sleepservice After=network.target [Service] Type=simple User=sleepservice Group=sleepservice LimitNOFILE=1024 Restart=on-failure RestartSec=10 startLimitIntervalSec=60 WorkingDirectory=/home/ubuntu/work/src/sleepservice ExecStart=/home/ubuntu/work/src/sleepservice/sleepservice --name=foo # make sure log directory exists and owned by syslog PermissionsStartOnly=true ExecStartPre=/bin/mkdir -p /var/log/sleepservice ExecStartPre=/bin/chown syslog:adm /var/log/sleepservice ExecStartPre=/bin/chmod 755 /var/log/sleepservice SyslogIdentifier=sleepservice [Install] WantedBy=multi-user.target
The absolute paths in ‘ConditionPathExists’, ‘WorkingDirectory’, and ‘ExecStart’ all need to be modified per your environment. Notice that we have instructed systemd to run the process as the user ‘sleepservice’, so we need to create that user as well.
# make user to limit privileges sudo useradd -s /sbin/nologin -M sleepservice
Below are instructions for creating the user and moving the systemd unit service file to the correct location:
# make systemd unit file wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/sleepservice19/systemd/sleepservice.service sudo mv sleepservice.service /lib/systemd/system/. sudo chmod 755 /lib/systemd/system/sleepservice.service sudo systemctl daemon-reload
Now enable the service, start it, then monitor the logs by tailing the systemd journal:
# enable serice and start sudo systemctl enable sleepservice.service sudo systemctl start sleepservice # tails logs sudo journalctl -f -u sleepservice
These logs are going to the journald binary files, but what we ultimately want is for them to be created as an independent text file under “/var/log/sleepservice”.
Journalctl forwarding to syslog
In 2019, Systemd deprecated its functionality that allowed a Systemd unit to forward stdout/stderr directly to syslog. So instead, we will have journald forward to syslog [1].
sudo sed -ri "s/^[#]ForwardToSyslog=.*/ForwardToSyslog=yes/" /etc/systemd/journald.conf
And then restart journald.
systemctl restart systemd-journald
Syslog writing to file
Now journald is forwarding to syslog, but if we make no modifications then the output of sleepservice will end up in the generic “/var/log/syslog”. We instead want the output to go to its own independent file, so we start by creating that log directory and file with the proper permissions.
logdir=/var/log/sleepservice # create directory sudo mkdir $logdir sudo chown syslog:adm $logdir sudo chmod 755 $logdir # create file sudo touch $logdir/sleepservice.log sudo chown syslog:adm $logdir/sleepservice.log sudo chmod 664 $logdir/sleepservice.log
Then configure rsyslog by creating “/etc/rsyslog.d/30-sleepservice.conf” with the following 2 lines of content:
if $programname == 'sleepservice' then /var/log/sleepservice/sleepservice.log & stop
Naming it “30-sleepservice.conf” means it has higher priority than the “50-default.conf” that would send the output to the generic /var/log/syslog.
Restart the rsyslog service and sleepservice.
sudo systemctl restart rsyslog sudo systemctl restart sleepservice
And now you should see the sleepservice output being sent to its own log file.
$ tail -f /var/log/sleepservice/sleepservice.log Oct 29 06:54:50 myserver sleepservice[2566433]: 2022/10/29 06:54:50 hello foo Oct 29 06:54:50 myserver sleepservice[2566433]: 2022/10/29 06:54:50 About to sleep 2271ms before looping again Oct 29 06:54:53 myserver sleepservice[2566433]: 2022/10/29 06:54:53 hello foo Oct 29 06:54:53 myserver sleepservice[2566433]: 2022/10/29 06:54:53 About to sleep 894ms before looping again Oct 29 06:54:54 myserver sleepservice[2566433]: 2022/10/29 06:54:54 hello foo
And the Systemd process is indeed running as the “sleepservice” user.
$ ps -ef | grep sleepservice sleepse+ 4196 1 0 16:29 ? 00:00:00 /home/ubuntu/work/src/sleepservice/sleepservice --name=foo
Validate signals sent to Systemd service
If you send a SIGINT signal and kill the process, notice that systemd automatically restarts because of the “Restart=on-failure” we indicated in the service file (see Table1).
$ sudo killall -s SIGNINT sleepservice $ tail -n 10 -f /var/log/sleepservice Oct 29 07:18:01 myserver sleepservice[2566571]: 2022/10/29 07:18:01 hello foo Oct 29 07:18:01 myserver sleepservice[2566571]: 2022/10/29 07:18:01 About to sleep 2417ms before looping again Oct 29 07:18:02 myserver sleepservice[2566571]: 2022/10/29 07:18:02 RECEIVED SIGNAL: interrupt Oct 29 07:18:02 myserver sleepservice[2566571]: 2022/10/29 07:18:02 CLEANUP APP BEFORE EXIT!!! Oct 29 07:18:12 myserver sleepservice[2566607]: 2022/10/29 07:18:12 Starting sleepservice for foo Oct 29 07:18:12 myserver sleepservice[2566607]: 2022/10/29 07:18:12 hello foo
Stopping the service will show that the SIGTERM signal was sent to the application and it cleaned up before stopping.
$ sudo systemctl stop sleepservice $ tail -n2 /var/log/sleepservice/sleepservice.log Oct 29 07:11:18 myserver sleepservice[2566433]: 2022/10/29 07:11:18 RECEIVED SIGNAL: terminated Oct 29 07:11:18 myserver sleepservice[2566433]: 2022/10/29 07:11:18 CLEANUP APP BEFORE EXIT!!!
Privileged binding port
We did not have to consider it for this particular example, but if our service was binding to a privileged port (< 1024), such as a web server on port 80 or 443 then the ‘sleepservice’ user would not have enough permissions. This would result in a message like below.
bind: permission denied
The way to resolve this is NOT to run the application as root, but to set the capabilities of the binary. This can be done with setcap.
sudo apt install libcap2-bin -y sudo setcap 'cap_net_bind_service=+ep' /your/path/gobinary
REFERENCES
https://freedesktop.org/wiki/Software/systemd/
fabianlee, golang systemd service on Ubuntu 16.04
https://blog.xyzio.com/2016/06/14/setting-up-a-golang-website-to-autorun-on-ubuntu-using-systemd/
golang github issue, SIGURG signal used as preemption starting in go 1.14
https://serverfault.com/questions/479434/service-file-for-golang-app
https://vincent.bernat.im/en/blog/2017-systemd-golang
https://denbeke.be/blog/servers/running-caddy-server-as-a-service-with-systemd/
https://github.com/coreos/go-systemd
https://fabianlee.org/2017/05/20/golang-running-a-go-binary-as-a-sysv-service-on-ubuntu-14-04/
https://www.loggly.com/ultimate-guide/linux-logging-with-systemd/
https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files