GoLang: Running a Go binary as a systemd service on Ubuntu 22.04


Categories: Development, Linux

The Go language with its simplicity, concurrency support,  rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.

However, the Go language does not natively provide a reliable way to daemonize itself.  In this article I will describe how to take a couple of simple Go language programs and run them using a systemd service file that starts them at boot time on Ubuntu 22.04.

Prerequisites

If you have not installed Go on Ubuntu, install using my article here.

Service Design considerations

Before we start, let’s consider the issues we must address when going from running a foreground task versus a daemon.

First, the application needs to run in the background.  Because of complex interactions with the Go thread pool and forks/dropping permissions [1,2,3,4], running a simple nohup or double fork of the program is not an option – but truthfully it should not be anyway given the rich set of alternatives available today.

There are many process control systems such as Supervisor and monit, but with Ubuntu 22.04 we can use the systemd which is the default init system.  Systemd will also take care of restarting the process if it dies, as well as starting automatically after reboot.

Background processes are detached from the terminal, but can still receive signals, so we would like a way to catch those so we can gracefully exit if required.

For security, we should have the daemon run as its own user so that we can control exactly what privileges and file permissions are accessed.

Then we need to ensure that logging is available.  While ‘journald’ does capture the systemd logs in its binary format, what we really want is to have an independent file available in the standard “/var/log/<service>” location.

SleepService run in foreground

Let’s start with a simple Go program that goes into an infinite loop, printing “hello world” to the terminal with a random sleep delay in between.  Any signal sent to the process will end the program gracefully.

package main

import (
        "time"
        "log"
        "flag"
        "math/rand"
        "os"
        "os/signal"
        "syscall"
)

func main() {

        // load command line arguments
        name := flag.String("name","world","name to print")
        flag.Parse()

        log.Printf("Starting sleepservice for %s",*name)

        // setup signal catching
        sigs := make(chan os.Signal, 1)

        // catch all signals since not explicitly listing
        signal.Notify(sigs)
        //signal.Notify(sigs,syscall.SIGQUIT)

        // method invoked upon seeing signal
        go func() {
          for sig := range sigs {
            log.Printf("RECEIVED SIGNAL: %s",sig)

            switch sig {
            case syscall.SIGURG:
              log.Printf("ignoring sigurg")
            default:
              AppCleanup()
              os.Exit(1)
            } // switch
          } // for

        }()

        // infinite print loop
        for {
          log.Printf("hello %s",*name)

          // wait random number of milliseconds
          Nsecs := rand.Intn(3000)
          log.Printf("About to sleep %dms before looping again",Nsecs)
          time.Sleep(time.Millisecond * time.Duration(Nsecs))
        }

}

func AppCleanup() {
        log.Println("CLEANUP APP BEFORE EXIT!!!")
}

Run it in the foreground as our current user.  Below are the commands for Linux:

export GOPATH=$HOME/work
export PATH=$PATH:/usr/local/go/bin

# create directory for program
mkdir -p $GOPATH/src/sleepservice
cd $GOPATH/src/sleepservice

# get source and build
wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/sleepservice19/sleepservice.go
go build sleepservice.go

# run binary in foreground
./sleepservice

Which should produce output that looks something like below that exits when you Control-C out the execution:

2022/10/29 04:53:09 Starting sleepservice for world
2022/10/29 04:53:09 hello world
2022/10/29 04:53:09 About to sleep 2081ms before looping again
2022/10/29 04:53:11 hello world
2022/10/29 04:53:11 About to sleep 1887ms before looping again
^C2022/10/29 04:53:13 RECEIVED SIGNAL: interrupt
2022/10/29 04:53:13 CLEANUP APP BEFORE EXIT!!!

Notice that the application did not just halt abruptly.  It sensed the Control-C (SIGINT signal), performed custom cleanup of the application, then exited.

If you were to start sleepservice in one terminal, then go to a different terminal and send various signals to the process with killall:

killall --signal SIGTRAP sleepservice

# restart binary, then try this signal
killall --signal SIGINT sleepservice

You would see the application reflect those different signals, like below where a SIGTRAP was sent:

2022/10/29 05:39:33 RECEIVED SIGNAL: trace/breakpoint trap
2022/10/29 05:39:33 CLEANUP APP BEFORE EXIT!!!

SleepService as Systemd service

Turning this into a service for systemd requires that we create a unit service file at “/lib/systemd/system/sleepservice.service” like below:

[Unit]
Description=Sleep service
ConditionPathExists=/home/ubuntu/work/src/sleepservice/sleepservice
After=network.target
 
[Service]
Type=simple
User=sleepservice
Group=sleepservice
LimitNOFILE=1024

Restart=on-failure
RestartSec=10
startLimitIntervalSec=60

WorkingDirectory=/home/ubuntu/work/src/sleepservice
ExecStart=/home/ubuntu/work/src/sleepservice/sleepservice --name=foo

# make sure log directory exists and owned by syslog
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/sleepservice
ExecStartPre=/bin/chown syslog:adm /var/log/sleepservice
ExecStartPre=/bin/chmod 755 /var/log/sleepservice
SyslogIdentifier=sleepservice
 
[Install]
WantedBy=multi-user.target

The absolute paths in ‘ConditionPathExists’, ‘WorkingDirectory’, and ‘ExecStart’ all need to be modified per your environment.  Notice that we have instructed systemd to run the process as the user ‘sleepservice’, so we need to create that user as well.

# make user to limit privileges
sudo useradd -s /sbin/nologin -M sleepservice

Below are instructions for creating the user and moving the systemd unit service file to the correct location:

# make systemd unit file
wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/sleepservice19/systemd/sleepservice.service
sudo mv sleepservice.service /lib/systemd/system/.
sudo chmod 755 /lib/systemd/system/sleepservice.service
sudo systemctl daemon-reload

Now enable the service, start it, then monitor the logs by tailing the systemd journal:

# enable serice and start
sudo systemctl enable sleepservice.service
sudo systemctl start sleepservice

# tails logs
sudo journalctl -f -u sleepservice

These logs are going to the journald binary files, but what we ultimately want is for them to be created as an independent text file under “/var/log/sleepservice”.

Journalctl forwarding to syslog

In 2019, Systemd deprecated its functionality that allowed a Systemd unit to forward stdout/stderr directly to syslog.  So instead, we will have journald forward to syslog [1].

sudo sed -ri "s/^[#]ForwardToSyslog=.*/ForwardToSyslog=yes/" /etc/systemd/journald.conf

And then restart journald.

systemctl restart systemd-journald

Syslog writing to file

Now journald is forwarding to syslog, but if we make no modifications then the output of sleepservice will end up in the generic “/var/log/syslog”.  We instead want the output to go to its own independent file, so we start by creating that log directory and file with the proper permissions.

logdir=/var/log/sleepservice

# create directory
sudo mkdir $logdir
sudo chown syslog:adm $logdir
sudo chmod 755 $logdir

# create file
sudo touch $logdir/sleepservice.log
sudo chown syslog:adm $logdir/sleepservice.log
sudo chmod 664 $logdir/sleepservice.log

Then configure rsyslog by creating “/etc/rsyslog.d/30-sleepservice.conf” with the following 2 lines of content:

if $programname == 'sleepservice' then /var/log/sleepservice/sleepservice.log
& stop

Naming it “30-sleepservice.conf” means it has higher priority than the “50-default.conf” that would send the output to the generic /var/log/syslog.

Restart the rsyslog service and sleepservice.

sudo systemctl restart rsyslog
sudo systemctl restart sleepservice

And now you should see the sleepservice output being sent to its own log file.

$ tail -f /var/log/sleepservice/sleepservice.log

Oct 29 06:54:50 myserver sleepservice[2566433]: 2022/10/29 06:54:50 hello foo
Oct 29 06:54:50 myserver sleepservice[2566433]: 2022/10/29 06:54:50 About to sleep 2271ms before looping again
Oct 29 06:54:53 myserver sleepservice[2566433]: 2022/10/29 06:54:53 hello foo
Oct 29 06:54:53 myserver sleepservice[2566433]: 2022/10/29 06:54:53 About to sleep 894ms before looping again
Oct 29 06:54:54 myserver sleepservice[2566433]: 2022/10/29 06:54:54 hello foo

And the Systemd process is indeed running as the “sleepservice” user.

$ ps -ef | grep sleepservice

sleepse+  4196     1  0 16:29 ?        00:00:00 /home/ubuntu/work/src/sleepservice/sleepservice --name=foo

Validate signals sent to Systemd service

If you send a SIGINT signal and kill the process, notice that systemd automatically restarts because of the “Restart=on-failure” we indicated in the service file (see Table1).

$ sudo killall -s SIGNINT sleepservice

$ tail -n 10 -f /var/log/sleepservice

Oct 29 07:18:01 myserver sleepservice[2566571]: 2022/10/29 07:18:01 hello foo
Oct 29 07:18:01 myserver sleepservice[2566571]: 2022/10/29 07:18:01 About to sleep 2417ms before looping again
Oct 29 07:18:02 myserver sleepservice[2566571]: 2022/10/29 07:18:02 RECEIVED SIGNAL: interrupt
Oct 29 07:18:02 myserver sleepservice[2566571]: 2022/10/29 07:18:02 CLEANUP APP BEFORE EXIT!!!
Oct 29 07:18:12 myserver sleepservice[2566607]: 2022/10/29 07:18:12 Starting sleepservice for foo
Oct 29 07:18:12 myserver sleepservice[2566607]: 2022/10/29 07:18:12 hello foo

Stopping the service will show that the SIGTERM signal was sent to the application and it cleaned up before stopping.

$ sudo systemctl stop sleepservice

$ tail -n2 /var/log/sleepservice/sleepservice.log

Oct 29 07:11:18 myserver sleepservice[2566433]: 2022/10/29 07:11:18 RECEIVED SIGNAL: terminated
Oct 29 07:11:18 myserver sleepservice[2566433]: 2022/10/29 07:11:18 CLEANUP APP BEFORE EXIT!!!

Privileged binding port

We did not have to consider it for this particular example, but if our service was binding to a privileged port (< 1024), such as a web server on port 80 or 443 then the ‘sleepservice’ user would not have enough permissions.  This would result in a message like below.

bind: permission denied

The way to resolve this is NOT to run the application as root, but to set the capabilities of the binary. This can be done with setcap.

sudo apt install libcap2-bin -y
sudo setcap 'cap_net_bind_service=+ep' /your/path/gobinary

 

REFERENCES

https://freedesktop.org/wiki/Software/systemd/

fabianlee, golang systemd service on Ubuntu 16.04

https://blog.xyzio.com/2016/06/14/setting-up-a-golang-website-to-autorun-on-ubuntu-using-systemd/

golang github issue, SIGURG signal used as preemption starting in go 1.14

github issue, SIGURG

https://serverfault.com/questions/479434/service-file-for-golang-app

https://vincent.bernat.im/en/blog/2017-systemd-golang

https://denbeke.be/blog/servers/running-caddy-server-as-a-service-with-systemd/

https://github.com/coreos/go-systemd

https://fabianlee.org/2017/05/20/golang-running-a-go-binary-as-a-sysv-service-on-ubuntu-14-04/

https://www.loggly.com/ultimate-guide/linux-logging-with-systemd/

https://www.digitalocean.com/community/tutorials/understanding-systemd-units-and-unit-files

https://www.digitalocean.com/community/tutorials/how-to-use-systemctl-to-manage-systemd-services-and-units