GoLang: Running a Go binary as a SysV service on Ubuntu 14.04

The Go language with its simplicity, concurrency support,  rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.

However, the Go language does not natively provide a reliable way to daemonize itself.  In this article I will describe how to take a couple of simple Go language programs, run them using SystemV init scripts with their own process owner, standard logs, and started at boot time on Ubuntu 14.04.

If you have not installed Go on Ubuntu, first read my article here.

If you are on Ubuntu 16.04 and want to use systemd instead, read my article here.

Service Considerations

Before we start, let’s consider the issues we must address when going from running a foreground task versus a daemon.

First, the application needs to run in the background.  Because of complex interactions with the Go thread pool and forks/dropping permissions [1,2,3,4], running a simple nohup or double fork of the program is not an option – but truthfully it should not be anyway given the rich set of alternatives available today.

There are many process control systems such as Supervisor and go-daemon, but there are also OS level utilities like start-stop-daemon, upstart, and systemd.  We are going to use the start-stop-daemon in this article for Ubuntu 14.04.

Background processes are detached from the terminal, but can still receive signals, so we would like a way to catch those so we can gracefully exit if required.

For security, we should have the daemon run as its own user so that we can control exactly what privileges and file permissions are accessed.

Then we need to ensure that logging is going to the standard “/var/log/<service>” location.

Finally, the service should be part of the boot process, so that it automatically starts after reboot.

SleepService in foreground

Let’s start with a simple Go program that goes into an infinite loop, printing “hello world” to the terminal with a random sleep delay in between.  The real program logic is highlighted below, the rest is setup to catch any signals that are received.

package main

import (
        "time"
        "log"
        "flag"
        "math/rand"
        "os"
        "os/signal"
        //"syscall"
)

func main() {

        // load command line arguments
        name := flag.String("name","world","name to print")
        flag.Parse()

        log.Printf("Starting sleepservice for %s",*name)

        // setup signal catching
        sigs := make(chan os.Signal, 1)

        // catch all signals since not explicitly listing
        signal.Notify(sigs)
        //signal.Notify(sigs,syscall.SIGQUIT)

        // method invoked upon seeing signal
        go func() {
          s := <-sigs
          log.Printf("RECEIVED SIGNAL: %s",s)
          AppCleanup()
          os.Exit(1)
        }()

        // infinite print loop
        for {
          log.Printf("hello %s",*name)

          // wait random number of milliseconds
          Nsecs := rand.Intn(3000)
          log.Printf("About to sleep %dms before looping again",Nsecs)
          time.Sleep(time.Millisecond * time.Duration(Nsecs))
        }

}

func AppCleanup() {
        log.Println("CLEANUP APP BEFORE EXIT!!!")
}

First we will run it in the foreground as our current user.  Below are the commands for Linux:

$ mkdir -p $GOPATH/src/sleepservice
$ cd $GOPATH/src/sleepservice
$ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/sleepservice/sleepservice.go
$ go get
$ go build
$ ./sleepservice

Which should produce output that looks something like below that exits when you Control-C out the execution:

2017/05/20 13:41:15 Starting sleepservice for world
2017/05/20 13:41:15 hello world
2017/05/20 13:41:15 About to sleep 2081ms before looping again
2017/05/20 13:41:17 hello world
2017/05/20 13:41:17 About to sleep 1887ms before looping again
2017/05/20 13:41:19 hello world
2017/05/20 13:41:19 About to sleep 1847ms before looping again
^C2017/05/20 13:41:20 RECEIVED SIGNAL: interrupt
2017/05/20 13:41:20 CLEANUP APP BEFORE EXIT!!!

Notice that the application did not just halt abruptly.  It sensed the Control-C (SIGINT signal), performed custom cleanup of the application, then exited.

If you were to start sleepservice in one terminal, then go to a different terminal and send various signals to the process with killall:

$ sudo killall --signal SIGTRAP sleepservice
$ sudo killall --signal SIGINT sleepservice
$ sudo killall --signal SIGTERM sleepservice

You would see the application reflect those different signals, like below where a SIGTRAP was sent:

2017/05/20 13:35:23 RECEIVED SIGNAL: trace/breakpoint trap
2017/05/20 13:35:23 CLEANUP APP BEFORE EXIT!!!

SleepService as sysV service

Turning this into a service requires that we create a user just for this service and deploy an init script into “/etc/init.d”, which is based upon the “/etc/init.d/skeleton” script.

$ cd /tmp
$ sudo useradd sleepservice -s /sbin/nologin -M
$ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/sleepservice/sysv/sleepservice
$ sudo mv sleepservice /etc/init.d/.
$ sudo chmod 755 /etc/init.d/sleepservice

Then, modify the $EA_DIR variable in “/etc/init.d/sleepservice” to reflect the absolute path to the sleepservice binary.  It must be a full path because your environment variables are not valid to this context.

Now, you should be able to start the service and monitor the logs which are being sent to the standard location.

$ sudo service sleepservice start

$ tail -f /var/log/sleepservice/sleepservice.log

Note that the directory and logs under “/var/log/sleepservice” are all owned by the user:group of sleepservice:sleepservice.

Listing the running processes shows that the process is running as the “sleepservice” user.

$ ps -ef | grep sleepservice

sleepse+ 19876     1  0 12:37 ?        00:00:00 /home/vagrant/work/src/sleepservice/sleepservice --name foo

Stopping the service will show that the SIGTERM signal was sent to the application and it cleaned up before stopping.

$ sudo service sleepservice stop

$ tail -n2 /var/log/sleepservice/sleepservice.log
2017/05/20 13:44:49 RECEIVED SIGNAL: terminated
2017/05/20 13:44:49 CLEANUP APP BEFORE EXIT!!!

To have this script start at boot time:

$ sudo update-rc.d sleepservice defaults 95 10

Which will link it into the appropriate “/etc/rc?.d” directories.

EchoService in foreground

Now let’s move on to building a simple REST service that listens on port 8080 and responds to HTTP requests.  Below is a snippet of the main functionality which configures a router and handler:

func main() {

	router := mux.NewRouter().StrictSlash(true)
	router.HandleFunc("/hello/{name}", hello).Methods("GET")

	// want to start server, BUT
	// not on loopback or internal "10.x.x.x" network
	DoesNotStartWith := "10."
	IP := GetLocalIP(DoesNotStartWith)

	// start listening server
	log.Printf("creating listener on %s:%d",IP,8080)
	log.Fatal(http.ListenAndServe(fmt.Sprintf("%s:8080",IP), router))
}

func hello(w http.ResponseWriter, r *http.Request) {
	log.Println("Responding to /hello request")
	log.Println(r.UserAgent())

	// request variables
	vars := mux.Vars(r)
	log.Println("request:",vars)

	// query string parameters
	rvars := r.URL.Query()
	log.Println("query string",rvars)

	name := vars["name"]
	if name == "" {
	  name = "world"
	}

	w.WriteHeader(http.StatusOK)
	fmt.Fprintf(w, "Hello %s\n", name)
}

Here is an example of building and running the service:

$ mkdir -p $GOPATH/src/echoservice
$ cd $GOPATH/src/echoservice
$ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/echoservice/echoservice.go
$ go get
$ go build
$ sudo ufw allow 8080/tcp
$ ./echoservice

Which should produce output on the server that looks something like:

2017/05/20 06:09:52 creating listener on 192.168.2.65:8080

We can see that the server is listening on port 8080.  So now moving over to a client host, we run curl against the “/hello” service (or use a browser), sending a parameter of “foo”.

$ sudo apt-get install curl -y

$ curl "http://192.168.2.65:8080/hello/foo"
Hello foo

And on the server side the output looks like:

2017/05/20 06:10:46 Responding to /hello request
2017/05/20 06:10:46 curl/7.35.0
2017/05/20 06:10:46 request: map[name:foo]
2017/05/20 06:10:46 query string map[]

EchoService as sysV service

Turning this into a service requires that we create a user just for this service and deploy an init script into “/etc/init.d”.

$ cd /tmp
$ sudo useradd echoservice -s /sbin/nologin -M
$ https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/echoservice/sysv/echoservice
$ sudo mv echoservice /etc/init.d/.
$ sudo chmod 755 /etc/init.d/echoservice

Then, modify the $EA_DIR variable in “/etc/init.d/echoservice” to reflect the absolute path to the echoservice binary.  It must be a full path because your environment variables are not valid to this context.

Now, you should be able to start the service and monitor the logs which are being sent to the standard location.

$ sudo service echoservice start

$ tail -f /var/log/echoservice/echoservice.log

$ sudo service echoservice stop

To have this script start at boot time:

$ sudo update-rc.d echoservice defaults 95 10

Which will link it into the appropriate “/etc/rc?.d” directories.

Privileged Ports

In the above example, we have the echoservice listening on port 8080.  But if we used a port less than 1024, special privileges would need to be granted for this to run as a service (or in the foreground for that matter).

2017/05/20 06:19:15 creating listener on 192.168.2.65:80
2017/05/20 06:19:15 listen tcp 192.168.2.65:80: bind: permission denied

The way to resolve this is not to run the application as root, but to set the capabilities of the binary.  This can be done with setcap:

$ sudo apt-get install libcap2-bin -y

$ sudo setcap 'cap_net_bind_service=+ep' /your/path/gobinary

 

 

REFERENCES

https://github.com/kardianos/service

http://wiki.colar.net/golang_revel_init_script

https://github.com/golang/go/issues/227

https://github.com/sevlyar/go-daemon

https://github.com/takama/daemon

http://grokbase.com/t/gg/golang-nuts/129h03gcgp/go-nuts-how-to-write-a-daemon-process-of-linux-in-golang

http://stackoverflow.com/questions/14537045/how-i-should-run-my-golang-process-in-background

http://big-elephants.com/2013-01/writing-your-own-init-scripts/

https://github.com/mindreframer/golang-devops-stuff/blob/master/src/github.com/jordansissel/lumberjack/logstash-forwarder.init

https://unix.stackexchange.com/questions/196166/how-to-find-out-if-a-system-uses-sysv-upstart-or-systemd-initsystem

https://github.com/golang/go/issues/1435

https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/BZWXqv3YSg4

https://vincent.bernat.im/en/blog/2017-systemd-golang

http://stackoverflow.com/questions/958249/whats-the-difference-between-nohup-and-a-daemon

http://stackoverflow.com/questions/8777602/why-must-detach-from-tty-when-writing-a-linux-daemon

http://ipengineer.net/2017/03/linux-systemd-golang-services-using-kardianos-service/

https://rcrowley.org/articles/golang-graceful-stop.html

https://nathanleclaire.com/blog/2014/08/24/handling-ctrl-c-interrupt-signal-in-golang-programs/

https://gobyexample.com/signals

https://gist.github.com/reiki4040/be3705f307d3cd136e85

https://bash.cyberciti.biz/guide/Sending_signal_to_Processes

http://dustin.sallings.org/2010/02/28/running-processes.html

http://stackoverflow.com/questions/14537045/how-i-should-run-my-golang-process-in-background