The Go language with its simplicity, concurrency support, rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.
However, the Go language does not natively provide a reliable way to daemonize itself. In this article I will describe how to take a couple of simple Go language programs, run them using SystemV init scripts with their own process owner, standard logs, and started at boot time on Ubuntu 14.04.
If you have not installed Go on Ubuntu, first read my article here.
If you are on Ubuntu 16.04 and want to use systemd instead, read my article here.
Service Considerations
Before we start, let’s consider the issues we must address when going from running a foreground task versus a daemon.
First, the application needs to run in the background. Because of complex interactions with the Go thread pool and forks/dropping permissions [1,2,3,4], running a simple nohup or double fork of the program is not an option – but truthfully it should not be anyway given the rich set of alternatives available today.
There are many process control systems such as Supervisor and go-daemon, but there are also OS level utilities like start-stop-daemon, upstart, and systemd. We are going to use the start-stop-daemon in this article for Ubuntu 14.04.
Background processes are detached from the terminal, but can still receive signals, so we would like a way to catch those so we can gracefully exit if required.
For security, we should have the daemon run as its own user so that we can control exactly what privileges and file permissions are accessed.
Then we need to ensure that logging is going to the standard “/var/log/<service>” location.
Finally, the service should be part of the boot process, so that it automatically starts after reboot.
SleepService in foreground
Let’s start with a simple Go program that goes into an infinite loop, printing “hello world” to the terminal with a random sleep delay in between. The real program logic is highlighted below, the rest is setup to catch any signals that are received.
package main
import (
"time"
"log"
"flag"
"math/rand"
"os"
"os/signal"
//"syscall"
)
func main() {
// load command line arguments
name := flag.String("name","world","name to print")
flag.Parse()
log.Printf("Starting sleepservice for %s",*name)
// setup signal catching
sigs := make(chan os.Signal, 1)
// catch all signals since not explicitly listing
signal.Notify(sigs)
//signal.Notify(sigs,syscall.SIGQUIT)
// method invoked upon seeing signal
go func() {
s := <-sigs
log.Printf("RECEIVED SIGNAL: %s",s)
AppCleanup()
os.Exit(1)
}()
// infinite print loop
for {
log.Printf("hello %s",*name)
// wait random number of milliseconds
Nsecs := rand.Intn(3000)
log.Printf("About to sleep %dms before looping again",Nsecs)
time.Sleep(time.Millisecond * time.Duration(Nsecs))
}
}
func AppCleanup() {
log.Println("CLEANUP APP BEFORE EXIT!!!")
}
First we will run it in the foreground as our current user. Below are the commands for Linux:
$ mkdir -p $GOPATH/src/sleepservice $ cd $GOPATH/src/sleepservice $ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/sleepservice/sleepservice.go $ go get $ go build $ ./sleepservice
Which should produce output that looks something like below that exits when you Control-C out the execution:
2017/05/20 13:41:15 Starting sleepservice for world 2017/05/20 13:41:15 hello world 2017/05/20 13:41:15 About to sleep 2081ms before looping again 2017/05/20 13:41:17 hello world 2017/05/20 13:41:17 About to sleep 1887ms before looping again 2017/05/20 13:41:19 hello world 2017/05/20 13:41:19 About to sleep 1847ms before looping again ^C2017/05/20 13:41:20 RECEIVED SIGNAL: interrupt 2017/05/20 13:41:20 CLEANUP APP BEFORE EXIT!!!
Notice that the application did not just halt abruptly. It sensed the Control-C (SIGINT signal), performed custom cleanup of the application, then exited.
If you were to start sleepservice in one terminal, then go to a different terminal and send various signals to the process with killall:
$ sudo killall --signal SIGTRAP sleepservice $ sudo killall --signal SIGINT sleepservice $ sudo killall --signal SIGTERM sleepservice
You would see the application reflect those different signals, like below where a SIGTRAP was sent:
2017/05/20 13:35:23 RECEIVED SIGNAL: trace/breakpoint trap 2017/05/20 13:35:23 CLEANUP APP BEFORE EXIT!!!
SleepService as sysV service
Turning this into a service requires that we create a user just for this service and deploy an init script into “/etc/init.d”, which is based upon the “/etc/init.d/skeleton” script.
$ cd /tmp $ sudo useradd sleepservice -s /sbin/nologin -M $ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/sleepservice/sysv/sleepservice $ sudo mv sleepservice /etc/init.d/. $ sudo chmod 755 /etc/init.d/sleepservice
Then, modify the $EA_DIR variable in “/etc/init.d/sleepservice” to reflect the absolute path to the sleepservice binary. It must be a full path because your environment variables are not valid to this context.
Now, you should be able to start the service and monitor the logs which are being sent to the standard location.
$ sudo service sleepservice start $ tail -f /var/log/sleepservice/sleepservice.log
Note that the directory and logs under “/var/log/sleepservice” are all owned by the user:group of sleepservice:sleepservice.
Listing the running processes shows that the process is running as the “sleepservice” user.
$ ps -ef | grep sleepservice sleepse+ 19876 1 0 12:37 ? 00:00:00 /home/vagrant/work/src/sleepservice/sleepservice --name foo
Stopping the service will show that the SIGTERM signal was sent to the application and it cleaned up before stopping.
$ sudo service sleepservice stop $ tail -n2 /var/log/sleepservice/sleepservice.log 2017/05/20 13:44:49 RECEIVED SIGNAL: terminated 2017/05/20 13:44:49 CLEANUP APP BEFORE EXIT!!!
To have this script start at boot time:
$ sudo update-rc.d sleepservice defaults 95 10
Which will link it into the appropriate “/etc/rc?.d” directories.
EchoService in foreground
Now let’s move on to building a simple REST service that listens on port 8080 and responds to HTTP requests. Below is a snippet of the main functionality which configures a router and handler:
func main() {
router := mux.NewRouter().StrictSlash(true)
router.HandleFunc("/hello/{name}", hello).Methods("GET")
// want to start server, BUT
// not on loopback or internal "10.x.x.x" network
DoesNotStartWith := "10."
IP := GetLocalIP(DoesNotStartWith)
// start listening server
log.Printf("creating listener on %s:%d",IP,8080)
log.Fatal(http.ListenAndServe(fmt.Sprintf("%s:8080",IP), router))
}
func hello(w http.ResponseWriter, r *http.Request) {
log.Println("Responding to /hello request")
log.Println(r.UserAgent())
// request variables
vars := mux.Vars(r)
log.Println("request:",vars)
// query string parameters
rvars := r.URL.Query()
log.Println("query string",rvars)
name := vars["name"]
if name == "" {
name = "world"
}
w.WriteHeader(http.StatusOK)
fmt.Fprintf(w, "Hello %s\n", name)
}
Here is an example of building and running the service:
$ mkdir -p $GOPATH/src/echoservice $ cd $GOPATH/src/echoservice $ wget https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/echoservice/echoservice.go $ go get $ go build $ sudo ufw allow 8080/tcp $ ./echoservice
Which should produce output on the server that looks something like:
2017/05/20 06:09:52 creating listener on 192.168.2.65:8080
We can see that the server is listening on port 8080. So now moving over to a client host, we run curl against the “/hello” service (or use a browser), sending a parameter of “foo”.
$ sudo apt-get install curl -y $ curl "http://192.168.2.65:8080/hello/foo" Hello foo
And on the server side the output looks like:
2017/05/20 06:10:46 Responding to /hello request 2017/05/20 06:10:46 curl/7.35.0 2017/05/20 06:10:46 request: map[name:foo] 2017/05/20 06:10:46 query string map[]
EchoService as sysV service
Turning this into a service requires that we create a user just for this service and deploy an init script into “/etc/init.d”.
$ cd /tmp $ sudo useradd echoservice -s /sbin/nologin -M $ https://raw.githubusercontent.com/fabianlee/blogcode/master/golang/echoservice/sysv/echoservice $ sudo mv echoservice /etc/init.d/. $ sudo chmod 755 /etc/init.d/echoservice
Then, modify the $EA_DIR variable in “/etc/init.d/echoservice” to reflect the absolute path to the echoservice binary. It must be a full path because your environment variables are not valid to this context.
Now, you should be able to start the service and monitor the logs which are being sent to the standard location.
$ sudo service echoservice start $ tail -f /var/log/echoservice/echoservice.log $ sudo service echoservice stop
To have this script start at boot time:
$ sudo update-rc.d echoservice defaults 95 10
Which will link it into the appropriate “/etc/rc?.d” directories.
Privileged Ports
In the above example, we have the echoservice listening on port 8080. But if we used a port less than 1024, special privileges would need to be granted for this to run as a service (or in the foreground for that matter).
2017/05/20 06:19:15 creating listener on 192.168.2.65:80 2017/05/20 06:19:15 listen tcp 192.168.2.65:80: bind: permission denied
The way to resolve this is not to run the application as root, but to set the capabilities of the binary. This can be done with setcap:
$ sudo apt-get install libcap2-bin -y $ sudo setcap 'cap_net_bind_service=+ep' /your/path/gobinary
REFERENCES
https://github.com/kardianos/service
http://wiki.colar.net/golang_revel_init_script
https://github.com/golang/go/issues/227
https://github.com/sevlyar/go-daemon
https://github.com/takama/daemon
http://stackoverflow.com/questions/14537045/how-i-should-run-my-golang-process-in-background
http://big-elephants.com/2013-01/writing-your-own-init-scripts/
https://github.com/golang/go/issues/1435
https://groups.google.com/forum/?fromgroups=#!topic/golang-nuts/BZWXqv3YSg4
https://vincent.bernat.im/en/blog/2017-systemd-golang
http://stackoverflow.com/questions/958249/whats-the-difference-between-nohup-and-a-daemon
http://stackoverflow.com/questions/8777602/why-must-detach-from-tty-when-writing-a-linux-daemon
http://ipengineer.net/2017/03/linux-systemd-golang-services-using-kardianos-service/
https://rcrowley.org/articles/golang-graceful-stop.html
https://nathanleclaire.com/blog/2014/08/24/handling-ctrl-c-interrupt-signal-in-golang-programs/
https://gobyexample.com/signals
https://gist.github.com/reiki4040/be3705f307d3cd136e85
https://bash.cyberciti.biz/guide/Sending_signal_to_Processes
http://dustin.sallings.org/2010/02/28/running-processes.html
http://stackoverflow.com/questions/14537045/how-i-should-run-my-golang-process-in-background