Java: Spring Boot application as a service using systemd on Ubuntu 16.04

Although in modern architectures you typically see Spring Boot executable jars running as the primary process of a container, there are still many deployment scenarios where running the jar as a service at boot time is required.

With Ubuntu 16.04, we can use the built-in systemd supervisor to run a Spring Boot application at boot time. This will enable the Java based service to run in the background as a distinct user, fully integrated into the syslog framework.

Build the echo service

The first step is to build the echo service, which is a Spring Boot based application that receives an HTTP request and outputs the context path and request parameters as an HTTP response.

$ sudo apt-get install openjdk-8-jdk git curl -y
$ sudo ufw allow 8080; sudo ufw allow 80
$ cd ~
$ git clone https://github.com/fabianlee/spring-echo-example.git
$ cd spring-echo-example
$ ./gradlew clean assemble
$ java -jar build/libs/spring-echo-example-1.0.0.jar --server.port=8080

When run, it starts by outputting a Spring banner to the console, and then you will see messages from the embedded Jetty container, and finally the message “Alive and kicking!!!” when the Spring application is started.

2018-04-17 15:30:00.422 INFO 2539 --- [ main] com.waiamu.open.SpringEchoApp : Alive and kicking!!!

From a different console, use curl to invoke the echo service and you will see output like below.

$ curl http://localhost:8080/echo?message=hello

{
 "headers" : {
 "Accept" : "*/*",
 "User-Agent" : "curl/7.47.0",
 "Host" : "localhost"
 },
 "path" : "/echo",
 "protocol" : "HTTP/1.1",
 "method" : "GET",
 "body" : null,
 "parameters" : {
 "message" : [ "hello" ]
 },
 "cookies" : null

This is basic validation of the service when run as a normal foreground process.

Creating systemd service

Turning this into a service for systemd requires that we create a unit service file “springechoservice.service” in the “/lib/systemd/system” directory.

$ sudo cp src/main/resources/scripts/springechoservice.service /lib/systemd/system/.
$ sudo sed -i -e "s#/home/vagrant/spring-echo-example#`pwd`#g" /lib/systemd/system/springechoservice.service
$ sudo chown root:root /lib/systemd/system/springechoservice.service
$ sudo chmod 755 /lib/systemd/system/springechoservice.service

For your convenience, the content of the file is also shown below.

[Unit]
Description=Spring Echo service
After=syslog.target

# CHANGE TO YOUR ENV !!!
Environment=MYDIR=/home/vagrant/spring-echo-example
Environment=SVCPORT=8080
ConditionPathExists=${MYDIR}

[Service]
Type=simple
User=springecho
Group=springecho
LimitNOFILE=1024

Environment=SVCNAME=springechoservice

Restart=on-failure
RestartSec=10
startLimitIntervalSec=60

WorkingDirectory=${MYDIR}
ExecStart=/usr/bin/java -jar ${MYDIR}/build/libs/spring-echo-example-1.0.0.jar --server.port=${SVCPORT} -Xms256m -Xmx768m

# make sure log directory exists and owned by syslog
PermissionsStartOnly=true
ExecStartPre=/bin/mkdir -p /var/log/${SVCNAME}
ExecStartPre=/bin/chown syslog:adm /var/log/${SVCNAME}
ExecStartPre=/bin/chmod 755 /var/log/${SVCNAME}
ExecStartPre=/usr/bin/touch /var/log/${SVCNAME}/${SVCNAME}.log
ExecStartPre=/bin/chown syslog:adm /var/log/${SVCNAME}/${SVCNAME}.log
ExecStartPre=/bin/chmod 755 /var/log/${SVCNAME}/${SVCNAME}.log
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=${SVCNAME}

[Install]
WantedBy=multi-user.target

We have instructed systemd to run the process as the user ‘springecho’, so we need to create that user and group.

$ sudo useradd springecho -s /sbin/nologin -M

Now, you should be able to enable the service, start it, then monitor the logs by tailing the systemd journal:

$ sudo systemctl enable springechoservice.service
$ sudo systemctl start springechoservice
$ sudo journalctl -f -u springechoservice

Listing the process should should you that the process is indeed running as the “springecho” user, and netstat should show a service running at port 8080.

$ sudo systemctl status springechoservice

$ sudo ps -ef | grep spring-echo | grep -v color
springe+ 4827 1 5 17:01 ? 00:00:04 /usr/bin/java -jar /home/vagrant/spring-echo-example/build/libs/spring-echo-example-1.0.0.jar --server.port=8080 -Xms256m -Xmx768m
$ netstat -an | grep "LISTEN " | grep 8080
tcp6 0 0 :::8080 :::* LISTEN

Logging to syslog

The systemd journal is stored as a binary file, so it cannot be tailed directly. But we have syslog forwarding enabled in the systemd service file, so it is just a matter of configuring our syslog server. For full instructions on configuring syslog on Ubuntu, read my article here. But here are quick instructions for Ubuntu 16.04.

First modify “/etc/rsyslog.conf” and uncomment the lines below which tell the server to listen for syslog messages on port 514/TCP.

module(load="imtcp")
input(type="imtcp" port="514")

Then, create “/etc/rsyslog.d/30-springechoservice.conf” with the following content:

if $programname == 'springechoservice' or $syslogtag == 'springechoservice' then /var/log/springechoservice/springechoservice.log
& stop

Now restart the rsyslog and springechoservice and you should see the logs going to the log file specified.

$ sudo systemctl restart rsyslog
$ sudo systemctl restart springechoservice
$ tail -f /var/log/springechoservice/springechoservice.log

In another terminal, if you now make a request to the service using curl

$ curl http://localhost:8080/echo?message=helloagain

You will see a message similar to the following appended to “springechoservice.log”.

Apr 15 22:51:38 xenial1 springechoservice[1859]: 2018-04-15 22:51:38.789 INFO 1859 --- [tp1006485584-15] com.waiamu.open.SpringEchoApp : REQUEST: {message=[helloagain]}

Privileged ports

In the above example, we have the springechoservice listening on port 8080. But if we used a port less than 1024 as a non-root user, special privileges would need to be granted for this to run as a service (or in the foreground for that matter).

If we changed the server port to 80 in the systemd “springechoservice.service” file, and then restarted the service, it would not come back up properly.

$ sudo systemctl daemon-reload
$ sudo systemctl restart springechoservice

Instead, we would see errors similar to below in the log.

Apr 15 19:12:29 xenial1 springechoservice[9572]: org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Jetty servlet container
...
Apr 15 19:12:29 xenial1 springechoservice[9572]: Caused by: java.net.SocketException: Permission denied

Instead of running the service as root, we will run ‘setcap’ against the Java binary which will allow it to bind to these ports. But then any JVM that uses the java binary would be allowed to bind to these privileged ports…so we will create a copy of the OpenJDK8 that will have these privileges.

In this way, we could use chown/chmod and the Linux file permissions to control access to this copy of the Java binary (removing group and world permissions). I won’t go through this exercise here, but I think the concept is clear enough.

$ sudo update-alternatives --config java

$ cd /usr/lib/jvm

$ sudo cp -r java-8-openjdk-amd64 java-8-openjdk-amd64-setcap

$ sudo setcap 'cap_net_bind_service=+eip' /usr/lib/jvm/java-8-openjdk-amd64-setcap/jre/bin/java

The java binary in our new directory now has privileges to bind to ports less than 1024. So we edit “/lib/systemd/system/springechoservice.service” so that it runs the jar using this specific binary like below:

ExecStart=/usr/lib/jvm/java-8-openjdk-amd64-setcap/jre/bin/java -jar /home/vagrant/spring-echo-example/build/libs/spring-echo-example-1.0.0.jar --server.port=80 -Xms256m -Xmx768m

Then reload the systemd configuration and restart the service.

$ sudo systemctl daemon-reload 
$ sudo systemctl restart springechoservice

And now the logs once again reflect success and requests can be made successfully to port 80.

$ curl http://localhost:80/echo?message=port80

Signals and process monitoring

Java has only rudimentary signal handling, so if you send a signal to the springechoservice, the process will be killed. But systemd will restart it after 10 seconds as defined in the “RestartSec” key of the service file.

Although the Java program cannot determine all the signal types received, systemd certainly can and it is reported in the journal log.

The one exception is the QUIT signal which generates a thread dump to the console and journal.

Sending QUIT

$ kill -s QUIT $(ps -ef | grep spring-echo-example | grep -v color | awk {'print $2'})

Results in a thread dump to the journal as well as the syslog file.

Sending SIGUSR1

$ kill -s SIGUSR1 $(ps -ef | grep spring-echo-example | grep -v color | awk {'print $2'})

Results in the following message in the journal log, and then the process is restarted after 10 seconds.

Apr 15 18:35:15 xenial1 systemd[1]: springechoservice.service: Main process exited, code=killed, status=10/USR1
 Apr 15 18:35:15 xenial1 systemd[1]: springechoservice.service: Unit entered failed state.
 Apr 15 18:35:15 xenial1 systemd[1]: springechoservice.service: Failed with result 'signal'.
 Apr 15 18:35:26 xenial1 systemd[1]: springechoservice.service: Service hold-off time over, scheduling restart.

Sending SIGINT

$ kill -s SIGINT $(ps -ef | grep spring-echo-example | grep -v color | awk {'print $2'})

Results in the following message in the journal log, and then the process is restarted after 10 seconds.

Apr 15 18:35:58 xenial1 systemd[1]: springechoservice.service: Main process exited, code=exited, status=130/n/a
 Apr 15 18:35:58 xenial1 systemd[1]: springechoservice.service: Unit entered failed state.
 Apr 15 18:35:58 xenial1 systemd[1]: springechoservice.service: Failed with result 'exit-code'.

Sending SIGKILL

$ kill -s SIGKILL $(ps -ef | grep spring-echo-example | grep -v color | awk {'print $2'})

Results in the following message in the journal log, and then the process is restarted after 10 seconds.

Apr 15 18:36:24 xenial1 systemd[1]: springechoservice.service: Main process exited, code=killed, status=9/KILL
 Apr 15 18:36:24 xenial1 systemd[1]: springechoservice.service: Unit entered failed state.
 Apr 15 18:36:24 xenial1 systemd[1]: springechoservice.service: Failed with result 'signal'.

REFERENCES

https://wiki.ubuntu.com/SystemdForUpstartUsers

https://github.com/raonigabriel/spring-echo-example (original project)

https://www.petrikainulainen.net/programming/gradle/getting-started-with-gradle-creating-a-spring-boot-web-application-project/ (example Spring Boot project)

https://patrickgrimard.io/2017/10/29/running-spring-boot-apps-with-systemd/

http://www.baeldung.com/spring-boot-app-as-a-service

https://techdev.io/en/developer-blog/jvm-applications-as-a-service-with-systemd

https://stackoverflow.com/questions/21503883/spring-boot-application-as-a-service/22121547

https://www.linuxhelp.com/how-to-install-gradle-on-ubuntu-16-04-and-derivatives/ (ppa for gradle)

https://howtoprogram.xyz/2016/09/06/install-gradle-ubuntu-16-04/ (manual install gradle on Ubuntu 16.04)

https://stackoverflow.com/questions/25769536/how-when-to-generate-gradle-wrapper-files (creating gradle wrapper)

https://www.mkyong.com/gradle/how-to-use-gradle-wrapper/ (creating gradle wrapper)