Docker: Sending Spring Boot logging to syslog

Building services using Spring Boot gives a development team a jump start on many production concerns, including logging. But unlike a standard deployment where logging to a local file is where the developer’s responsibility typically ends, with Docker we must think about how to log to a public space outside our ephemeral container space.

The Docker logging drivers capture all the output from a container’s stdout/stderr, and can send a container’s logs directly to most major logging solutions (syslog, Logstash, gelf, fluentd).

As an added benefit, by making the logging implementation a runtime choice for the container, it provides flexibility to use a simpler implementation during development but a highly-available, scalable logging solution in production.

If you would rather explore an alternate Docker logging architecture, using a dedicated container that senses all other containers and routes their log events, see my article on Docker logspout.

Spring Boot logging to the console

Using our Spring Boot example fully described here, we will have it output DEBUG level logs to the console. It uses SLF4J/Logback so we configure ‘src/main/resources/logback.xml’ to send events to its ConsoleAppender.

Later in this article we put this application into a Docker container, but for now let’s run it directly on our Ubuntu host machine.

[Shell]
$ sudo apt-get install curl git openjdk-7-jdk maven -y
$ git clone https://github.com/fabianlee/gs-rest-service.git
$ cd gs-rest/service/complete
$ ./mvnw package
$ ufw allow 8080/tcp
$ java -jar target/gs-rest-service-0.1.0.jar
[/Shell]

Then you can make a client call into this service by using curl:

[Shell]
$ curl http://localhost:8080/greeting
[/Shell]

Each call to the service will output debug level logs to the console. This console level output will be redirected using the appropriate Docker logging driver in the subsequent sections.

Spring Boot project deployed in Docker container

Now that we have seen our example project deployed directly on our host machine, let’s move on to deploying it into a Docker container. If you have not installed Docker, here are instructions for Ubuntu.

If you want to build the Docker image and run gs-rest-service in a docker container where port 8080 is mapped directly to the docker host:

[Shell]
$ sudo docker build -t gs-rest-service -f src/main/docker/Dockerfile .
$ sudo docker run -p 8080:8080 gs-rest-service
[/Shell]

We have the container’s port 8080 mapped directly to the host, so you can call curl on localhost:8080 exactly like before.

Docker container sending logs to json-file

Break out of the container started earlier, and start the container again explicitly specifying a json-file logging driver.

[Shell]
$ sudo docker run -p 8080:8080 –log-driver=json-file gs-rest-service
[/Shell]

Now you can view the latest logs and check the path of the persisted json file by using:

[Shell]
$ sudo docker logs $(docker ps -q)
$ sudo docker inspect –format='{{.LogPath}}’ $(docker ps -q)
[/Shell]

Note: This .json file is located on the Docker host, not the container.

Docker container sending logs to syslog

Before we send syslog output to a host, let’s make sure the host either has rsyslog enabled (see my article here for Ubuntu), or we use netcat on the host to echo what comes into port 514 via UDP (use sudo or root because port 514 < 1024).

[Shell]

# ufw allow 514/udp

# while true; do { nc -vlu 514; } done

[/Shell]

Now break out of the Docker container started earlier, and start the container again explicitly specifying a syslog logging driver to the host where you have enabled the syslog listener on 514/udp.

[Shell]
$ sudo docker run -p 8080:8080 –log-driver=syslog –log-opt syslog-address=udp://192.168.1.2:514 –log-opt syslog-facility=daemon –log-opt syslog-format=rfc5424 –log-opt tag=”{{.ImageName}}/{{.ID}}” gs-rest-service
[/Shell]

The ‘Docker logs’ command does not have the ability to show you these network logs, but you should have received logs from netcat or ‘/var/log/syslog’ that looks like:

[RAW]

<30>1 2017-03-20T22:00:00-05:00 hpi5 gs-rest-service/78eab1df792a 26022 gs-rest-service/78eab1df792a 03:00:00.806 [http-nio-8080-exec-3] DEBUG hello.GreetingController – debug message slf4j
<30>1 2017-03-20T22:00:00-05:00 hpi5 gs-rest-service/78eab1df792a 26022 gs-rest-service/78eab1df792a 03:00:00.807 [http-nio-8080-exec-3] INFO hello.GreetingController – info message slf4j

[/RAW]

Summary

The overarching idea is that these container log messages are now being directed by the person deploying the Docker container, and not the developer who packages the application.

This loose coupling makes the logging implementation an orthogonal concern that can be centrally operated for production environments (e.g. ELK), yet simple to debug during the development lifecycle.

REFERENCES

https://spring.io/guides/gs/spring-boot-docker/

https://docs.docker.com/engine/installation/linux/ubuntu/#install-from-a-package

https://docs.docker.com/engine/admin/logging/overview/

https://docs.docker.com/engine/admin/logging/log_tags/

https://github.com/docker/docker/issues/12377

https://geowarin.github.io/spring-boot-app-in-docker-image.html

https://dzone.com/articles/running-spring-boot-in-a-docker-container

http://containertutorials.com/docker-compose/spring-boot-app.html

https://github.com/Yelp/dumb-init

https://dmp.fabric8.io/

https://github.com/fabric8io/docker-maven-plugin