Docker log collection can be done using various methods, one method that is particularly effective is having a dedicated container whose sole purpose is to automatically sense other deployed containers and aggregate their log events.
This is the architectural model of logspout, an open-source project that acts as a router for the stdout/stderr logs of other containers.
If you do not have docker installed yet, see my article here. Before moving on, you should be able to run the hello-world container.
Syslog listener
Although logspout has third-party modules for sending log events to Kafka, Redis, Logstash, and Gelf, the easiest way to illustrate its functionality in this article is to send them to syslog.
For simplicity, we will send the logs to the syslog port of the parent Docker host. And to make it easy, instead of configuring the rsyslog service on our Ubuntu host, we will just use the standard utility netcat to listen on the syslog 514 tcp port and echo any data received.
First we ensure the rsyslog service is disabled, and start our TCP listener on port 514:
$ sudo service rsyslog stop
$ sudo ufw allow 514
$ sudo sh -c "while true; do { nc -vl 514; } done"
Listening on [0.0.0.0] (family 0, port 514)
Then we run a quick sanity test from another console on the Docker host that sends a message to this TCP server (which happens to be 192.168.1.4 on my system)
$ nc -w0 192.168.1.4 514 <<< "<14>User Info msg from `hostname` on `date`"
And then you will see a message at the pseudo syslog server console:
<14>User Info msg from mydockerhost on Mon Apr 17 18:57:04 CDT 2017
This proves out our “syslog server” which is where logspout will be output all its messages.
Logspout container
Now we want to start the logspout container, which grabs the logs from all deployed containers and will send all events to our listening syslog server.
$ sudo docker pull gliderlabs/logspout:latest $ sudo docker run \ --volume=/var/run/docker.sock:/var/run/docker.sock \ gliderlabs/logspout \ syslog+tcp://192.168.1.4:514
If you need to ignore certain images by name/tag, that is possible usingthe LOGSPOUT and EXCLUDE_LABEL environment variables.
Validate simple container
Now let’s check if the hello-world container will now send logs through the logspout container, and subsequently to our syslog console.
$ sudo docker run hello-world
Hello from Docker!
This message shows that your installation appears to be working correctly.
To generate this message, Docker took the following steps:
1. The Docker client contacted the Docker daemon.
2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
3. The Docker daemon created a new container from that image which runs the
executable that produces the output you are currently reading.
4. The Docker daemon streamed that output to the Docker client, which sent it
to your terminal.
To try something more ambitious, you can run an Ubuntu container with:
$ docker run -it ubuntu bash
Share images, automate workflows, and more with a free Docker ID:
https://cloud.docker.com/
For more examples and ideas, visit:
https://docs.docker.com/engine/userguide/
Which shows the typical output to the console where it is invoked. Now let’s check the container id and name (“b2e3f870ddf7” and “happy_sammet”)
$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES b2e3f870ddf7 hello-world "/hello" About a minute ago Exited (0) About a minute ago happy_sammet
Now, if we look at the console of the syslog server we see these stdout lines all converted to syslog messages with priority, timestamp, docker container id, container name, and message.
1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - Hello from Docker! 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - This message shows that your installation appears to be working correctly. 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - To generate this message, Docker took the following steps: 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - 1. The Docker client contacted the Docker daemon. 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - 3. The Docker daemon created a new container from that image which runs the 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - executable that produces the output you are currently reading. 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - 4. The Docker daemon streamed that output to the Docker client, which sent it 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - to your terminal. 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - To try something more ambitious, you can run an Ubuntu container with: 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - $ docker run -it ubuntu bash 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - Share images, automate workflows, and more with a free Docker ID: 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - https://cloud.docker.com/ 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - For more examples and ideas, visit: 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - - https://docs.docker.com/engine/userguide/ 1 2017-04-18T00:34:13Z b2e3f870ddf7 happy_sammet 32295 - -
Another example
The whalesay container is another popular example, let’s instantiate it with our own custom message of ‘YAYYYYY!’
$ sudo docker run docker/whalesay cowsay YAYYYYYY!
sudo docker run docker/whalesay cowsay YAYYYYYY!
___________
< YAYYYYYY! >
-----------
\
\
\
## .
## ## ## ==
## ## ## ## ===
/""""""""""""""""___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\______/
Checking the docker container info:
$ sudo docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c3c5223b2441 docker/whalesay "cowsay YAYYYYYY!" 2 minutes ago Exited (0) 2 minutes ago goofy_ardinghelli
And once again, you should see the syslog messages using the “c3c5223b2441” container id and “goofy_ardinghelli” name.
REFERENCES
https://github.com/gliderlabs/logspout
https://docs.logentries.com/docs/logspout-docker-container
http://gliderlabs.com/devlog/2015/new-logspout-extensible-docker-logging/
https://hub.docker.com/r/gliderlabs/logspout/~/dockerfile/
https://docs.docker.com/engine/getstarted/step_three/#step-2-run-the-whalesay-image
http://gliderlabs.com/devlog/2015/new-logspout-extensible-docker-logging/