CloudFoundry: Enabling Java JMX/RMI access for remote containers

Enabling the use of real-time JVM monitoring tools like jconsole and VisualVM can be extremely beneficial when troubleshooting issues.  These tools work by enabling a JMX/RMI communication channel to the JVM.

These are typically thought of as local development tools, but they can also be used on remote CF containers running Java.  In this article, I’ll show you how to enable JMX/RMI to your remote Cloud Foundry container.

Continue reading “CloudFoundry: Enabling Java JMX/RMI access for remote containers”

CloudFoundry: Java thread and heap dump analysis on remote containers

Java thread and heap dumps are valuable tools for troubleshooting local development,  but they can also be used on remote CF containers running a JVM.  In this article, we’ll go through various method of gathering this data from a Cloud Foundry container and then tools for analyzing this data.

Now matter how uniform your environments, whether using Cloud Foundry stemcells/containers, configuration management tools,  or Docker images, there are always real-world issues that show up only in certain environments (especially production!).  There are unique corner cases that get exposed by end user experimentation, unexpected thread locking,  generational memory issues,  etc… and thread and heap dump analysis tools can assist.

Continue reading “CloudFoundry: Java thread and heap dump analysis on remote containers”

CloudFoundry: Enabling Java remote debugging with Eclipse

Remote debugging of Java applications from an IDE can be essential when debugging difficult issues.  There is no reason to give this functionality up just because you are deploying to a container in Cloud Foundry.

In this article we’ll go over how to enable remote debugging from a local Eclipse IDE to a public CF provider like Pivotal CloudFoundry.

Continue reading “CloudFoundry: Enabling Java remote debugging with Eclipse”

CloudFoundry: Monitoring the spring-music webapp, Part 5

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale. This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example.

This article is Part 5 of  a series on Cloud Foundry concepts:

In this particular article, we will look at application level monitoring of CF deployed applications using the New Relic Service Broker.  The New Relic product enables real-time monitoring of applications.

Continue reading “CloudFoundry: Monitoring the spring-music webapp, Part 5”

CloudFoundry: Logging for the spring-music webapp, Part 4

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example.

This article is Part 4 of  a series on Cloud Foundry concepts:

In this particular article, we will look at the Cloud Foundry log types, how to configure logback for spring-music, and then how to inject those events into a log pipeline.

Continue reading “CloudFoundry: Logging for the spring-music webapp, Part 4”

CloudFoundry: Exploring Cloud Foundry using the spring-music application

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  It supports multiple infrastructure platforms (EC2, VMware, OpenStack), and is able to standardize deployment, logging,  scaling, and routing in a way that is friendly to a continuous delivery pipeline.

In this series of articles, we will use the spring-music web application to explore Cloud Foundry features and concepts.

CloudFoundry: Scaling the spring-music webapp, Part 3

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example.

This article is Part 3 of  a series on Cloud Foundry concepts:

Specifically in this article, we will horizontally and vertically scale up the spring-music application and show how this affects the routing and logging.

Continue reading “CloudFoundry: Scaling the spring-music webapp, Part 3”

CloudFoundry: Persisting spring-music data using Postgres service, Part 2

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example.

This article is Part 2 of  a series on Cloud Foundry concepts:

In this particular article, we will create a Cloud Foundry Postgres service to externalize the persistent store instead of using the default in-memory H2 database which is destroyed every time the application is restarted or restaged.

Continue reading “CloudFoundry: Persisting spring-music data using Postgres service, Part 2”

CloudFoundry: PCF Dev for local development on Ubuntu

PCF Dev is a distribution of Cloud Foundry that has a minimal footprint and is designed to run locally on a developer’s machine.  Using this lightweight distribution of Cloud Foundry, a developer can debug and deploy applications locally.

In this article, we’ll go through the installation of PCF Dev on an Ubuntu development host.

Continue reading “CloudFoundry: PCF Dev for local development on Ubuntu”

CloudFoundry: Deploying the spring-music webapp, Part 1

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  It supports multiple infrastructure platforms, and is able to standardize deployment, logging,  scaling, and routing in a way friendly to a continuous delivery pipeline.

This article is Part 1 of  a series on Cloud Foundry concepts:

In this particular article, we will install the command line interface for Cloud Foundry on Ubuntu and then use that to deploy the Spring Boot based spring-music project to a CF provider.

Continue reading “CloudFoundry: Deploying the spring-music webapp, Part 1”

HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 16.04 with Systemd

The reload functionality in HAProxy till now has always been “not perfect but good enough”, perhaps dropping a few connections under heavy load but within parameters everyone was willing to accept. And because of the potential impact, a reload was typically only done during non-peak traffic times.

But with the popularity of microservices, containerization, continuous deployment, and dynamically scalable architecture, it has become critical for our load balancers to provide zero downtime reloads because reloading can potentially happen every few seconds even during peak production load.

There have been some seminal pieces written on how to achieve this level of availability with HAProxy. Yelp Engineering wrote up how to use qdiscs to delay the SYN packets, then followed up with using a combination of Nginx and HAProxy communicating over unix sockets. An alternative solution used two instances of HAProxy with an iptables flip.

But now with the ability in HAProxy 1.8 to pass listening sockets from the old process, along with Linux kernel 3.9 support of SO_REUSEPORT we finally have a solution that doesn’t feel like an ingenious hack of the Linux kernel and networking stack.

Continue reading “HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 16.04 with Systemd”

HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 14.04

The reload functionality in HAProxy till now has always been “not perfect but good enough”, perhaps dropping a few connections under heavy load but within parameters everyone was willing to accept. And because of the potential impact, a reload was typically only done during non-peak traffic times.

But with the popularity of microservices, containerization, continuous deployment, and dynamically scalable architecture, it has become critical for our load balancers to provide zero downtime reloads because reloading can potentially happen every few seconds even during peak production load.

There have been some seminal pieces written on how to achieve this level of availability with HAProxy.  Yelp Engineering wrote up how to use qdiscs to delay the SYN packets, then followed up with using a combination of Nginx and HAProxy communicating over unix sockets. An alternative solution used two instances of HAProxy with an iptables flip.

But now with the ability in HAProxy 1.8 to pass listening sockets from the old process, along with Linux kernel 3.9 support of SO_REUSEPORT we finally have a solution that doesn’t feel like an ingenious hack of the Linux kernel and networking stack.

Continue reading “HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 14.04”

Docker: Visualizing image hierarchy and container dependency using dockviz

The Docker console commands for listing and viewing containers and images (ps, images, history, inspect) provides a wealth of information, but when you are managing hundreds of containers, a graph view of the container inventory and their dependencies can be critical for operations.

Dockviz can help you visualize your containers and images by creating an PNG image representing the container links and image lineage.

Continue reading “Docker: Visualizing image hierarchy and container dependency using dockviz”

Docker: logspout for Docker log collection

Docker log collection can be done using various methods, one method that is particularly effective is having a dedicated container whose sole purpose is to automatically sense other deployed containers and aggregate their log events.

This is the architectural model of logspout, an open-source project that acts as a router for the stdout/stderr logs of other containers.

If you do not have docker installed yet, see my article here.  Before moving on, you should be able to run the hello-world container.

Continue reading “Docker: logspout for Docker log collection”

Docker: Sending Spring Boot logging to syslog

Building services using Spring Boot gives a development team a jump start on many production concerns, including logging.  But unlike a standard deployment where logging to a local file is where the developer’s responsibility typically ends, with Docker we must think about how to log to a public space outside our ephemeral container space.

The Docker logging drivers capture all the output from a container’s stdout/stderr, and can send a container’s logs directly to most major logging solutions (syslog, Logstash, gelf, fluentd).

As an added benefit, by making the logging implementation a runtime choice for the container, it provides flexibility to use a simpler implementation during development but a highly-available, scalable logging solution in production.

Continue reading “Docker: Sending Spring Boot logging to syslog”

Docker: Installing Docker CE on Ubuntu 14.04 and 16.04

Docker is a container platform that streamlines software delivery and provides isolation, scalability, and efficiency with less overhead than OS level virtualization.

These instructions are taken directly from the official Docker for Ubuntu page, but I wanted to reiterate those tasks essential for installing the Docker Community Edition on Ubuntu 14.04 and 16.04.

Continue reading “Docker: Installing Docker CE on Ubuntu 14.04 and 16.04”