CloudFoundry: Deploying the spring-music webapp, Part 1

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  It supports multiple infrastructure platforms, and is able to standardize deployment, logging,  scaling, and routing in a way friendly to a continuous delivery pipeline.

This article is Part 1 of  a series on Cloud Foundry concepts:

In this particular article, we will install the command line interface for Cloud Foundry on Ubuntu and then use that to deploy the Spring Boot based spring-music project to a CF provider.

Continue reading “CloudFoundry: Deploying the spring-music webapp, Part 1”

HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 16.04 with Systemd

The reload functionality in HAProxy till now has always been “not perfect but good enough”, perhaps dropping a few connections under heavy load but within parameters everyone was willing to accept. And because of the potential impact, a reload was typically only done during non-peak traffic times.

But with the popularity of microservices, containerization, continuous deployment, and dynamically scalable architecture, it has become critical for our load balancers to provide zero downtime reloads because reloading can potentially happen every few seconds even during peak production load.

There have been some seminal pieces written on how to achieve this level of availability with HAProxy. Yelp Engineering wrote up how to use qdiscs to delay the SYN packets, then followed up with using a combination of Nginx and HAProxy communicating over unix sockets. An alternative solution used two instances of HAProxy with an iptables flip.

But now with the ability in HAProxy 1.8 to pass listening sockets from the old process, along with Linux kernel 3.9 support of SO_REUSEPORT we finally have a solution that doesn’t feel like an ingenious hack of the Linux kernel and networking stack.

Continue reading “HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 16.04 with Systemd”

HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 14.04

The reload functionality in HAProxy till now has always been “not perfect but good enough”, perhaps dropping a few connections under heavy load but within parameters everyone was willing to accept. And because of the potential impact, a reload was typically only done during non-peak traffic times.

But with the popularity of microservices, containerization, continuous deployment, and dynamically scalable architecture, it has become critical for our load balancers to provide zero downtime reloads because reloading can potentially happen every few seconds even during peak production load.

There have been some seminal pieces written on how to achieve this level of availability with HAProxy.  Yelp Engineering wrote up how to use qdiscs to delay the SYN packets, then followed up with using a combination of Nginx and HAProxy communicating over unix sockets. An alternative solution used two instances of HAProxy with an iptables flip.

But now with the ability in HAProxy 1.8 to pass listening sockets from the old process, along with Linux kernel 3.9 support of SO_REUSEPORT we finally have a solution that doesn’t feel like an ingenious hack of the Linux kernel and networking stack.

Continue reading “HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 14.04”

SaltStack: Installing an older Salt Master or Minion for compatibility

If your Salt Minion version is too far removed from the Salt Master version, you may find yourself with unexplained errors.

This problem can be faced when the OS template you are deploying was packaged years earlier with an older Salt minion while the Salt Master has been kept up to date.

But it can also happen with a relatively recent version Master like 2016.11, if you use the latest 2017.7 Minion which has major changes in the fileclient.

In this article I will show you how to use apt-get to install an earlier version of the Salt Master or Salt Minion.

Continue reading “SaltStack: Installing an older Salt Master or Minion for compatibility”

Windows: Windows 2012 Sysprep for Vagrant readiness

Many developers like to use Vagrant from HashiCorp to standardize the workflow of virtual machines: creation, running, destroying, taking snapshots, etc..

Usually Vagrant is used for Linux hosts, but it also works with Windows as long as you prepare the template properly.

In a previous article I went over the detailed steps to create a template image for Windows 2012 server using Sysprep.  Consider this the second part in that series, where Vagrant has specific additional requirements.

Continue reading “Windows: Windows 2012 Sysprep for Vagrant readiness”

Ubuntu: Standing up a Windows 2012 instance on Ubuntu using Sysprep

In the world of Linux containers where deployment takes on the order of seconds, even the best-case scenario for spinning up a new Windows host can seem like an eternity.

Clearly, you don’t want to wait for the entire Windows install process each time you bring up a Windows guest OS.  Even automated, this would take 15+ minutes and all it would deliver is a base, non-patched, non-customized system.

Windows Sysprep allows you to build a base Windows template with any patches, customizations, and files that you want in a base system.  And then any subsequent guest OS created with that template will inherit all those template basics.

I wrote this article to give developers a peek into how these templates are created so they can influence the base images that their Operations teams generate.

Continue reading “Ubuntu: Standing up a Windows 2012 instance on Ubuntu using Sysprep”

Ubuntu: Installing the Genymotion Android emulator

Android is one of the leading platforms of the mobile industry.  By installing an Android emulator on your Ubuntu desktop, you can bring this power to your desktop.

More often than not, an Android emulator is used for custom development of mobile apps, but don’t overlook its utility as a way to access your favorite mobile applications directly from your desktop, or as a way to preview upcoming Android releases.

Continue reading “Ubuntu: Installing the Genymotion Android emulator”

Ubuntu: Installing Tor on Ubuntu 14.04 and 16.04

The Tor project is free software that helps protect your privacy by making it difficult for a 3rd party to analyze your network requests or link your traffic back to your network access point.  See the Tor overview page for reasons why this may be important to world citizens, corporations, or specific professions.

Simplified, this is done by using a large pool of distributed hosts and using varied and encrypted paths through these hosts to deliver your original request.

Be aware that no one is saying Tor provides fullproof anonymity on the internet, there are documented weaknesses [1,2,3].  But by now, it should be clear the security exists on a spectrum and not in absolute terms.

I will detail how to install both the Tor service and Tor browser which is designed to address the most common threats to remaining anonymous while browsing.

Continue reading “Ubuntu: Installing Tor on Ubuntu 14.04 and 16.04”

Ubuntu: Testing authenticated SMTP over TLS/SSL

SMTP mail relays exposed to the internet typically use a combination of SSL and authenticated SMTP to avoid abuse by malicious actors.

This is an excellent choice from a security perspective, but makes smoke testing a bit more complex than just opening telnet.

Continue reading “Ubuntu: Testing authenticated SMTP over TLS/SSL”

Ansible: Installing Ansible on Ubuntu 16.04

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

In this article I’ll describe how to deploy the latest release of Ansible using pip on Ubuntu 16.04, and then perform a quick validation against a client.

Continue reading “Ansible: Installing Ansible on Ubuntu 16.04”

Ansible: Managing a Windows host using Ansible

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

Ansible was started as a Linux only solution, leveraging ssh to provide a management channel to a target server.  However, starting at Ansible 1.7, support for Windows hosts was added by using Powershell remoting over WinRM.

Continue reading “Ansible: Managing a Windows host using Ansible”

Ansible: Installing Ansible on Ubuntu 14.04

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

In this article I’ll describe how to deploy the latest release of Ansible using pip on Ubuntu 14.04, and then perform a quick validation against a client.

Continue reading “Ansible: Installing Ansible on Ubuntu 14.04”

GoLang: Running a Go binary as a systemd service on Ubuntu 16.04

The Go language with its simplicity, concurrency support,  rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.

However, the Go language does not natively provide a reliable way to daemonize itself.  In this article I will describe how to take a couple of simple Go language programs and run them using a systemd service file that starts them at boot time on Ubuntu 16.04.

Continue reading “GoLang: Running a Go binary as a systemd service on Ubuntu 16.04”

GoLang: Running a Go binary as a SysV service on Ubuntu 14.04

The Go language with its simplicity, concurrency support,  rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.

However, the Go language does not natively provide a reliable way to daemonize itself.  In this article I will describe how to take a couple of simple Go language programs, run them using SystemV init scripts with their own process owner, standard logs, and started at boot time on Ubuntu 14.04.

Continue reading “GoLang: Running a Go binary as a SysV service on Ubuntu 14.04”

GoLang: Cross Compiling for Linux and Windows platforms

A nice feature of the Go language is the ability to build binaries for multiple platforms directly from a single source system.  As an example, even from a development Windows 7 32-bit machine, you can build binaries for both 64 bit Linux and Windows 2012 Servers.

Before Go 1.5, you needed a compiler for the target architecture, but now that the entire tool chain is written in Go, building for multiple architectures is easy.

And unlike other languages where additional external libraries need to be copied or downloaded on the target system, Go dependencies are generally statically linked [1,2,3,4] into a single binary which makes portability that much easier.

Continue reading “GoLang: Cross Compiling for Linux and Windows platforms”

GoLang: Installing the Go Programming language on Ubuntu 14.04

The Go programming language has gotten considerable momentum, and the fact that it compiles down to machine code has made it popular in containers like Docker where a single executable binary fits the execution model perfectly.

This article will detail installation on Ubuntu 14.04 with the standard hello world validation.

Continue reading “GoLang: Installing the Go Programming language on Ubuntu 14.04”

SaltStack: Installing a Salt Master on Ubuntu 14.04

saltstack_logo-thumbnailConfiguration Management tools like SaltStack are invaluable for managing infrastructure at scale.  Even in the growing world of containerization where immutable image deployment is the norm, those images need to be built in a repeatable and auditable fashion.

This article will detail installation of the SaltStack master on Ubuntu 14.04, with validation using a single Minion.  Note that Minion installation is not mandatory if using Salt SSH.

Continue reading “SaltStack: Installing a Salt Master on Ubuntu 14.04”

ELK: Installing Logstash on Ubuntu 14.04

elastic-logstash-fwLogstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch.

Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 2.x and 5.x.

Continue reading “ELK: Installing Logstash on Ubuntu 14.04”

ELK: Running ElastAlert as a service on Ubuntu 14.04

ElastAlert from the Yelp Engineering group provides a very flexible platform for alerting on conditions coming from ElasticSearch.

In a previous article I fully describe running interactively on an Ubuntu server, and now I’ll expand on that by running it at system startup using a System-V init script.

One of the challenges of getting ElastAlert to run as a service is that is has  a very strict set of module requirements that very easily conflicts with other Python applications, and so we will use Python’s virtualenv to build it in isolation and then call that wrapper from the service script.

Continue reading “ELK: Running ElastAlert as a service on Ubuntu 14.04”

ELK: ElastAlert for alerting based on data from ElasticSearch

ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert.

ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.

Continue reading “ELK: ElastAlert for alerting based on data from ElasticSearch”

Docker: Installing Docker CE on Ubuntu 14.04 and 16.04

Docker is a container platform that streamlines software delivery and provides isolation, scalability, and efficiency with less overhead than OS level virtualization.

These instructions are taken directly from the official Docker for Ubuntu page, but I wanted to reiterate those tasks essential for installing the Docker Community Edition on Ubuntu 14.04 and 16.04.

Continue reading “Docker: Installing Docker CE on Ubuntu 14.04 and 16.04”

Squid: Configuring an Ubuntu host to use a Squid proxy for internet access

Once you have a Squid proxy setup as described in my article here, the next challenge is configuring your Ubuntu servers so that they use this proxy by default instead of attempting direct internet connections.

There are several entities we want using Squid by default: apt package manager, interactive consoles and wget/curl, and Java applications.

Continue reading “Squid: Configuring an Ubuntu host to use a Squid proxy for internet access”

Squid: Controlling network access using Squid and whitelisted domains

Having your production servers go through a proxy like Squid for internet access can be an architectural best practice that provides network security as well as caching efficiencies.

For further security, denying access to all requests but an explicit whitelist of domains provides auditable control.

Continue reading “Squid: Controlling network access using Squid and whitelisted domains”

HAProxy: Using HAProxy for SSL termination on Ubuntu

HAProxy is a high performance TCP/HTTP (Level 4 and Level 7) load balancer and reverse proxy.  A common pattern is allowing HAProxy to be the fronting SSL-termination point, and then HAProxy determines which pooled backend server serves the request.

Continue reading “HAProxy: Using HAProxy for SSL termination on Ubuntu”

Nginx: Using Nginx for SSL termination on Ubuntu

Nginx is a popular reverse proxy and load balancer that focuses on level 7 (application) traffic.  A common pattern is allowing Nginx to be the fronting SSL-termination point, and then Nginx determines which pooled backend server is best available to serve the request.

Continue reading “Nginx: Using Nginx for SSL termination on Ubuntu”

Apache2: Enable LDAP authentication and SSL termination for Ubuntu

Some web applications leave authentication as an orthogonal concern to the application – not including any kind of login functionality and instead leaving authentication as an operational concern.

When this happens, a reverse proxy that has an LDAP integration can act as an architectural sentry in front of the web application and also fulfills the requirements for Single Sign-On.  Apache2 serves this purpose very well with minimal overhead.

Continue reading “Apache2: Enable LDAP authentication and SSL termination for Ubuntu”

Ubuntu: Creating a self-signed certificate using OpenSSL on Ubuntu

There are numerous articles I’ve written  where a self-signed certificate is a prerequisite for deploying a piece of infrastructure.

Here are the quick steps for installing a self-signed certificate on an Ubuntu server.

Some of you will want a full explanation of the steps required, others will only want to run the script I’m putting on github.

Continue reading “Ubuntu: Creating a self-signed certificate using OpenSSL on Ubuntu”