Ubuntu: Testing the official released kernel patches for Meltdown CVE-2017-5754

ubuntuThe Meltdown vulnerability affects Intel and some ARM (but not AMD) processor chips and can allow unprivileged access to memory in the kernel and other processes.

Canonical has committed to kernel patches to address this issue and they are now available from the both the updates and security official Ubuntu repositories.

In this article, I’ll step through patching an Ubuntu kernel with the candidate kernel fixes.

Continue reading “Ubuntu: Testing the official released kernel patches for Meltdown CVE-2017-5754”

Ubuntu: Testing the first candidate kernel patches for Meltdown CVE-2017-5754

ubuntuThe Meltdown vulnerability affects Intel and some ARM (but not AMD) processor chips and can allow unprivileged access to memory in the kernel and other processes.

Canonical has committed to kernel patches to address this issue by January 9, 2018 and the first candidate kernel patches have now been released for Xenial and Trusty LTS.

UPDATE Jan 11 2018: The main Ubuntu repositories now have the official patches.  Read my article here for more information.

In this article, I’ll step through patching an Ubuntu 16.04 kernel with the candidate kernel fixes.

Continue reading “Ubuntu: Testing the first candidate kernel patches for Meltdown CVE-2017-5754”

Ubuntu: Testing the KAISER kernel patch for Meltdown CVE-2017-5754

ubuntuThe Meltdown vulnerability affects Intel and some ARM (but not AMD) processor chips and can allow unprivileged access to memory in the kernel and other processes.  Canonical has committed to kernel patches to address this issue by January 9, 2018.

A paper coming out of Graz University of Technology in Austria and written by Daniel Gruss, Moritz Lipp, Michael Schwarz, Richard Fellner, Clementine Maurice, and Stefan Mangard provides a patched 4.10.0 kernel that isolates the kernel address space and resolves CVE-2017-5754 (Meltdown).

No one is advocating this as the fix for your production instances, but if you want to play around with this patched kernel in a virtualized environment, I’ll lead you through the steps in this article.

UPDATE Jan 11 2018: The main Ubuntu repositories now have the official patches.  Read my article here for more information.

Continue reading “Ubuntu: Testing the KAISER kernel patch for Meltdown CVE-2017-5754”

Ubuntu: Determine system vulnerability for Meltdown CVE-2017-5754

ubuntuThe Meltdown vulnerability affects Intel and some ARM (but not AMD) processor chips and can allow unprivileged access to memory in the kernel and other processes.  Canonical has committed to kernel patches to address this issue by January 9, 2018.

If you need to check your system, or perhaps have already patched your systems but want to verify that the issue truly is resolved, there is a proof of concept available on github that exercises a rogue data cache load (Variant 3).

In this article I will show you how to compile and run this non-destructive C++ program on Ubuntu 14.04 and 16.04.

Continue reading “Ubuntu: Determine system vulnerability for Meltdown CVE-2017-5754”

Ubuntu: Determine system vulnerability for Spectre CVE-2017-5715 CVE-2017-5753

ubuntuThe Spectre vulnerability affects Intel, AMD, and ARM processor chips (each to various degrees) and can allow unprivileged access to memory in the kernel and other processes.  Canonical has committed to kernel patches to address this issue by January 9, 2018.

If you need to check your system, or perhaps have already patched your systems but want to verify that the issue truly is resolved, there is a simple proof of concept that exercises the bounds check bypass within the same process (Variant 1, CVE-2017-5753).

In this article I will show you how to compile and run this small, non-destructive C program that is included as Appendix A in the Spectre whitepaper.

Continue reading “Ubuntu: Determine system vulnerability for Spectre CVE-2017-5715 CVE-2017-5753”

HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 16.04 with Systemd

The reload functionality in HAProxy till now has always been “not perfect but good enough”, perhaps dropping a few connections under heavy load but within parameters everyone was willing to accept. And because of the potential impact, a reload was typically only done during non-peak traffic times.

But with the popularity of microservices, containerization, continuous deployment, and dynamically scalable architecture, it has become critical for our load balancers to provide zero downtime reloads because reloading can potentially happen every few seconds even during peak production load.

There have been some seminal pieces written on how to achieve this level of availability with HAProxy. Yelp Engineering wrote up how to use qdiscs to delay the SYN packets, then followed up with using a combination of Nginx and HAProxy communicating over unix sockets. An alternative solution used two instances of HAProxy with an iptables flip.

But now with the ability in HAProxy 1.8 to pass listening sockets from the old process, along with Linux kernel 3.9 support of SO_REUSEPORT we finally have a solution that doesn’t feel like an ingenious hack of the Linux kernel and networking stack.

Continue reading “HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 16.04 with Systemd”

HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 14.04

The reload functionality in HAProxy till now has always been “not perfect but good enough”, perhaps dropping a few connections under heavy load but within parameters everyone was willing to accept. And because of the potential impact, a reload was typically only done during non-peak traffic times.

But with the popularity of microservices, containerization, continuous deployment, and dynamically scalable architecture, it has become critical for our load balancers to provide zero downtime reloads because reloading can potentially happen every few seconds even during peak production load.

There have been some seminal pieces written on how to achieve this level of availability with HAProxy.  Yelp Engineering wrote up how to use qdiscs to delay the SYN packets, then followed up with using a combination of Nginx and HAProxy communicating over unix sockets. An alternative solution used two instances of HAProxy with an iptables flip.

But now with the ability in HAProxy 1.8 to pass listening sockets from the old process, along with Linux kernel 3.9 support of SO_REUSEPORT we finally have a solution that doesn’t feel like an ingenious hack of the Linux kernel and networking stack.

Continue reading “HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 14.04”

SaltStack: Installing an older Salt Master or Minion for compatibility

If your Salt Minion version is too far removed from the Salt Master version, you may find yourself with unexplained errors.

This problem can be faced when the OS template you are deploying was packaged years earlier with an older Salt minion while the Salt Master has been kept up to date.

But it can also happen with a relatively recent version Master like 2016.11, if you use the latest 2017.7 Minion which has major changes in the fileclient.

In this article I will show you how to use apt-get to install an earlier version of the Salt Master or Salt Minion.

Continue reading “SaltStack: Installing an older Salt Master or Minion for compatibility”

Windows: Windows 2012 Sysprep for Vagrant readiness

Many developers like to use Vagrant from HashiCorp to standardize the workflow of virtual machines: creation, running, destroying, taking snapshots, etc..

Usually Vagrant is used for Linux hosts, but it also works with Windows as long as you prepare the template properly.

In a previous article I went over the detailed steps to create a template image for Windows 2012 server using Sysprep.  Consider this the second part in that series, where Vagrant has specific additional requirements.

Continue reading “Windows: Windows 2012 Sysprep for Vagrant readiness”

Ubuntu: Standing up a Windows 2012 instance on Ubuntu using Sysprep

In the world of Linux containers where deployment takes on the order of seconds, even the best-case scenario for spinning up a new Windows host can seem like an eternity.

Clearly, you don’t want to wait for the entire Windows install process each time you bring up a Windows guest OS.  Even automated, this would take 15+ minutes and all it would deliver is a base, non-patched, non-customized system.

Windows Sysprep allows you to build a base Windows template with any patches, customizations, and files that you want in a base system.  And then any subsequent guest OS created with that template will inherit all those template basics.

I wrote this article to give developers a peek into how these templates are created so they can influence the base images that their Operations teams generate.

Continue reading “Ubuntu: Standing up a Windows 2012 instance on Ubuntu using Sysprep”

Ubuntu: Installing the Genymotion Android emulator

Android is one of the leading platforms of the mobile industry.  By installing an Android emulator on your Ubuntu desktop, you can bring this power to your desktop.

More often than not, an Android emulator is used for custom development of mobile apps, but don’t overlook its utility as a way to access your favorite mobile applications directly from your desktop, or as a way to preview upcoming Android releases.

Continue reading “Ubuntu: Installing the Genymotion Android emulator”

Ubuntu: Installing Tor on Ubuntu 14.04 and 16.04

The Tor project is free software that helps protect your privacy by making it difficult for a 3rd party to analyze your network requests or link your traffic back to your network access point.  See the Tor overview page for reasons why this may be important to world citizens, corporations, or specific professions.

Simplified, this is done by using a large pool of distributed hosts and using varied and encrypted paths through these hosts to deliver your original request.

Be aware that no one is saying Tor provides fullproof anonymity on the internet, there are documented weaknesses [1,2,3].  But by now, it should be clear the security exists on a spectrum and not in absolute terms.

I will detail how to install both the Tor service and Tor browser which is designed to address the most common threats to remaining anonymous while browsing.

Continue reading “Ubuntu: Installing Tor on Ubuntu 14.04 and 16.04”

Ubuntu: Testing authenticated SMTP over TLS/SSL

SMTP mail relays exposed to the internet typically use a combination of SSL and authenticated SMTP to avoid abuse by malicious actors.

This is an excellent choice from a security perspective, but makes smoke testing a bit more complex than just opening telnet.

Continue reading “Ubuntu: Testing authenticated SMTP over TLS/SSL”

Ansible: Installing Ansible on Ubuntu 16.04

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

In this article I’ll describe how to deploy the latest release of Ansible using pip on Ubuntu 16.04, and then perform a quick validation against a client.

Continue reading “Ansible: Installing Ansible on Ubuntu 16.04”

Ansible: Managing a Windows host using Ansible

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

Ansible was started as a Linux only solution, leveraging ssh to provide a management channel to a target server.  However, starting at Ansible 1.7, support for Windows hosts was added by using Powershell remoting over WinRM.

Continue reading “Ansible: Managing a Windows host using Ansible”

Ansible: Installing Ansible on Ubuntu 14.04

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

In this article I’ll describe how to deploy the latest release of Ansible using pip on Ubuntu 14.04, and then perform a quick validation against a client.

Continue reading “Ansible: Installing Ansible on Ubuntu 14.04”

GoLang: Running a Go binary as a systemd service on Ubuntu 16.04

The Go language with its simplicity, concurrency support,  rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.

However, the Go language does not natively provide a reliable way to daemonize itself.  In this article I will describe how to take a couple of simple Go language programs and run them using a systemd service file that starts them at boot time on Ubuntu 16.04.

Continue reading “GoLang: Running a Go binary as a systemd service on Ubuntu 16.04”

GoLang: Running a Go binary as a SysV service on Ubuntu 14.04

The Go language with its simplicity, concurrency support,  rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.

However, the Go language does not natively provide a reliable way to daemonize itself.  In this article I will describe how to take a couple of simple Go language programs, run them using SystemV init scripts with their own process owner, standard logs, and started at boot time on Ubuntu 14.04.

Continue reading “GoLang: Running a Go binary as a SysV service on Ubuntu 14.04”

GoLang: Cross Compiling for Linux and Windows platforms

A nice feature of the Go language is the ability to build binaries for multiple platforms directly from a single source system.  As an example, even from a development Windows 7 32-bit machine, you can build binaries for both 64 bit Linux and Windows 2012 Servers.

Before Go 1.5, you needed a compiler for the target architecture, but now that the entire tool chain is written in Go, building for multiple architectures is easy.

And unlike other languages where additional external libraries need to be copied or downloaded on the target system, Go dependencies are generally statically linked [1,2,3,4] into a single binary which makes portability that much easier.

Continue reading “GoLang: Cross Compiling for Linux and Windows platforms”

GoLang: Installing the Go Programming language on Ubuntu 14.04

The Go programming language has gotten considerable momentum, and the fact that it compiles down to machine code has made it popular in containers like Docker where a single executable binary fits the execution model perfectly.

This article will detail installation on Ubuntu 14.04 with the standard hello world validation.

Continue reading “GoLang: Installing the Go Programming language on Ubuntu 14.04”

SaltStack: Installing a Salt Master on Ubuntu 14.04

saltstack_logo-thumbnailConfiguration Management tools like SaltStack are invaluable for managing infrastructure at scale.  Even in the growing world of containerization where immutable image deployment is the norm, those images need to be built in a repeatable and auditable fashion.

This article will detail installation of the SaltStack master on Ubuntu 14.04, with validation using a single Minion.  Note that Minion installation is not mandatory if using Salt SSH.

Continue reading “SaltStack: Installing a Salt Master on Ubuntu 14.04”

ELK: Installing Logstash on Ubuntu 14.04

elastic-logstash-fwLogstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch.

Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 2.x and 5.x.

Continue reading “ELK: Installing Logstash on Ubuntu 14.04”

ELK: Running ElastAlert as a service on Ubuntu 14.04

ElastAlert from the Yelp Engineering group provides a very flexible platform for alerting on conditions coming from ElasticSearch.

In a previous article I fully describe running interactively on an Ubuntu server, and now I’ll expand on that by running it at system startup using a System-V init script.

One of the challenges of getting ElastAlert to run as a service is that is has  a very strict set of module requirements that very easily conflicts with other Python applications, and so we will use Python’s virtualenv to build it in isolation and then call that wrapper from the service script.

Continue reading “ELK: Running ElastAlert as a service on Ubuntu 14.04”

ELK: ElastAlert for alerting based on data from ElasticSearch

ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert.

ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.

Continue reading “ELK: ElastAlert for alerting based on data from ElasticSearch”

Docker: Installing Docker CE on Ubuntu 14.04 and 16.04

Docker is a container platform that streamlines software delivery and provides isolation, scalability, and efficiency with less overhead than OS level virtualization.

These instructions are taken directly from the official Docker for Ubuntu page, but I wanted to reiterate those tasks essential for installing the Docker Community Edition on Ubuntu 14.04 and 16.04.

Continue reading “Docker: Installing Docker CE on Ubuntu 14.04 and 16.04”

Squid: Configuring an Ubuntu host to use a Squid proxy for internet access

Once you have a Squid proxy setup as described in my article here, the next challenge is configuring your Ubuntu servers so that they use this proxy by default instead of attempting direct internet connections.

There are several entities we want using Squid by default: apt package manager, interactive consoles and wget/curl, and Java applications.

Continue reading “Squid: Configuring an Ubuntu host to use a Squid proxy for internet access”

Squid: Controlling network access using Squid and whitelisted domains

Having your production servers go through a proxy like Squid for internet access can be an architectural best practice that provides network security as well as caching efficiencies.

For further security, denying access to all requests but an explicit whitelist of domains provides auditable control.

Continue reading “Squid: Controlling network access using Squid and whitelisted domains”