Ubuntu: Testing the official released kernel patches for Meltdown CVE-2017-5754

ubuntuThe Meltdown vulnerability affects Intel and some ARM (but not AMD) processor chips and can allow unprivileged access to memory in the kernel and other processes.

Canonical has committed to kernel patches to address this issue and they are now available from the both the updates and security official Ubuntu repositories.

In this article, I’ll step through patching an Ubuntu kernel with the candidate kernel fixes.

Continue reading “Ubuntu: Testing the official released kernel patches for Meltdown CVE-2017-5754”

Ubuntu: Testing the first candidate kernel patches for Meltdown CVE-2017-5754

ubuntuThe Meltdown vulnerability affects Intel and some ARM (but not AMD) processor chips and can allow unprivileged access to memory in the kernel and other processes.

Canonical has committed to kernel patches to address this issue by January 9, 2018 and the first candidate kernel patches have now been released for Xenial and Trusty LTS.

UPDATE Jan 11 2018: The main Ubuntu repositories now have the official patches.  Read my article here for more information.

In this article, I’ll step through patching an Ubuntu 16.04 kernel with the candidate kernel fixes.

Continue reading “Ubuntu: Testing the first candidate kernel patches for Meltdown CVE-2017-5754”

Ubuntu: Testing the KAISER kernel patch for Meltdown CVE-2017-5754

ubuntuThe Meltdown vulnerability affects Intel and some ARM (but not AMD) processor chips and can allow unprivileged access to memory in the kernel and other processes.  Canonical has committed to kernel patches to address this issue by January 9, 2018.

A paper coming out of Graz University of Technology in Austria and written by Daniel Gruss, Moritz Lipp, Michael Schwarz, Richard Fellner, Clementine Maurice, and Stefan Mangard provides a patched 4.10.0 kernel that isolates the kernel address space and resolves CVE-2017-5754 (Meltdown).

No one is advocating this as the fix for your production instances, but if you want to play around with this patched kernel in a virtualized environment, I’ll lead you through the steps in this article.

UPDATE Jan 11 2018: The main Ubuntu repositories now have the official patches.  Read my article here for more information.

Continue reading “Ubuntu: Testing the KAISER kernel patch for Meltdown CVE-2017-5754”

Ubuntu: Determine system vulnerability for Meltdown CVE-2017-5754

ubuntuThe Meltdown vulnerability affects Intel and some ARM (but not AMD) processor chips and can allow unprivileged access to memory in the kernel and other processes.  Canonical has committed to kernel patches to address this issue by January 9, 2018.

If you need to check your system, or perhaps have already patched your systems but want to verify that the issue truly is resolved, there is a proof of concept available on github that exercises a rogue data cache load (Variant 3).

In this article I will show you how to compile and run this non-destructive C++ program on Ubuntu 14.04 and 16.04.

Continue reading “Ubuntu: Determine system vulnerability for Meltdown CVE-2017-5754”

Ubuntu: Determine system vulnerability for Spectre CVE-2017-5715 CVE-2017-5753

ubuntuThe Spectre vulnerability affects Intel, AMD, and ARM processor chips (each to various degrees) and can allow unprivileged access to memory in the kernel and other processes.  Canonical has committed to kernel patches to address this issue by January 9, 2018.

If you need to check your system, or perhaps have already patched your systems but want to verify that the issue truly is resolved, there is a simple proof of concept that exercises the bounds check bypass within the same process (Variant 1, CVE-2017-5753).

In this article I will show you how to compile and run this small, non-destructive C program that is included as Appendix A in the Spectre whitepaper.

Continue reading “Ubuntu: Determine system vulnerability for Spectre CVE-2017-5715 CVE-2017-5753”

CloudFoundry: Enabling Java JMX/RMI access for remote containers

Enabling the use of real-time JVM monitoring tools like jconsole and VisualVM can be extremely beneficial when troubleshooting issues.  These tools work by enabling a JMX/RMI communication channel to the JVM.

These are typically thought of as local development tools, but they can also be used on remote CF containers running Java.  In this article, I’ll show you how to enable JMX/RMI to your remote Cloud Foundry container.

Continue reading “CloudFoundry: Enabling Java JMX/RMI access for remote containers”

CloudFoundry: Java thread and heap dump analysis on remote containers

Java thread and heap dumps are valuable tools for troubleshooting local development,  but they can also be used on remote CF containers running a JVM.  In this article, we’ll go through various method of gathering this data from a Cloud Foundry container and then tools for analyzing this data.

Now matter how uniform your environments, whether using Cloud Foundry stemcells/containers, configuration management tools,  or Docker images, there are always real-world issues that show up only in certain environments (especially production!).  There are unique corner cases that get exposed by end user experimentation, unexpected thread locking,  generational memory issues,  etc… and thread and heap dump analysis tools can assist.

Continue reading “CloudFoundry: Java thread and heap dump analysis on remote containers”

CloudFoundry: Enabling Java remote debugging with Eclipse

Remote debugging of Java applications from an IDE can be essential when debugging difficult issues.  There is no reason to give this functionality up just because you are deploying to a container in Cloud Foundry.

In this article we’ll go over how to enable remote debugging from a local Eclipse IDE to a public CF provider like Pivotal CloudFoundry.

Continue reading “CloudFoundry: Enabling Java remote debugging with Eclipse”

CloudFoundry: Monitoring the spring-music webapp, Part 5

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale. This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example.

This article is Part 5 of  a series on Cloud Foundry concepts:

In this particular article, we will look at application level monitoring of CF deployed applications using the New Relic Service Broker.  The New Relic product enables real-time monitoring of applications.

Continue reading “CloudFoundry: Monitoring the spring-music webapp, Part 5”

CloudFoundry: Logging for the spring-music webapp, Part 4

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example.

This article is Part 4 of  a series on Cloud Foundry concepts:

In this particular article, we will look at the Cloud Foundry log types, how to configure logback for spring-music, and then how to inject those events into a log pipeline.

Continue reading “CloudFoundry: Logging for the spring-music webapp, Part 4”

CloudFoundry: Exploring Cloud Foundry using the spring-music application

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  It supports multiple infrastructure platforms (EC2, VMware, OpenStack), and is able to standardize deployment, logging,  scaling, and routing in a way that is friendly to a continuous delivery pipeline.

In this series of articles, we will use the spring-music web application to explore Cloud Foundry features and concepts.

CloudFoundry: Scaling the spring-music webapp, Part 3

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example.

This article is Part 3 of  a series on Cloud Foundry concepts:

Specifically in this article, we will horizontally and vertically scale up the spring-music application and show how this affects the routing and logging.

Continue reading “CloudFoundry: Scaling the spring-music webapp, Part 3”

CloudFoundry: Persisting spring-music data using Postgres service, Part 2

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  This article is part of a series that explores different facets of a Cloud Foundry deployment using the spring-music project as an example.

This article is Part 2 of  a series on Cloud Foundry concepts:

In this particular article, we will create a Cloud Foundry Postgres service to externalize the persistent store instead of using the default in-memory H2 database which is destroyed every time the application is restarted or restaged.

Continue reading “CloudFoundry: Persisting spring-music data using Postgres service, Part 2”

CloudFoundry: PCF Dev for local development on Ubuntu

PCF Dev is a distribution of Cloud Foundry that has a minimal footprint and is designed to run locally on a developer’s machine.  Using this lightweight distribution of Cloud Foundry, a developer can debug and deploy applications locally.

In this article, we’ll go through the installation of PCF Dev on an Ubuntu development host.

Continue reading “CloudFoundry: PCF Dev for local development on Ubuntu”

CloudFoundry: Deploying the spring-music webapp, Part 1

Cloud Foundry is an opinionated Platform-as-a-Service that allows you to manage applications at scale.  It supports multiple infrastructure platforms, and is able to standardize deployment, logging,  scaling, and routing in a way friendly to a continuous delivery pipeline.

This article is Part 1 of  a series on Cloud Foundry concepts:

In this particular article, we will install the command line interface for Cloud Foundry on Ubuntu and then use that to deploy the Spring Boot based spring-music project to a CF provider.

Continue reading “CloudFoundry: Deploying the spring-music webapp, Part 1”

HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 16.04 with Systemd

The reload functionality in HAProxy till now has always been “not perfect but good enough”, perhaps dropping a few connections under heavy load but within parameters everyone was willing to accept. And because of the potential impact, a reload was typically only done during non-peak traffic times.

But with the popularity of microservices, containerization, continuous deployment, and dynamically scalable architecture, it has become critical for our load balancers to provide zero downtime reloads because reloading can potentially happen every few seconds even during peak production load.

There have been some seminal pieces written on how to achieve this level of availability with HAProxy. Yelp Engineering wrote up how to use qdiscs to delay the SYN packets, then followed up with using a combination of Nginx and HAProxy communicating over unix sockets. An alternative solution used two instances of HAProxy with an iptables flip.

But now with the ability in HAProxy 1.8 to pass listening sockets from the old process, along with Linux kernel 3.9 support of SO_REUSEPORT we finally have a solution that doesn’t feel like an ingenious hack of the Linux kernel and networking stack.

Continue reading “HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 16.04 with Systemd”

HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 14.04

The reload functionality in HAProxy till now has always been “not perfect but good enough”, perhaps dropping a few connections under heavy load but within parameters everyone was willing to accept. And because of the potential impact, a reload was typically only done during non-peak traffic times.

But with the popularity of microservices, containerization, continuous deployment, and dynamically scalable architecture, it has become critical for our load balancers to provide zero downtime reloads because reloading can potentially happen every few seconds even during peak production load.

There have been some seminal pieces written on how to achieve this level of availability with HAProxy.  Yelp Engineering wrote up how to use qdiscs to delay the SYN packets, then followed up with using a combination of Nginx and HAProxy communicating over unix sockets. An alternative solution used two instances of HAProxy with an iptables flip.

But now with the ability in HAProxy 1.8 to pass listening sockets from the old process, along with Linux kernel 3.9 support of SO_REUSEPORT we finally have a solution that doesn’t feel like an ingenious hack of the Linux kernel and networking stack.

Continue reading “HAProxy: Zero downtime reloads with HAProxy 1.8 on Ubuntu 14.04”

SaltStack: Installing an older Salt Master or Minion for compatibility

If your Salt Minion version is too far removed from the Salt Master version, you may find yourself with unexplained errors.

This problem can be faced when the OS template you are deploying was packaged years earlier with an older Salt minion while the Salt Master has been kept up to date.

But it can also happen with a relatively recent version Master like 2016.11, if you use the latest 2017.7 Minion which has major changes in the fileclient.

In this article I will show you how to use apt-get to install an earlier version of the Salt Master or Salt Minion.

Continue reading “SaltStack: Installing an older Salt Master or Minion for compatibility”

Windows: Windows 2012 Sysprep for Vagrant readiness

Many developers like to use Vagrant from HashiCorp to standardize the workflow of virtual machines: creation, running, destroying, taking snapshots, etc..

Usually Vagrant is used for Linux hosts, but it also works with Windows as long as you prepare the template properly.

In a previous article I went over the detailed steps to create a template image for Windows 2012 server using Sysprep.  Consider this the second part in that series, where Vagrant has specific additional requirements.

Continue reading “Windows: Windows 2012 Sysprep for Vagrant readiness”

Ubuntu: Standing up a Windows 2012 instance on Ubuntu using Sysprep

In the world of Linux containers where deployment takes on the order of seconds, even the best-case scenario for spinning up a new Windows host can seem like an eternity.

Clearly, you don’t want to wait for the entire Windows install process each time you bring up a Windows guest OS.  Even automated, this would take 15+ minutes and all it would deliver is a base, non-patched, non-customized system.

Windows Sysprep allows you to build a base Windows template with any patches, customizations, and files that you want in a base system.  And then any subsequent guest OS created with that template will inherit all those template basics.

I wrote this article to give developers a peek into how these templates are created so they can influence the base images that their Operations teams generate.

Continue reading “Ubuntu: Standing up a Windows 2012 instance on Ubuntu using Sysprep”

Ubuntu: Installing the Genymotion Android emulator

Android is one of the leading platforms of the mobile industry.  By installing an Android emulator on your Ubuntu desktop, you can bring this power to your desktop.

More often than not, an Android emulator is used for custom development of mobile apps, but don’t overlook its utility as a way to access your favorite mobile applications directly from your desktop, or as a way to preview upcoming Android releases.

Continue reading “Ubuntu: Installing the Genymotion Android emulator”

Ubuntu: Installing Tor on Ubuntu 14.04 and 16.04

The Tor project is free software that helps protect your privacy by making it difficult for a 3rd party to analyze your network requests or link your traffic back to your network access point.  See the Tor overview page for reasons why this may be important to world citizens, corporations, or specific professions.

Simplified, this is done by using a large pool of distributed hosts and using varied and encrypted paths through these hosts to deliver your original request.

Be aware that no one is saying Tor provides fullproof anonymity on the internet, there are documented weaknesses [1,2,3].  But by now, it should be clear the security exists on a spectrum and not in absolute terms.

I will detail how to install both the Tor service and Tor browser which is designed to address the most common threats to remaining anonymous while browsing.

Continue reading “Ubuntu: Installing Tor on Ubuntu 14.04 and 16.04”

Ubuntu: Testing authenticated SMTP over TLS/SSL

SMTP mail relays exposed to the internet typically use a combination of SSL and authenticated SMTP to avoid abuse by malicious actors.

This is an excellent choice from a security perspective, but makes smoke testing a bit more complex than just opening telnet.

Continue reading “Ubuntu: Testing authenticated SMTP over TLS/SSL”

Ansible: Installing Ansible on Ubuntu 16.04

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

In this article I’ll describe how to deploy the latest release of Ansible using pip on Ubuntu 16.04, and then perform a quick validation against a client.

Continue reading “Ansible: Installing Ansible on Ubuntu 16.04”

Ansible: Managing a Windows host using Ansible

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

Ansible was started as a Linux only solution, leveraging ssh to provide a management channel to a target server.  However, starting at Ansible 1.7, support for Windows hosts was added by using Powershell remoting over WinRM.

Continue reading “Ansible: Managing a Windows host using Ansible”

Ansible: Installing Ansible on Ubuntu 14.04

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

In this article I’ll describe how to deploy the latest release of Ansible using pip on Ubuntu 14.04, and then perform a quick validation against a client.

Continue reading “Ansible: Installing Ansible on Ubuntu 14.04”

Zabbix: LLD low-level discovery returning multiple values

Zabbix low-level discovery (LLD) provides a way to create an array of related items, triggers, or graphs without needing to know the exact number of entities up front.

The easiest way to populate the keys of a discovery item is to add a “UserParameter” in zabbix_agentd.conf, and then the Zabbix agent will  invokes a script which returns the set of keys.

But the keys are only the first part of a real solution, because what you really want to send back are the values associated with those keys.  For example, if you are monitoring a database, you don’t want to just send the list of tables available, you may want to send back each table name and then its row count and size on disk.

Unfortunately Zabbix does not support sending back multiple values [1,2,3,4].  There are various workarounds such as using one UserParameter for the discovery key and another with a UserParameter=key[*] to fetch each row of data, or using vfs.file.regexp to parse values that have been written to a file.

But I think the cleanest solution, and one that requires the minimal number of spawned processes on the agent host is to invoke zabbix_sender from inside the script to send back all the values you want to populate.

Continue reading “Zabbix: LLD low-level discovery returning multiple values”

Docker: Visualizing image hierarchy and container dependency using dockviz

The Docker console commands for listing and viewing containers and images (ps, images, history, inspect) provides a wealth of information, but when you are managing hundreds of containers, a graph view of the container inventory and their dependencies can be critical for operations.

Dockviz can help you visualize your containers and images by creating an PNG image representing the container links and image lineage.

Continue reading “Docker: Visualizing image hierarchy and container dependency using dockviz”

GoLang: Running a Go binary as a systemd service on Ubuntu 16.04

The Go language with its simplicity, concurrency support,  rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.

However, the Go language does not natively provide a reliable way to daemonize itself.  In this article I will describe how to take a couple of simple Go language programs and run them using a systemd service file that starts them at boot time on Ubuntu 16.04.

Continue reading “GoLang: Running a Go binary as a systemd service on Ubuntu 16.04”