HAProxy: Using HAProxy for SSL termination on Ubuntu

HAProxy is a high performance TCP/HTTP (Level 4 and Level 7) load balancer and reverse proxy.  A common pattern is allowing HAProxy to be the fronting SSL-termination point, and then HAProxy determines which pooled backend server serves the request.

Continue reading “HAProxy: Using HAProxy for SSL termination on Ubuntu”

Nginx: Using Nginx for SSL termination on Ubuntu

Nginx is a popular reverse proxy and load balancer that focuses on level 7 (application) traffic.  A common pattern is allowing Nginx to be the fronting SSL-termination point, and then Nginx determines which pooled backend server is best available to serve the request.

Continue reading “Nginx: Using Nginx for SSL termination on Ubuntu”

Apache2: Enable LDAP authentication and SSL termination for Ubuntu

Some web applications leave authentication as an orthogonal concern to the application – not including any kind of login functionality and instead leaving authentication as an operational concern.

When this happens, a reverse proxy that has an LDAP integration can act as an architectural sentry in front of the web application and also fulfills the requirements for Single Sign-On.  Apache2 serves this purpose very well with minimal overhead.

Continue reading “Apache2: Enable LDAP authentication and SSL termination for Ubuntu”

Ubuntu: Creating a self-signed certificate using OpenSSL on Ubuntu

There are numerous articles I’ve written  where a self-signed certificate is a prerequisite for deploying a piece of infrastructure.

Here are the quick steps for installing a self-signed certificate on an Ubuntu server.

Some of you will want a full explanation of the steps required, others will only want to run the script I’m putting on github.

Continue reading “Ubuntu: Creating a self-signed certificate using OpenSSL on Ubuntu”

Jenkins: Setting up a continuous integration server on Ubuntu

Jenkins is the open-source automation server that is critical in building a continuous integration and delivery pipeline.  It is extensible and has a wealth of plugins that  integrate with numerous enterprise systems.

Here are the detailed steps for installing a Jenkins server on Ubuntu.

Continue reading “Jenkins: Setting up a continuous integration server on Ubuntu”

Monitoring: Java JMX exploration from the console using jmxterm

Java JMX (Java Management Extensions) is a standardized way of monitoring Java based applications.  The managed resources (MBeans) are defined and exposed by the JVM, application server, and application – and offer a view into these layers that can provide invaluable monitoring data.

But in order to report back the JMX data you must know the fully expanded path of the MBean and it’s available attributes/operations.  If you are on a desktop, tools like jsonsole provide a nice GUI interface for drilling down into the MBean hierarchy.  But, if you are in a server environment and JMX is not enabled for remote access on a desktop, you may need a console alternative.

An open-source project call jmxterm comes packaged as a single uber jar that makes it easy to enumerate and explore the available MBean exposed in a Java based application.

Continue reading “Monitoring: Java JMX exploration from the console using jmxterm”

Ubuntu: Using strace to get a view into file and network activity of a process

strace is a handy utility for tracing system, file, and network calls on a Linux system.  It can produce trace output for either an already running process, or it can create a new process.

Some of the most common troubleshooting scenarios are needing to isolate either the network or file system activity of a process.  For example to determine whether an application was attempting to reaching out to a server on the expected port, or to understand why a startup configuration file was not being read from the expected directory.

Continue reading “Ubuntu: Using strace to get a view into file and network activity of a process”

Ubuntu: Using tcpdump for analysis of network traffic and port usage

tcpdump comes standard on Ubuntu servers and is an invaluable tool in determining traffic coming in and out of a host.

As network infrastructures have become more complex and security conscious, validating network flow from client hosts through potentially multiple proxies and ultimately to a destination host and port has become more important than ever.

Let me list a few of the more common use cases.

Continue reading “Ubuntu: Using tcpdump for analysis of network traffic and port usage”

Nginx: Custom access log format and error levels

Nginx is a powerful application level proxy server.  Whether for troubleshooting or analysis, enabling log levels and custom formats for the access/error logs is a common requirement.

Error Logs

By default, only messages in the error category are logged.  If you want to enable more details, then modify nginx.conf like:

error_log file [level]

Enabling debug level on Linux would usually look like:

error_log /var/log/nginx/error.log debug;

Access Logs

Access logs and their format are also customized in nginx.conf.  By default, if no format is specified then the combined format is used.

access_log file [format]

Continue reading “Nginx: Custom access log format and error levels”

PingIdentity: Disabling SSLv3 and weak ciphers for PingFederate

The PingFederate server provides best-in-class Identity Management and SSO.  However, due to US laws governing export of cryptography, the default SSL protocols and cipher suites need to be configured to harden the solution.

Below are the steps involved with making these post-installation changes.

Continue reading “PingIdentity: Disabling SSLv3 and weak ciphers for PingFederate”

AppDynamics: Java Spring PetClinic and PostgreSQL configured for monitoring

As an exploration of AppDynamics’ APM functionality, you may find it useful to deploy a sample application that can quickly return back useful data.  The Java Spring PetClinic connecting back to a PostgreSQL database provides a simple code base that exercises both database and application monitoring.

In a previous article, I went over the detailed steps for monitoring PetClinic with a MySQL backend, so I will refer back to that article for some of the details and will focus on the PostgreSQL specific steps here.

Continue reading “AppDynamics: Java Spring PetClinic and PostgreSQL configured for monitoring”

OpenSSL: Using OpenSSL to enumerate protocols and ciphers in use by web applications

While enabling HTTPS is a important step in securing your web application, it is critical that you also take steps to disable legacy protocols and low strength ciphers that can circumvent the very security you are attempting to implement.

As long as you have the latest version of openssl then you should be able to use a bash script like below (credit for this script goes here) to enumerate every matching protocol and cipher that a server is exposing:

Continue reading “OpenSSL: Using OpenSSL to enumerate protocols and ciphers in use by web applications”

Ubuntu: Silent package installation and debconf

If you have worked on deploying packages via apt-get, you are probably familiar with a couple of forms of interruption during the package installation and upgrade process.

The first is the text menu shown during package upgrades that informs you that a new configuration file is available and asks if you want to keep your current one, use the new one from the package maintainer, or show the difference.

The second is the occasional ASCII dialog that interrupts the install/upgrade and ask for essential information before moving forward.  The screenshot below is the dialog you get when installing MySQL or MariaDB, asking to set the initial root password for the database.

The problem, in this age of cloud scale, is that you often need completely silent installations and upgrades that can be pushed out via Configuration Management.  Even if this is a build for an immutable image, you would prefer a completely automated construction process instead of manual intervention each time you build an image.

Continue reading “Ubuntu: Silent package installation and debconf”

AppDynamics: Installing a Machine Agent on Ubuntu 14.04

The AppDynamics Machine Agent is used not only to report back on basic hardware metrics (cpu/memory/disk/network), but also as the hook for custom plugins that can report back on any number of applications including: .NET, Apache, AWS, MongoDB, Cassandra, and many others.

In this article, I’ll go over the details to install the Machine Agent unto an Ubuntu 14.04 system.

Continue reading “AppDynamics: Installing a Machine Agent on Ubuntu 14.04”

Grafana: Installation on Ubuntu 14.04

Grafana is an open-source visualization suite that is able to generate graphs and dashboards, in addition to alerting.

It is designed to retrieve data from various backends including: Graphite, ElasticSearch, Prometheus, and Zabbix.

This article will lead you through an installation of the latest stable version on Ubuntu 14.04.

Continue reading “Grafana: Installation on Ubuntu 14.04”

Unbutu: Removing unused kernel images and headers

As part of normal long-term operations, the number of kernel images on your system will accumulate and take up disk space.  This issue with space will be even more pronounced if /boot is mounted to its own smaller partition.

With Ubuntu 16.04, ‘apt autoremove –purge’ and configuration of the unattended upgrades can ensure that old kernel images are cleaned, but if you are using Ubuntu 14.04 or need to manually purge, then the instructions below can lead you through the process.

Before removing this unnecessary baggage, the first step is to check what kernel version is currently being used and the installation state.

> uname -r
4.4.0-57-generic

Continue reading “Unbutu: Removing unused kernel images and headers”

Ubuntu: Unattended Upgrades for security patches

ubuntuIf you are running an Ubuntu server for any extended period of time, security issues will arise that affect the kernel, distribution, or packages installed on that host.

While there are always minimal risks associated with automatically applying security fixes, I feel those are dwarfed by the risks of running hosts that have known security flaws.  For example, a media frenzy over the OpenSSL vulnerability Heartbleed may have forced administrators the world over to go out and manually patch their fleet of Linux hosts, but the truth is there is a constant stream of public vulnerabilities.

Expecting system administrators to manually patch each of these (in addition to their other daily tasks) is unrealistic, and therefore Ubuntu provides a simple way of scheduling unattended security updates.

First, install the unattended-upgrades package:

> sudo apt install unattended-upgrades

Continue reading “Ubuntu: Unattended Upgrades for security patches”

SaltStack: Running a masterless minion on Ubuntu

saltstack_logo-thumbnailIt may be hard to imagine on the development side, but there are instances where a deployed host is not accessible from the Salt Master in a production environment.  This forces a bit of creativity if you have a set of standard formulas you need to apply to the host.

For instance, imagine a host sitting in a highly restricted DMZ network. Even with the advent of Salt SSH for minionless administration, SSH access may only be opened from a jumpbox and not the Salt Master itself.  In cases like this, a Masterless Minion is a viable alternative.

Continue reading “SaltStack: Running a masterless minion on Ubuntu”

AppDynamics: Silent Install of Controller on Ubuntu and license directory

appdynamics_logo_gray_rgb-170924For full instructions on installing the AppDynamics Controller on Linux, see the official documentation.  However, when you get to the step for installing in silent mode, it can be confusing because although it shows you how to specify the path to a response file and the keys available, it does not give you a sample file.

./controller_64bit_linux.sh -q -c -varfile /home/user/response.varfile

One way to generate a sample file that matches the responses you want in production is to manually install the controller in a development environment first. If you run the installer:

Continue reading “AppDynamics: Silent Install of Controller on Ubuntu and license directory”

Ubuntu: Determine system vulnerability for Dirty COW CVE-2016-5195

ubuntuThe Dirty COW vulnerability affects the kernel of most base Ubuntu versions.  Especially when running an Ubutu HWE stack, it can be a bit confusing to determine if your kernel and Ubuntu version are affected.

If you need to validate patching, then you can use a simple C program to exercise this read-only write vulnerability and check your system.

Continue reading “Ubuntu: Determine system vulnerability for Dirty COW CVE-2016-5195”

SaltStack: Keeping Salt Pillar data encrypted using GPG

saltstack_logo-thumbnailWhen automating software and infrastructure, it is not uncommon to need to supply a user id and password for installation or other operations.  While it is certainly possible to pass these plaintext credentials directly in the state, this is not best practice.

# not best practice!!!

testdb_user:
  mysql_user.present:
    - name: frank
    - password: "test3rdb"
    - host: localhost

There are several issues with this approach.

Continue reading “SaltStack: Keeping Salt Pillar data encrypted using GPG”

Ubuntu: Using Fiddler to analyze Chrome/Firefox network capture

ubuntuThe prevalence of the long chains of firewall and reverse proxy solutions present in production infrastructure (and made even more popular with the dynamic routing introduced with containers) has made analysis of the end-user side of the network exchange a critical tool in troubleshooting.

Fiddler has long been a solid tool for both proxy capture as well as analysis of the end user application traffic on the Windows platform.  However, troubleshooting issues with customers always required them to first install the tool on their desktop, and at times corporate policies would prevent installation.

Now, with the built-in capabilities of the Chrome DevTools and Firefox Network Monitor, the capture can happen directly from the end user’s browser without any external tool installation.  If that session needs to be analyzed by higher level support resources, it can be exported as an HTTP Archive (HAR), and then imported into Fiddler for analysis at a later time.

And since the release of Fiddler for Linux, the analysis of the HAR can be done directly on the Ubuntu desktop.

Continue reading “Ubuntu: Using Fiddler to analyze Chrome/Firefox network capture”

Ubuntu: HWE Hardware Enablement Stacks, LTS, and the Kernel

ubuntuIf you installed (or upgraded to) a later Ubuntu point release:  >= 12.04.2, >=14.04.2, or >=16.04.2, you may now be wondering why the system is warning you upon every login that you will no longer receive security updates.

WARNING: Security updates for your current Hardware Enablement Stack ended on 2016-08-04:
 * http://wiki.ubuntu.com/1404_HWE_EOL

Although the first point releases of an Ubuntu version 12.04.0 and 12.04.1, 14.04.0 and 14.04.1, and 16.04.0 and 16.04.1 maintain support of their kernel version until the standard 5 year End-Of-Life for that long-term release (LTS), subsequent point releases do not hold the same schedule.

14-04-x-ubuntu-kernel-support-scheduleThe reason why is that subsequent point releases ship with an updated kernel and X stack that require upgrade in order to maintain support. Referring to the support schedule above as an example, you can see that 14.04.3 was released with the Wily 15.04 Vivid HWE stack, and only supported for 12 months before requiring an upgrade to 14.04.5 and the Xenial 16.04 HWE.

Continue reading “Ubuntu: HWE Hardware Enablement Stacks, LTS, and the Kernel”

Ubuntu: Simulating a Web Server using Netcat

ubuntuWhen tasked with deploying a web application and it is not responsive to your browser requests, sometimes you need to take a step back from the complexity of your full stack and run a quick sanity check.

You can use netcat as a simple web server to prove to yourself that the network infrastructure is allowing the traffic, the guest OS is not blocking the port with its own firewall, and the browser can receive the HTTP response.

Start the netcat HTTP Server

If you want to refer back to my post on the minimal TCP server using netcat, read here.  Extending that concept, here is the bash command to echo out a basic set of HTTP headers and body on port 8080:

while true; do { echo -e "HTTP/1.1 200 OK\r\n$(date)\r\n\r\n<h1>hello world from $(hostname) on $(date)</h1>" |  nc -vl 8080; } done

Continue reading “Ubuntu: Simulating a Web Server using Netcat”

Ubuntu: Pre-Validate Network ACL and Firewall Connectivity with Netcat

ubuntuAlthough virtualization has pushed a self-service culture for infrastructure, it is still common in production environments to need your  Network Operations team to open the required ports necessary for any new application deployment.

So, while you may be able to create the base virtualized host, you can’t go much further without the network connectivity.  And there is nothing worse than finding out half way through your full stack deployment that the reason you keep hitting errors is because a stray port was not opened.

I would suggest pre-validating all the TCP and UDP ports you expect open.  This can be done pretty simply by using netcat on both sides of the communication.

Continue reading “Ubuntu: Pre-Validate Network ACL and Firewall Connectivity with Netcat”

SaltStack: Troubleshooting Basic Network Connectivity of Minion on Ubuntu

saltstack_logo-thumbnailWhen troubleshooting basic connectivity from your SaltStack minions to your Salt master, the first thing to remember is the basic flow – the minions initiate the connection to port 4505/4506 on the Salt master.

With this in mind, if you have modified /etc/salt/minion so that the master is explicitly set and logs are set to debug levels as shown below:

master: mysaltmaster
log_level_logfile: debug

And the minion key is still not showing up on the Salt master list (salt-key -L), and the minion log file (/var/log/salt/minion) is not providing any hints, you should try a basic network connectivity test using netcat.  From the console of the Salt minion:

Continue reading “SaltStack: Troubleshooting Basic Network Connectivity of Minion on Ubuntu”

Ubuntu: Ignoring Transitive Trust Domains when using Samba/Winbind

ubuntuIf your Ubuntu host is authenticating against an Active Directory Domain Controller, you may find there are multiple subdomains or transitive trusts visible.  Which is not a problem in most cases – but if your host is in a subnet where a connection to these other subdomain or transitive trust domains is not possible, you can experience long delays until a timeout period is reached by the SMB client.

To get a list of all the visible domains, including transitive trusts:

wbinfo -m

Continue reading “Ubuntu: Ignoring Transitive Trust Domains when using Samba/Winbind”

OpenWrt: Flashing Linksys WRT1X00AC/S from USB-TTL Using Ubuntu

openwrt_logoFlashing the firmware of the Linksys WRT1X00AC/S is well documented on the OpenWrt wiki.  So I don’t feel the need to go over the architectural concepts in this article, but I did want to provide instructions for the Ubuntu specific tools you can use to flash the firmware.

If you want to try flashing to OpenWrt using the factory LinkSys ‘Router Firmware Update’ feature, that is your choice, but it really is working blind and you have no ability to fix issues if something goes wrong.  After bricking my router once, I now rely solely on the Serial to USB-TTL cable which is the highly recommended connectivity method from the OpenWrt page.

Step 1. Connect via USB-TTL cable

I wrote a detailed article about using the Adafruit USB TTL Serial cable to connect to the Linksys WRT1X00AC/S for an Ubuntu host.

After powering off/on the router, you should be able to clearly the see the boot sequence of your Linksys firmware in your terminal program.  Below is a snippet of the output showing the Linksys logo in ASCII art which scrolls by as the router brings up all its services.

linksys_factor_booting2

Continue reading “OpenWrt: Flashing Linksys WRT1X00AC/S from USB-TTL Using Ubuntu”

Ubuntu: Enabling the Ubuntu universe Repository

ubuntuThere are four main repositories for Ubuntu: Main, Universe, Restricted, and Multiverse.  The Ubuntu CD contains the packages from the Main and Restricted repositories, so even if you do not have an Internet connections those will be available.

However, if you have booted from the LiveCD, and did not initially configure a wired or wireless network connection, then the ‘Universe’ repository will not be enabled.

If you were trying to install a package such as putty and the Universe repository source was disabled, you would get ‘E: Unable to locate package’ responses when trying to install and an empty response from apt-cache when searching for this package:

Continue reading “Ubuntu: Enabling the Ubuntu universe Repository”

OpenWrt: Installing a TFTP Server on Ubuntu for OpenWrt Firmware Updates

openwrt_logoThe Trivial File Transfer Protocol (TFTP) is an extremely simple protocol most often used for network booting strategies, such as PXE and flashing OpenWrt images unto consumer routers.

I go over full instructions for flashing OpenWrt using Ubuntu and flashing a sysupgrade in another post, this article will focus specifically on setting up a tftp server daemon on Ubuntu that can be used to serve the binary image file.

Installation

First, install the tftp server and client packages:

# apt-get install tftpd-hpa tftp-hpa -y

Continue reading “OpenWrt: Installing a TFTP Server on Ubuntu for OpenWrt Firmware Updates”