Syslog: Sending Java log4j2 to rsyslog on Ubuntu

log4j-logoLogging has always been a critical part of application development.  But the rise of OS virtualization, applications containers, and cloud-scale logging solutions has turned logging into something bigger that managing local debug files.

Modern applications and services are now expected to feed log aggregation and analysis stacks (ELK, Graylog, Loggly, Splunk, etc).  This can be done a multitude of ways, in this post I want to focus on modifying log4j2 so that it sends directly to an rsyslog server.

Even though we focus on sending to an Ubuntu ryslog server in this post, this could be any entity listening for syslog traffic, such as Logstash.

Continue reading “Syslog: Sending Java log4j2 to rsyslog on Ubuntu”

SaltStack: Creating a Custom Grain using Python

saltstack_logo-thumbnailSaltStack grains are used for relatively static information such as operating system, IP address, and other system properties.  They are also useful for targeting minions, for example whether a system is part of  dev/test/prod, or a flag on whether it falls under LifeScience or HIPAA regulation.

In this article we will implement a custom grain that determines whether a host is part of development, test, or production environment based on a simplistic naming scheme.   This custom grain will be written in Python.

Continue reading “SaltStack: Creating a Custom Grain using Python”

Ubuntu: Using Fiddler to analyze Chrome/Firefox network capture

ubuntuThe prevalence of the long chains of firewall and reverse proxy solutions present in production infrastructure (and made even more popular with the dynamic routing introduced with containers) has made analysis of the end-user side of the network exchange a critical tool in troubleshooting.

Fiddler has long been a solid tool for both proxy capture as well as analysis of the end user application traffic on the Windows platform.  However, troubleshooting issues with customers always required them to first install the tool on their desktop, and at times corporate policies would prevent installation.

Now, with the built-in capabilities of the Chrome DevTools and Firefox Network Monitor, the capture can happen directly from the end user’s browser without any external tool installation.  If that session needs to be analyzed by higher level support resources, it can be exported as an HTTP Archive (HAR), and then imported into Fiddler for analysis at a later time.

And since the release of Fiddler for Linux, the analysis of the HAR can be done directly on the Ubuntu desktop.

Continue reading “Ubuntu: Using Fiddler to analyze Chrome/Firefox network capture”

Ubuntu: HWE Hardware Enablement Stacks, LTS, and the Kernel

ubuntuIf you installed (or upgraded to) a later Ubuntu point release:  >= 12.04.2, >=14.04.2, or >=16.04.2, you may now be wondering why the system is warning you upon every login that you will no longer receive security updates.

WARNING: Security updates for your current Hardware Enablement Stack ended on 2016-08-04:

Although the first point releases of an Ubuntu version 12.04.0 and 12.04.1, 14.04.0 and 14.04.1, and 16.04.0 and 16.04.1 maintain support of their kernel version until the standard 5 year End-Of-Life for that long-term release (LTS), subsequent point releases do not hold the same schedule.

14-04-x-ubuntu-kernel-support-scheduleThe reason why is that subsequent point releases ship with an updated kernel and X stack that require upgrade in order to maintain support. Referring to the support schedule above as an example, you can see that 14.04.3 was released with the Wily 15.04 Vivid HWE stack, and only supported for 12 months before requiring an upgrade to 14.04.5 and the Xenial 16.04 HWE.

Continue reading “Ubuntu: HWE Hardware Enablement Stacks, LTS, and the Kernel”

Node.js: Packaging modules for offline deployment using npm-bundle

nodejs-logoIn a production environment, it is common to have restricted internet access on the production deployment hosts.  This means that using the standard ‘npm install’ and pulling modules from the repository is not an option.

Given the breadth of the dependency graph required for most modules, this packaging is something you want automated without needing to modify the package.json file by hand.

After various failed attempts at: using npmbox, scripts wrapping up ‘npm pack’, and archiving the entire node_modules directory – the npm-bundle module finally provided a proper solution.

Continue reading “Node.js: Packaging modules for offline deployment using npm-bundle”

Ubuntu: Simulating a Web Server using Netcat

ubuntuWhen tasked with deploying a web application and it is not responsive to your browser requests, sometimes you need to take a step back from the complexity of your full stack and run a quick sanity check.

You can use netcat as a simple web server to prove to yourself that the network infrastructure is allowing the traffic, the guest OS is not blocking the port with its own firewall, and the browser can receive the HTTP response.

Start the netcat HTTP Server

If you want to refer back to my post on the minimal TCP server using netcat, read here.  Extending that concept, here is the bash command to echo out a basic set of HTTP headers and body on port 8080:

while true; do { echo -e "HTTP/1.1 200 OK\r\n$(date)\r\n\r\n<h1>hello world from $(hostname) on $(date)</h1>" |  nc -vl 8080; } done

Continue reading “Ubuntu: Simulating a Web Server using Netcat”

Ubuntu: Pre-Validate Network ACL and Firewall Connectivity with Netcat

ubuntuAlthough virtualization has pushed a self-service culture for infrastructure, it is still common in production environments to need your  Network Operations team to open the required ports necessary for any new application deployment.

So, while you may be able to create the base virtualized host, you can’t go much further without the network connectivity.  And there is nothing worse than finding out half way through your full stack deployment that the reason you keep hitting errors is because a stray port was not opened.

I would suggest pre-validating all the TCP and UDP ports you expect open.  This can be done pretty simply by using netcat on both sides of the communication.

Continue reading “Ubuntu: Pre-Validate Network ACL and Firewall Connectivity with Netcat”

SaltStack: Troubleshooting Basic Network Connectivity of Minion on Ubuntu

saltstack_logo-thumbnailWhen troubleshooting basic connectivity from your SaltStack minions to your Salt master, the first thing to remember is the basic flow – the minions initiate the connection to port 4505/4506 on the Salt master.

With this in mind, if you have modified /etc/salt/minion so that the master is explicitly set and logs are set to debug levels as shown below:

master: mysaltmaster
log_level_logfile: debug

And the minion key is still not showing up on the Salt master list (salt-key -L), and the minion log file (/var/log/salt/minion) is not providing any hints, you should try a basic network connectivity test using netcat.  From the console of the Salt minion:

Continue reading “SaltStack: Troubleshooting Basic Network Connectivity of Minion on Ubuntu”

Ubuntu: Ignoring Transitive Trust Domains when using Samba/Winbind

ubuntuIf your Ubuntu host is authenticating against an Active Directory Domain Controller, you may find there are multiple subdomains or transitive trusts visible.  Which is not a problem in most cases – but if your host is in a subnet where a connection to these other subdomain or transitive trust domains is not possible, you can experience long delays until a timeout period is reached by the SMB client.

To get a list of all the visible domains, including transitive trusts:

wbinfo -m

Continue reading “Ubuntu: Ignoring Transitive Trust Domains when using Samba/Winbind”

OpenWrt: Enabling HTTPS for the LuCI Web Admin Interface

openwrt_logoBy default, LuCI, the web admin interface for OpenWrt is not HTTPS enabled.  This may not be a critical issue for you since it is a LAN facing service, but the type of infrastructure information being exchanged combined with the fact that it is usually accessed over WiFi protocols might make you want to consider it – especially considering it is a 5 minute fix.

First connect to OpenWrt either via ssh with Dropbear, or via the USB-TTL cable and a terminal program.  Install the following packages:

# opkg update
# opkg install luci-lib-px5g px5g-standalone libustream-openssl
# opkg install luci

Continue reading “OpenWrt: Enabling HTTPS for the LuCI Web Admin Interface”

OpenWrt: Flashing Linksys WRT1X00AC/S from USB-TTL Using Ubuntu

openwrt_logoFlashing the firmware of the Linksys WRT1X00AC/S is well documented on the OpenWrt wiki.  So I don’t feel the need to go over the architectural concepts in this article, but I did want to provide instructions for the Ubuntu specific tools you can use to flash the firmware.

If you want to try flashing to OpenWrt using the factory LinkSys ‘Router Firmware Update’ feature, that is your choice, but it really is working blind and you have no ability to fix issues if something goes wrong.  After bricking my router once, I now rely solely on the Serial to USB-TTL cable which is the highly recommended connectivity method from the OpenWrt page.

Step 1. Connect via USB-TTL cable

I wrote a detailed article about using the Adafruit USB TTL Serial cable to connect to the Linksys WRT1X00AC/S for an Ubuntu host.

After powering off/on the router, you should be able to clearly the see the boot sequence of your Linksys firmware in your terminal program.  Below is a snippet of the output showing the Linksys logo in ASCII art which scrolls by as the router brings up all its services.


Continue reading “OpenWrt: Flashing Linksys WRT1X00AC/S from USB-TTL Using Ubuntu”

OpenWrt: Installing LuCI Web Interface after Deploying latest OpenWrt Image

openwrt_logoThe stable OpenWrt images are built with LuCI, an OpenWrt web administration interface.  But if you are using the bleeding edge or trunk OpenWrt images, then you won’t get this package.

Luckily, it is not difficult to add the LuCI package to the install.  As long as you have Dropbear enabled for ssh access, or you are connected via UBS-TTL and have shell access to your router then it only takes a few commands.

opkg update
opkg install luci
/etc/init.d/uhttpd enable

Continue reading “OpenWrt: Installing LuCI Web Interface after Deploying latest OpenWrt Image”

Ubuntu: Enabling the Ubuntu universe Repository

ubuntuThere are four main repositories for Ubuntu: Main, Universe, Restricted, and Multiverse.  The Ubuntu CD contains the packages from the Main and Restricted repositories, so even if you do not have an Internet connections those will be available.

However, if you have booted from the LiveCD, and did not initially configure a wired or wireless network connection, then the ‘Universe’ repository will not be enabled.

If you were trying to install a package such as putty and the Universe repository source was disabled, you would get ‘E: Unable to locate package’ responses when trying to install and an empty response from apt-cache when searching for this package:

Continue reading “Ubuntu: Enabling the Ubuntu universe Repository”

OpenWrt: Installing a TFTP Server on Ubuntu for OpenWrt Firmware Updates

openwrt_logoThe Trivial File Transfer Protocol (TFTP) is an extremely simple protocol most often used for network booting strategies, such as PXE and flashing OpenWrt images unto consumer routers.

I go over full instructions for flashing OpenWrt using Ubuntu and flashing a sysupgrade in another post, this article will focus specifically on setting up a tftp server daemon on Ubuntu that can be used to serve the binary image file.


First, install the tftp server and client packages:

# apt-get install tftpd-hpa tftp-hpa -y

Continue reading “OpenWrt: Installing a TFTP Server on Ubuntu for OpenWrt Firmware Updates”

SaltStack: Validating States of Minion without Execution

saltstack_logo-thumbnailBefore running state.apply against a minion, especially in a production environment, a good sanity test can be to list the states that will be executed without actually running those states.

This can be done by adding tests=True to the end of the state command. For example, to check all the states that will be applied to a minion:

salt 'myminion' state.apply tests=True

Or to check which states would be run for the apache formula:

salt 'myminion' state.sls apache tests=True


GIT: Calling git Clone Using Password with Special Character

gitlogoIt is more popular to use an ssh key instead of a password when automating a git clone from a guest OS.  But if you do need to specify the password directly into the console command, it takes this form:

$ git clone https://<user>:<password>@<gitserver>/<path>/<repo>.git

Which works fine if the password is plaintext, but if it has special characters like an exclamation mark, you need to use percent encoding which is often called URL encoding.

Continue reading “GIT: Calling git Clone Using Password with Special Character”

Ubuntu: Hang While Installing gutenprint as Network Driver

If you experience hanging when installing the gutenprint drivers for a network printer from the desktop, try manually installing the gutenprint drivers from the console first.

Most likely, you will see a screen like below, and the progress bar will continually cycle but never end.


If you can’t cancel, you can use the ‘xkill’ command from the console and click on the dialog window.  But you will also need to kill the process, and that can be done by  finding the process id using:

Continue reading “Ubuntu: Hang While Installing gutenprint as Network Driver”

Ubuntu: Installing Packages without Public Internet Access

ubuntuIn production data centers, it is not uncommon to have limited public internet access due to security policies.  So while running ‘apt-get’ or adding a repository to sources.list is easy in your development lab, you have to figure out an alternative installation strategy because you need a process that looks the same across both development and production.

For some, building containers or images will satisfy this requirement.  The container/image can be built once in development, and transferred as an immutable entity to production.

But for those that use automated configuration management such as Salt/Chef/Ansible/Puppet to layer components on top of a base image inside a restricted environment, there is a need to get binary packages to these guest OS without requiring public internet access.

There are several approaches that could be taken: using an offline repository or a tool such as Synaptic or Keryx or apt-mirror, but in this post I’ll go over using apt-get on an internet connected source machine to download the  necessary packages for Apache2, and then running dpkg on the non-connected target machine to install each required .deb package and get a running instance of Apache2.

Note that this solution only addresses the apt packages.  If you need to pull down Javascript packages from npm or Python modules from pypi,  then you might want to look at my article on using a squid proxy to whitelist specific URL.

Continue reading “Ubuntu: Installing Packages without Public Internet Access”

Ubuntu: Extending a virtualized disk when using LVM

ubuntuIt is common for a virtualized Guest OS base image to have a generic minimal storage capacity.  But this capacity can easily be exceeded by production scenarios, performance testing, logging, or even the general cruft of running a machine 24×7.

In a previous post, I described extending a virtualized disk when using classic partitions.  In this post, I will perform the same task but with an LVM enabled system.  We will use console level tools so that it could be done from a remote terminal or by automation.

Continue reading “Ubuntu: Extending a virtualized disk when using LVM”

Ubuntu: Creating a Samba/CIFS share to quickly share files with Windows

ubuntuWe live in a multi-platform world, and the ability to easily share folders of content between users in the same protected network is a function made very convenient in the Windows world with CIFS shares (e.g. \\mydesktop\sharedfolder).

Luckily for Ubuntu users, it is pretty easy to setup CIFS shares to offer that same interoperability with Windows hosts on your network.  Start by installing the Samba components.

apt-get install samba -y

Continue reading “Ubuntu: Creating a Samba/CIFS share to quickly share files with Windows”

vRealize Log Insight: Creating your own content pack for field extraction

vmware_logo Content Packs are plugins that allow you you to create pre-packaged knowledge about specific event types.

For example, you can create a content pack that knows how to extract fields from one of your custom log sources.  Beyond extracted fields, you can also add saved queries, aggregations, alerts, dashboards, and visualizations.

Incoming Events from Agent

First, let’s examine our sample log file on the agent side, in a file named /tmp/test.log.

2016-07-14 22:04:13.233 INFO      - [  150] 200

Continue reading “vRealize Log Insight: Creating your own content pack for field extraction”

OpenWrt: Use setenv firmwareName for newer versions of Linksys WRT1900AC/S

openwrt_logoWhen flashing an OpenWrt image to your newer versioned WRT1900AC/S, be aware that instead of using ‘setenv firmware_name’, you should instead use ‘setenv firmwareName’.

The command will not fail, but the router will not understand that it should look for a non-default name for the image and your tftp transfer will fail.

This change appears to have been made between WRT1900AC V1 and WRT1900AC V2.  So, for the latest versions such as WRT1900ACS, be sure to use ‘setenv firmwareName’.

Ubuntu: Serial level access to your Linksys WRT1X00AC/S

ubuntuWhether you are updating the official LinkSys router firmware or taking it a step further and installing open-source firware like OpenWrt, serial level access to your Linksys router is the most dependable way of guaranteeing a connection.

And if you have tried to flash the firmware via the web admin interface and after a reboot you cannot get web access again, then you have no choice.  You have to be able to plug directly into the router’s serial interface and troubleshoot.

Continue reading “Ubuntu: Serial level access to your Linksys WRT1X00AC/S”

Ubuntu: Extending a virtualized disk using fdisk when not using LVM

ubuntuIt is common for a virtualized Guest OS base image to have a generic minimal storage capacity.  But this capacity can easily be exceeded by production scenarios, performance testing, logging, or even the general cruft of running a machine 24×7.

For this reason, extending a virtualized disk can be extremely helpful.  Here is a walk through for extending a disk using fdisk on an Ubuntu system that is using classic partitions.  For performing this operation with LVM enabled, see my post here.

This type of change is typically made with a live CD to ensure exclusive disk access and gparted GUI for convenience.  But we will use fdisk here so that it could be done from a remote terminal or by automation.

Continue reading “Ubuntu: Extending a virtualized disk using fdisk when not using LVM”

Logstash: Using metrics to debug the filtering process

elastic-logstash-fw When building your logstash filter, you would often like to validate your assumptions on a large sampling of input events without sending all the output to ElasticSearch.

Using Logstash metrics and conditionals, we can easily show:

  • How many input events were processed successfully
  • How many input events had errors
  • An error file containing each event that processed in error

This technique gives you the ability to track your success rate across a large input set, and then do a postmortem review of each event that failed.

I’ll walk you through a Logstash conf file that illustrates this concept.

Continue reading “Logstash: Using metrics to debug the filtering process”

Ubuntu: Using a swap file instead of swap partition for virtualized server VMs

ubuntuBefore virtualization, there was a stronger argument for using a swap partition instead of a swap file for servers.  A fragmented swap file could lead to performance issues that a statically sized and placed partition did not have consider.

But once virtualization comes into play, unless you go to great lengths to segment your storage pools, that swap partition is not guaranteed to be either statically sized or statically placed on a physical platter.  And at that point, you should consider using a swap file which provides more flexibility in sizing and capacity planning.

Here are instructions for adding a 16Gb swap file to Ubuntu:

Continue reading “Ubuntu: Using a swap file instead of swap partition for virtualized server VMs”

Ubuntu: Using pdftk to stitch together two-sided PDF

ubuntuThere are many consumer side printers that provide the ability to scan a document to PDF.  But unless you have a high-end series, the printer may only be capable of scanning one side at a time, which means you end up with a “front.pdf” and “back.pdf”.

If you have a Linux desktop or laptop, luckily the solution is as simple as calling ‘pdftk’.

Continue reading “Ubuntu: Using pdftk to stitch together two-sided PDF”

Logstash: Testing Logstash grok patterns online

elastic-logstash-fwIn my previous posts, I have shown how to test grok patterns locally using Ruby on Linux and Windows.  This works well when your VM do not have full internet access, or only have console access, or any reason that you want to test it locally.

If you have access to a graphical web browser and the log file, there is a nice online grok constructor here and here. and by simply entering a sampling of the log lines and a grok pattern, you can verify that all the lines are parsed correctly.

Here is a small example to start you off:

Continue reading “Logstash: Testing Logstash grok patterns online”

Logstash: Testing Logstash grok patterns locally on Windows

elastic-logstash-fwIf the logs you are shipping to Logstash are from a Windows OS, it makes it even more difficult to quickly troubleshoot a grok pattern being sent to the Logstash service.

It can be beneficial to quickly validate your grok patterns directly on the Windows host.  Here is an easy way to test a log against a grok pattern:

Continue reading “Logstash: Testing Logstash grok patterns locally on Windows”