Ubuntu: Installing the Genymotion Android emulator

Android is one of the leading platforms of the mobile industry.  By installing an Android emulator on your Ubuntu desktop, you can bring this power to your desktop.

More often than not, an Android emulator is used for custom development of mobile apps, but don’t overlook its utility as a way to access your favorite mobile applications directly from your desktop, or as a way to preview upcoming Android releases.

Continue reading “Ubuntu: Installing the Genymotion Android emulator”

Ubuntu: Installing Tor on Ubuntu 14.04/16.04

The Tor project is free software that helps protect your privacy by making it difficult for a 3rd party to analyze your network requests or link your traffic back to your network access point.  See the Tor overview page for reasons why this may be important to world citizens, corporations, or specific professions.

Simplified, this is done by using a large pool of distributed hosts and using varied and encrypted paths through these hosts to deliver your original request.

Be aware that no one is saying Tor provides fullproof anonymity on the internet, there are documented weaknesses [1,2,3].  But by now, it should be clear the security exists on a spectrum and not in absolute terms.

I will detail how to install both the Tor service and Tor browser which is designed to address the most common threats to remaining anonymous while browsing.

Continue reading “Ubuntu: Installing Tor on Ubuntu 14.04/16.04”

Ubuntu: Testing authenticated SMTP over TLS/SSL

SMTP mail relays exposed to the internet typically use a combination of SSL and authenticated SMTP to avoid abuse by malicious actors.

This is an excellent choice from a security perspective, but makes smoke testing a bit more complex than just opening telnet.

Continue reading “Ubuntu: Testing authenticated SMTP over TLS/SSL”

Ansible: Installing Ansible on Ubuntu 16.04

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

In this article I’ll describe how to deploy the latest release of Ansible using pip on Ubuntu 16.04, and then perform a quick validation against a client.

Continue reading “Ansible: Installing Ansible on Ubuntu 16.04”

Ansible: Managing a Windows host using Ansible

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

Ansible was started as a Linux only solution, leveraging ssh to provide a management channel to a target server.  However, starting at Ansible 1.7, support for Windows hosts was added by using Powershell remoting over WinRM.

Continue reading “Ansible: Managing a Windows host using Ansible”

Ansible: Installing Ansible on Ubuntu 14.04

Ansible is an agentless configuration management tool that helps operations teams manage installation, patching, and command execution across a set of servers.

In this article I’ll describe how to deploy the latest release of Ansible using pip on Ubuntu 14.04, and then perform a quick validation against a client.

Continue reading “Ansible: Installing Ansible on Ubuntu 14.04”

Zabbix: LLD low-level discovery returning multiple values

Zabbix low-level discovery (LLD) provides a way to create an array of related items, triggers, or graphs without needing to know the exact number of entities up front.

The easiest way to populate the keys of a discovery item is to add a “UserParameter” in zabbix_agentd.conf, and then the Zabbix agent will  invokes a script which returns the set of keys.

But the keys are only the first part of a real solution, because what you really want to send back are the values associated with those keys.  For example, if you are monitoring a database, you don’t want to just send the list of tables available, you may want to send back each table name and then its row count and size on disk.

Unfortunately Zabbix does not support sending back multiple values [1,2,3,4].  There are various workarounds such as using one UserParameter for the discovery key and another with a UserParameter=key[*] to fetch each row of data, or using vfs.file.regexp to parse values that have been written to a file.

But I think the cleanest solution, and one that requires the minimal number of spawned processes on the agent host is to invoke zabbix_sender from inside the script to send back all the values you want to populate.

Continue reading “Zabbix: LLD low-level discovery returning multiple values”

Docker: Visualizing image hierarchy and container dependency using dockviz

The Docker console commands for listing and viewing containers and images (ps, images, history, inspect) provides a wealth of information, but when you are managing hundreds of containers, a graph view of the container inventory and their dependencies can be critical for operations.

Dockviz can help you visualize your containers and images by creating an PNG image representing the container links and image lineage.

Continue reading “Docker: Visualizing image hierarchy and container dependency using dockviz”

GoLang: Running a Go binary as a systemd service on Ubuntu 16.04

The Go language with its simplicity, concurrency support,  rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.

However, the Go language does not natively provide a reliable way to daemonize itself.  In this article I will describe how to take a couple of simple Go language programs and run them using a systemd service file that starts them at boot time on Ubuntu 16.04.

Continue reading “GoLang: Running a Go binary as a systemd service on Ubuntu 16.04”

ELK: Connecting to ElasticSearch with a Go client

ElasticSearch very often serves as a repository for monitoring, logging, and business data.  As such, integrations with external system are a requirement.

The Go programming language with its convenient deployment binary and rich set of packages can easily serve as a bridge between these systems and the ElasticSearch server.

We will use the olivere/elastic package for this purpose, it is well maintained and has support for both ElasticSearch 5.x and 2.x depending on your import statement.  In this article, we will be hitting an ElasticSearch 2.x backend.

Continue reading “ELK: Connecting to ElasticSearch with a Go client”

GoLang: Running a Go binary as a SysV service on Ubuntu 14.04

The Go language with its simplicity, concurrency support,  rich package ecosystem, and ability to compile down to a single binary is an attractive solution for writing services on Ubuntu.

However, the Go language does not natively provide a reliable way to daemonize itself.  In this article I will describe how to take a couple of simple Go language programs, run them using SystemV init scripts with their own process owner, standard logs, and started at boot time on Ubuntu 14.04.

Continue reading “GoLang: Running a Go binary as a SysV service on Ubuntu 14.04”

Zabbix: Sending Zabbix metrics using a Go client

The open-source Zabbix monitoring solution has a published, simple binary protocol that allows you to send metrics to the Zabbix server without relying on the Zabbix Agent – which makes it very convenient for integration with other parts of your infrastructure.

In this article, I’ll show how to use the go-zabbix package for sending metrics to the Zabbix server.  If instead you were looking to manipulate the backend server definitions (host, templates, hostgroups, etc.) using the REST API, then see my other article here.

Continue reading “Zabbix: Sending Zabbix metrics using a Go client”

GoLang: Glide for Go language package management

Downloading 3rd party packages from github is made very simple in the Go language with the import statement. But similar to other languages, the complexity of versions and inter-dependencies begs the use of a package manager for any projects that are non-trivial (think npm for Javascript, pip for Python, Maven for Java, etc.).

Glide is a package manager for the Go programming language that can greatly ease the chore of package management by supporting package independence between projects, versioning, and non-master branches.

Continue reading “GoLang: Glide for Go language package management”

Zabbix: Zabbix REST API using a Go client

The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems.

This fosters development of community-driven modules like Ryan Day’s zabbix Go language package, which is an easy way to automate Zabbix tasks like creating hosts and manipulating other back end structures.

One of the nice things about the Go language is that libraries are generally statically linked into a single executable, so you only need to copy over a single executable.  You don’t have to copy 3rd party jars (Java) or require an internet connection to pypi from production system (Python).

Continue reading “Zabbix: Zabbix REST API using a Go client”

GoLang: Vendor directory for github branches other than master

Using 3rd party packages from github is made very simple in the Go language with the import statement.  But one problem is that “go get” will always pull the HEAD of the master branch and there is no way to explicitly specify another branch.

The ultimate answer would be to use a package dependency manager like Glide, which I describe in this article.  But if you cannot introduce Glide into your workflow yet then manually populating the vendor directory (enabled by default since 1.6) is a viable alternative.

Continue reading “GoLang: Vendor directory for github branches other than master”

GoLang: Cross Compiling for Linux and Windows platforms

A nice feature of the Go language is the ability to build binaries for multiple platforms directly from a single source system.  As an example, even from a development Windows 7 32-bit machine, you can build binaries for both 64 bit Linux and Windows 2012 Servers.

Before Go 1.5, you needed a compiler for the target architecture, but now that the entire tool chain is written in Go, building for multiple architectures is easy.

And unlike other languages where additional external libraries need to be copied or downloaded on the target system, Go dependencies are generally statically linked [1,2,3,4] into a single binary which makes portability that much easier.

Continue reading “GoLang: Cross Compiling for Linux and Windows platforms”

GoLang: Installing the Go Programming language on Ubuntu 14.04

The Go programming language has gotten considerable momentum, and the fact that it compiles down to machine code has made it popular in containers like Docker where a single executable binary fits the execution model perfectly.

This article will detail installation on Ubuntu 14.04 with the standard hello world validation.

Continue reading “GoLang: Installing the Go Programming language on Ubuntu 14.04”

SaltStack: Combine multiple pillar files under a single key

saltstack_logo-thumbnailAn issue that keeps coming up on the mailing lists as well as Stackoverflow[1,2] is how to merge multiple pillar files for use with a single state.  The problem is that pillars using the same key overwrite each other, and there is no easy way to express the desire to merge instead.

There are various workarounds, but all of these expect the human operator to know about these disparate sources and manually mend them together with a unifying sls file (using includes or anchors/references).

The state and pillar files in this article can be downloaded from my github page.

Continue reading “SaltStack: Combine multiple pillar files under a single key”

SaltStack: Installing a Salt Master on Ubuntu 14.04

saltstack_logo-thumbnailConfiguration Management tools like SaltStack are invaluable for managing infrastructure at scale.  Even in the growing world of containerization where immutable image deployment is the norm, those images need to be built in a repeatable and auditable fashion.

This article will detail installation of the SaltStack master on Ubuntu 14.04, with validation using a single Minion.  Note that Minion installation is not mandatory if using Salt SSH.

Continue reading “SaltStack: Installing a Salt Master on Ubuntu 14.04”

ELK: Custom template mappings to force field types

It is very common to have Logstash create time-based indexes in ElasticSearch that fit the format, <indexName>-YYYY.MM.DD.  This means events submitted with @timestamp for that day all go to the same index.

However, if you do not explicitly specify an index template that maps each field to a type, you can end up with unexpected query results.  The reason is that without explicit mappings, the index (that is created fresh each day) uses its best judgement to assign field types based on the first event inserted.

In this article, I’ll show you how to create explicit custom index templates so that field types are uniform across your time-series indexes.

Continue reading “ELK: Custom template mappings to force field types”

ELK: Installing Logstash on Ubuntu 14.04

elastic-logstash-fwLogstash provides a powerful mechanism for listening to various input sources, filtering and extracting the fields, and then sending events to a persistence store like ElasticSearch.

Installing Logstash on Ubuntu is well documented, so in this article I will focus on Ubuntu specific steps required for Logstash 2.x and 5.x.

Continue reading “ELK: Installing Logstash on Ubuntu 14.04”

ELK: Using Ruby in Logstash filters

elastic-logstash-fwLogstash has a rich set of filters, and you can even write your own, but often this is not necessary since there is a out-of-the-box filter that allows you to embed Ruby code directly in the configuration file.

Using logstash-filter-ruby, you can use all the power of Ruby string manipulation to parse an exotic regular expression, an incomplete date format, write to a file, or even make a web service call.

Continue reading “ELK: Using Ruby in Logstash filters”

Zabbix: Accessing Zabbix using the py-zabbix Python module

The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems.

This fosters development of community-driven modules like the py-zabbix Python module, which is an easy way to automate Zabbix as well as send/retrieve metrics.

Continue reading “Zabbix: Accessing Zabbix using the py-zabbix Python module”

Docker: logspout for Docker log collection

Docker log collection can be done using various methods, one method that is particularly effective is having a dedicated container whose sole purpose is to automatically sense other deployed containers and aggregate their log events.

This is the architectural model of logspout, an open-source project that acts as a router for the stdout/stderr logs of other containers.

If you do not have docker installed yet, see my article here.  Before moving on, you should be able to run the hello-world container.

Continue reading “Docker: logspout for Docker log collection”

ELK: Running ElastAlert as a service on Ubuntu 14.04

ElastAlert from the Yelp Engineering group provides a very flexible platform for alerting on conditions coming from ElasticSearch.

In a previous article I fully describe running interactively on an Ubuntu server, and now I’ll expand on that by running it at system startup using a System-V init script.

One of the challenges of getting ElastAlert to run as a service is that is has  a very strict set of module requirements that very easily conflicts with other Python applications, and so we will use Python’s virtualenv to build it in isolation and then call that wrapper from the service script.

Continue reading “ELK: Running ElastAlert as a service on Ubuntu 14.04”

ELK: Installing MetricBeat for collecting system and application metrics

ElasticSearch’s Metricbeat is a lightweight shipper of both system and application metrics that runs as an agent on a client host.  That means that along with standard cpu/mem/disk/network metrics, you can also monitor Apache, Docker, Nginx, Redis, etc. as well as create your own collector in the Go language.

In this article we will describe installing Metricbeat 5.x on Ubuntu when the back end ElasticSearch version is either 5.x or 2.x.

Continue reading “ELK: Installing MetricBeat for collecting system and application metrics”

ELK: ElastAlert for alerting based on data from ElasticSearch

ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert.

ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.

Continue reading “ELK: ElastAlert for alerting based on data from ElasticSearch”