An issue that keeps coming up on the mailing lists as well as Stackoverflow[1,2] is how to merge multiple pillar files for use with a single state. The problem is that pillars using the same key overwrite each other, and there is no easy way to express the desire to merge instead.
There are various workarounds, but all of these expect the human operator to know about these disparate sources and manually mend them together with a unifying sls file (using includes or anchors/references).
The state and pillar files in this article can be downloaded from my github page.
Continue reading “SaltStack: Combine multiple pillar files under a single key”
Configuration Management tools like SaltStack are invaluable for managing infrastructure at scale. Even in the growing world of containerization where immutable image deployment is the norm, those images need to be built in a repeatable and auditable fashion.
This article will detail installation of the SaltStack master on Ubuntu 14.04, with validation using a single Minion. Note that Minion installation is not mandatory if using Salt SSH.
Continue reading “SaltStack: Installing a Salt Master on Ubuntu 14.04”
ElasticSearch’s Metricbeat is a lightweight shipper of both system and application metrics that runs as an agent on a client host. That means that along with standard cpu/mem/disk/network metrics, you can also monitor Apache, Docker, Nginx, Redis, etc. as well as create your own collector in the Go language.
In this article we will describe installing Metricbeat 5.x on Ubuntu when the back end ElasticSearch version is either 5.x or 2.x.
Continue reading “ELK: Installing MetricBeat for collecting system and application metrics”
ElasticSearch’s commercial X-Pack has alerting functionality based on ElasticSearch conditions, but there is also a strong open-source contender from Yelp’s Engineering group called ElastAlert.
ElastAlert offers developers the ultimate control, with the ability to easily create new rules, alerts, and filters using all the power and libraries of Python.
Continue reading “ELK: ElastAlert for alerting based on data from ElasticSearch”
The open-source Zabbix monitoring solution has very lightweight agents that are easy to install on Ubuntu.
Although the Ubuntu main repository has a build available, it is older and so we are going to choose to download and install the latest point version in this article. Unfortunately, the repo.zabbix.com cannot be added directly as an Ubuntu repository source.
Continue reading “Zabbix: Installing a Zabbix Agent on Ubuntu 14.04”
The Curator product from ElasticSearch allows you to apply batch actions to your indexes (close, create, delete, etc.). One specific use case is applying a retention policy to your indexes, deleting any indexes that are older than a certain threshold.
Start by installing Curator using apt and pip:
$ sudo apt-get install python-pip -y
$ sudo pip install elasticsearch-curator
$ /usr/local/bin/curator --version
Continue reading “ELK: Using Curator to manage the size and persistence of your index storage”
Although container based engines such as Docker are highly popularized for newer application deployment – there will still be widespread use of OS virtualization engines for years to come.
One of the most popular virtualization engines for development purposes is the open-source VirtualBox from Oracle. This article will detail its installation on Ubuntu 14.04.
Continue reading “VirtualBox: Installing VirtualBox and Vagrant on Ubuntu 14.04/16.04”
Building services using Spring Boot gives a development team a jump start on many production concerns, including logging. But unlike a standard deployment where logging to a local file is where the developer’s responsibility typically ends, with Docker we must think about how to log to a public space outside our ephemeral container space.
The Docker logging drivers capture all the output from a container’s stdout/stderr, and can send a container’s logs directly to most major logging solutions (syslog, Logstash, gelf, fluentd).
As an added benefit, by making the logging implementation a runtime choice for the container, it provides flexibility to use a simpler implementation during development but a highly-available, scalable logging solution in production.
Continue reading “Docker: Sending Spring Boot logging to syslog”
Once you have a Squid proxy setup as described in my article here, the next challenge is configuring your Ubuntu servers so that they use this proxy by default instead of attempting direct internet connections.
There are several entities we want using Squid by default: apt package manager, interactive consoles and wget/curl, and Java applications.
Continue reading “Squid: Configuring an Ubuntu host to use a Squid proxy for internet access”
HAProxy is a high performance TCP/HTTP (Level 4 and Level 7) load balancer and reverse proxy. A common pattern is allowing HAProxy to be the fronting SSL-termination point, and then HAProxy determines which pooled backend server serves the request.
Continue reading “HAProxy: Using HAProxy for SSL termination on Ubuntu”
Nginx is a popular reverse proxy and load balancer that focuses on level 7 (application) traffic. A common pattern is allowing Nginx to be the fronting SSL-termination point, and then Nginx determines which pooled backend server is best available to serve the request.
Continue reading “Nginx: Using Nginx for SSL termination on Ubuntu”
Some web applications leave authentication as an orthogonal concern to the application – not including any kind of login functionality and instead leaving authentication as an operational concern.
When this happens, a reverse proxy that has an LDAP integration can act as an architectural sentry in front of the web application and also fulfills the requirements for Single Sign-On. Apache2 serves this purpose very well with minimal overhead.
Continue reading “Apache2: Enable LDAP authentication and SSL termination for Ubuntu”
Jenkins is the open-source automation server that is critical in building a continuous integration and delivery pipeline. It is extensible and has a wealth of plugins that integrate with numerous enterprise systems.
Here are the detailed steps for installing a Jenkins server on Ubuntu.
Continue reading “Jenkins: Setting up a continuous integration server on Ubuntu”
Especially in enterprise application development, there can be 3rd party dependencies that are not available in public Maven repositories. These may be internal, business specific libraries or licensed libraries that have limitations on usage.
When this is the case, you can either publish to a private Maven repository that controls authorization or you can put them into your local cached maven repository.
Continue reading “Maven: Installing a 3rd party jar to a local or remote repository”
An essential part of the standard build process for Java applications is having a set of repositories where project artifacts are stored.
Artifact curation provides the ability to manage dependencies, quickly rollback releases, support compatibility of downstream projects, do QA promotion from test to production, support a continuous build pipeline, and provides auditability.
JFrog puts out an open-source Maven server called Artifactory that is perfect for setting up a private Maven repository for internal applications.
Continue reading “Maven: Installing a private Maven repository on Ubuntu using Artifactory”
Enabling verbose logs for an AppDynamics machine or database agents can be invaluable for troubleshooting connectivity or network issues.
Luckily, this is easily done by editing the conf/logging/log4j.xml file. By default, only the error level messages are sent to the logs:
But you can modify this so that debug level is sent:
Continue reading “AppDynamics: Enabling verbose debug logs for Agents”
As an exploration of AppDynamics’ APM functionality, you may find it useful to deploy a sample application that can quickly return back useful data. The Java Spring PetClinic connecting back to a PostgreSQL database provides a simple code base that exercises both database and application monitoring.
In a previous article, I went over the detailed steps for monitoring PetClinic with a MySQL backend, so I will refer back to that article for some of the details and will focus on the PostgreSQL specific steps here.
Continue reading “AppDynamics: Java Spring PetClinic and PostgreSQL configured for monitoring”
While enabling HTTPS is a important step in securing your web application, it is critical that you also take steps to disable legacy protocols and low strength ciphers that can circumvent the very security you are attempting to implement.
As long as you have the latest version of openssl then you should be able to use a bash script like below (credit for this script goes here) to enumerate every matching protocol and cipher that a server is exposing:
Continue reading “OpenSSL: Using OpenSSL to enumerate protocols and ciphers in use by web applications”
Selenium is an open-source solution for automating the browser allowing you to run continuous integration tests, validate performance and scalability, and perform regression testing of web applications.
This kind of automated testing is useful not only from desktop systems, but also from server machines where you may want to monitor availability or correctness of returned pages. For example, web site response monitoring or as part of a Jenkins validation pipeline.
The first method we can use to accomplish this is to use a headless driver such as the HtmlUnit or PhantomJS driver – these are tiny browser implementations that load and execute web pages but do not actually draw the results to a screen.
The second method is specific to Linux based systems, where you use the actual Chrome browser. The trick is to use Xvfb as a virtualized display.
Continue reading “Selenium: Running headless automated tests on Ubuntu”
If you have worked on deploying packages via apt-get, you are probably familiar with a couple of forms of interruption during the package installation and upgrade process.
The first is the text menu shown during package upgrades that informs you that a new configuration file is available and asks if you want to keep your current one, use the new one from the package maintainer, or show the difference.
The second is the occasional ASCII dialog that interrupts the install/upgrade and ask for essential information before moving forward. The screenshot below is the dialog you get when installing MySQL or MariaDB, asking to set the initial root password for the database.
The problem, in this age of cloud scale, is that you often need completely silent installations and upgrades that can be pushed out via Configuration Management. Even if this is a build for an immutable image, you would prefer a completely automated construction process instead of manual intervention each time you build an image.
Continue reading “Ubuntu: Silent package installation and debconf”
As an exploration of AppDynamics’ APM functionality, you may find it useful to deploy a sample application that can quickly return back useful data. The Java Spring PetClinic connecting back to a MySQL database provides a simple code base that exercises both database and application monitoring.
We’ll deploy the Java Spring PetClinic unto Tomcat running on Ubuntu 14.04. MySQL will be the backing persistence engine for the web application. The AppDynamics Java agent will be loaded into the JVM running Tomcat, and the AppDynamics Database Agent will connect to MySQL for metrics gathering.
Continue reading “AppDynamics: Java Spring PetClinic and MySQL configured for monitoring”
The AppDynamics Machine Agent is used not only to report back on basic hardware metrics (cpu/memory/disk/network), but also as the hook for custom plugins that can report back on any number of applications including: .NET, Apache, AWS, MongoDB, Cassandra, and many others.
In this article, I’ll go over the details to install the Machine Agent unto an Ubuntu 14.04 system.
Continue reading “AppDynamics: Installing a Machine Agent on Ubuntu 14.04”
The ElasticSearch stack (ELK) is popular open-source solution that serves as both repository and search interface for a wide range of applications including: log aggregation and analysis, analytics store, search engine, and document processing.
Its standard web front-end, Kibana, is a great product for data exploration and dashboards. However, if you have multiple data sources including ElasticSearch, want built-in LDAP authentication, or the ability to annotate graphs, you may want to consider Grafana to surface your dashboards and visualizations.
Continue reading “Grafana: Connecting to an ElasticSearch datasource”
Zabbix is an open-source monitoring solution that provides metrics collection, dynamic indexes, alerting, dashboards, and an API for external integration. But graphing is arguably one Zabbix’s weak points; it still builds static images while other enterprise and consumer applications have set end users’ expectations for graph visualization and interactivity very high.
Luckily, the Zabbix plugin for Grafana can put a facelift on the valuable data stored in Zabbix. With this new data source, your end users can get the beautiful dashboard view they expect from a modern application.
Continue reading “Grafana: Connecting to a Zabbix datasource”
Grafana is an open-source visualization suite that is able to generate graphs and dashboards, in addition to alerting.
It is designed to retrieve data from various backends including: Graphite, ElasticSearch, Prometheus, and Zabbix.
This article will lead you through an installation of the latest stable version on Ubuntu 14.04.
Continue reading “Grafana: Installation on Ubuntu 14.04”
Until Zabbix3, trend data was not available via the Zabbix API. This meant that you could retrieve the raw values of a key over time, but not the aggregated historical trends of that value (e.g. CPU average over 5 minute intervals).
The only way to monitor trends was to look at the visual graph generated by Zabbix or query the underlying database directly. Meanwhile, graphs are arguably one of Zabbix’s weak points, especially given newer solutions like Grafana.
This was a major oversight in Zabbix2 functionality, and led to community patches that enabled this functionality in Zabbix 2.x. With this trend data now exposed, the community was free to write custom alerting, graphing, and capacity planning tools. For example, the Zabbix plugin for Grafana relies on this patch when the data source is Zabbix 2.x.
Continue reading “Zabbix: Enabling API fetch of Trend data in Zabbix2”
Having Zabbix send alert mails directly to user groups is typically outgrown as the system matures and the number of alerts increase, new lines of business and engineering groups are on-boarded, and on-call scheduling is implemented.
If you already use PagerDuty for on-call scheduling, then it makes perfect sense to have Zabbix create incidents in PagerDuty. While it is possible to use standard email to perform some level of integration, the native library is the tightest integration you will find and supports multiple pager duty services.
The agent built by PagerDuty is especially well done, using their API to automatically create PagerDuty incidents as well as automatically mark them resolved if the trigger is only ephemeral (e.g. a temporary cpu spike).
Continue reading “Zabbix: Alert to PagerDuty using Zabbix3”
Oracle VirtualBox as a virtualization engine paired with Vagrant provides a cross-platform virtualization-agnostic workflow for Linux, Windows, and MacOS. It is light enough to allow a developer to setup, test, and tear down virtual infrastructure as part of a unit test.
You may find yourself in a position where you have built a VM in VirtualBox that you need to test in a lab running VMware vCloud Director. This may beg the question of, “Why not simply using the same script or process to rebuild the VM in the VMware lab?”
Perhaps the Vagrant box available in your VMware lab is not yet the latest version or OS flavor, maybe someone in the community constructed a Vagrant box for a software stack you have not yet scripted, or maybe internet access in the lab is limited to certain package repositories and you needed to reference a custom archive. Whatever the reason, you can export your VM from VirtualBox and import it into VMware if you are willing to jump through a few hoops.
Continue reading “VMware: Exporting from Oracle VirtualBox/Vagrant to vCloud Director”
The ELK stack (ElasticSearch-Logstash-Kibana), is a horizontally scalable solution with multiple tiers and points of extension and scalability.
Because so many companies have adopted the platform and tuned it for their specific use cases, it would be impossible to enumerate all the novel ways in which scalability and availability had been enhanced by load balancers, message queues, indexes on distinct physical drives, etc… So in this article I want to explore the obvious extension points, and encourage the reader to treat this as a starting point in their own design and deployment.
Continue reading “ELK: Architectural points of extension and scalability for the ELK stack”
The heart of the ELK stack is Elasticsearch. In order to provide high availability and scalability, it needs to be deployed as a cluster with master and data nodes. The Elasticsearch cluster is responsible for both indexing incoming data as well as searches against that indexed data.
As described in the documentation, if there is one absolutely critical resource it is memory. Keeping the heap size less than 32G will allow you to use compressed object pointers which is preferred. Swapping memory takes a big hit, so minimize swappiness on your Linux host.
Continue reading “ELK: Scaling an ElasticSearch Cluster”