Enabling verbose logs for an AppDynamics machine or database agents can be invaluable for troubleshooting connectivity or network issues. Luckily, this is easily done by editing the conf/logging/log4j.xml file. By default, only the error level messages are sent to the logs: <root> <priority value=”error”/> <appender-ref ref=”FileAppender”/> </root> But you can modify this so that debug … AppDynamics: Enabling verbose debug logs for Agents
As an exploration of AppDynamics’ APM functionality, you may find it useful to deploy a sample application that can quickly return back useful data. The Java Spring PetClinic connecting back to a PostgreSQL database provides a simple code base that exercises both database and application monitoring. In a previous article, I went over the detailed … AppDynamics: Java Spring PetClinic and PostgreSQL configured for monitoring
Selenium is an open-source solution for automating the browser allowing you to run continuous integration tests, validate performance and scalability, and perform regression testing of web applications. This kind of automated testing is useful not only from desktop systems, but also from server machines where you may want to monitor availability or correctness of returned … Selenium: Running headless automated tests on Ubuntu
If you have worked on deploying packages via apt-get, you are probably familiar with a couple of forms of interruption during the package installation and upgrade process. The first is the text menu shown during package upgrades that informs you that a new configuration file is available and asks if you want to keep your … Ubuntu: Silent package installation and debconf
As an exploration of AppDynamics’ APM functionality, you may find it useful to deploy a sample application that can quickly return back useful data. The Java Spring PetClinic connecting back to a MySQL database provides a simple code base that exercises both database and application monitoring. We’ll deploy the Java Spring PetClinic unto Tomcat running … AppDynamics: Java Spring PetClinic and MySQL configured for monitoring
The AppDynamics Machine Agent is used not only to report back on basic hardware metrics (cpu/memory/disk/network), but also as the hook for custom plugins that can report back on any number of applications including: .NET, Apache, AWS, MongoDB, Cassandra, and many others. In this article, I’ll go over the details to install the Machine Agent … AppDynamics: Installing a Machine Agent on Ubuntu 14.04
The ElasticSearch stack (ELK) is popular open-source solution that serves as both repository and search interface for a wide range of applications including: log aggregation and analysis, analytics store, search engine, and document processing. Its standard web front-end, Kibana, is a great product for data exploration and dashboards. However, if you have multiple data sources … Grafana: Connecting to an ElasticSearch datasource
Zabbix is an open-source monitoring solution that provides metrics collection, dynamic indexes, alerting, dashboards, and an API for external integration. But graphing is arguably one Zabbix’s weak points; it still builds static images while other enterprise and consumer applications have set end users’ expectations for graph visualization and interactivity very high. Luckily, the Zabbix plugin … Grafana: Connecting to a Zabbix datasource
Grafana is an open-source visualization suite that is able to generate graphs and dashboards, in addition to alerting. It is designed to retrieve data from various backends including: Graphite, ElasticSearch, Prometheus, and Zabbix. This article will lead you through an installation of the latest stable version on Ubuntu 14.04.
Until Zabbix3, trend data was not available via the Zabbix API. This meant that you could retrieve the raw values of a key over time, but not the aggregated historical trends of that value (e.g. CPU average over 5 minute intervals). The only way to monitor trends was to look at the visual graph generated … Zabbix: Enabling API fetch of Trend data in Zabbix2
Having Zabbix send alert mails directly to user groups is typically outgrown as the system matures and the number of alerts increase, new lines of business and engineering groups are on-boarded, and on-call scheduling is implemented. If you already use PagerDuty for on-call scheduling, then it makes perfect sense to have Zabbix create incidents in … Zabbix: Alert to PagerDuty using Zabbix3
Oracle VirtualBox as a virtualization engine paired with Vagrant provides a cross-platform virtualization-agnostic workflow for Linux, Windows, and MacOS. It is light enough to allow a developer to setup, test, and tear down virtual infrastructure as part of a unit test. You may find yourself in a position where you have built a VM in … VMware: Exporting from Oracle VirtualBox/Vagrant to vCloud Director
The ELK stack (ElasticSearch-Logstash-Kibana), is a horizontally scalable solution with multiple tiers and points of extension and scalability. Because so many companies have adopted the platform and tuned it for their specific use cases, it would be impossible to enumerate all the novel ways in which scalability and availability had been enhanced by load balancers, … ELK: Architectural points of extension and scalability for the ELK stack
The heart of the ELK stack is Elasticsearch. In order to provide high availability and scalability, it needs to be deployed as a cluster with master and data nodes. The Elasticsearch cluster is responsible for both indexing incoming data as well as searches against that indexed data. Resources As described in the documentation, if there … ELK: Scaling an ElasticSearch Cluster
The most varied point in an ELK (Elasticsearch-Logstash-Kibana) stack is the mechanism by which custom events and logs will get sent to Logstash for processing. Companies running Java applications with logging sent to log4j or SLF4J/Logback will have local log files that need to be tailed. Applications running in containers may send everything to stdout/stderr, … ELK: Feeding the logging pipeline
If you are running an Ubuntu server for any extended period of time, security issues will arise that affect the kernel, distribution, or packages installed on that host. While there is a minimal stability risk with automatically applying security fixes, I feel those are dwarfed by the risks of running hosts that have known security … Ubuntu: Unattended Upgrades for security patches
Although the ELK stack has rich support for clustering, clustering is not supported over WAN connections due to Elasticsearch being sensitive to latency. There are also practical concerns of network throughput given how much data some installations index on an hourly basis. So as nice as it would be to have a unified, eventually consistent … ELK: Federated Search with a Tribe node
Kibana is the end user web application that allows us to query Elasticsearch data and create dashboards that can be used for analysis and decision making. Although Kibana can be pointed to any of the nodes in your Elasticsearch cluster, the best way to distribute requests across the nodes is to use a non-master, non-data … ELK: Pointing Kibana to a Client Node
SaltStack has the ability to create custom states, grains, and external pillars. There is a long list of standard external pillars ranging from those which read from local JSON files, to those that pull from EC2, MongoDB, etcd, and MySQL. In this article, we will use Apache ZooKeeper as the storage facility for our SaltStack … SaltStack: Creating a ZooKeeper External Pillar using Python
It may be hard to imagine on the development side, but there are instances where a deployed host is not accessible from the Salt Master in a production environment. This forces a bit of creativity if you have a set of standard formulas you need to apply to the host. For instance, imagine a host … SaltStack: Running a masterless minion on Ubuntu
Python is a language whose advantages are well documented, and the fact that it has become ubiquitous on most Linux distributions makes it well suited for quick scripting duties. In this article I’ll go through an example of using Python to read entries from a JSON file, and from each of those entries create a … Python: Using Python, JSON, and Jinja2 to construct a set of Logstash filters
For full instructions on installing the AppDynamics Controller on Linux, see the official documentation. However, when you get to the step for installing in silent mode, it can be confusing because although it shows you how to specify the path to a response file and the keys available, it does not give you a sample … AppDynamics: Silent Install of Controller on Ubuntu and license directory
The Logstash Indexing layer receives data from any number of input sources, transforms the data, and then submits it to Elasticsearch for indexing. Transforming and extracting data from every event can be both I/O as well as CPU intensive. Horizontal or Vertical Vertical scaling will only go so far in the Logstash indexing layer. In … ELK: Performance of the Logstash Indexing layer
When using jinja2 for SaltStack formulas you may be surprised to find that your global scoped variables do not have ability to be modified inside a loop. Although this is counter intuitive given the scope behavior of most scripting languages it is unfortunately the case that a jinja2 globally scoped variable cannot be modified from … SaltStack: Setting a jinja2 variable from an inner block scope
SLF4J, the Simple Logging Facade for Java, is a popular front for various logging backends, one of the being Logback. With the advent of containerization, using syslog to send data to remote logging infrastructure has become a popular transport method. Enable Syslog Input The first step is to enable the receipt of syslog messages. This … Syslog: Sending Java SLF4J/Logback to Syslog
Logging has always been a critical part of application development. But the rise of OS virtualization, applications containers, and cloud-scale logging solutions has turned logging into something bigger that managing local debug files. Modern applications and services are now expected to feed log aggregation and analysis stacks (ELK, Graylog, Loggly, Splunk, etc). This can be … Syslog: Sending Java log4j2 to rsyslog on Ubuntu
SaltStack grains are used for relatively static information such as operating system, IP address, and other system properties. They are also useful for targeting minions, for example whether a system is part of dev/test/prod, or a flag on whether it falls under LifeScience or HIPAA regulation. In this article we will implement a custom grain … SaltStack: Creating a Custom Grain using Python
In a production environment, it is common to have restricted internet access on the production deployment hosts. This means that using the standard ‘npm install’ and pulling modules from the registry.npmjs.org repository is not an option. Given the breadth of the dependency graph required for most modules, this packaging is something you want automated without … Node.js: Packaging modules for offline deployment using npm-bundle
When tasked with deploying a web application and it is not responsive to your browser requests, sometimes you need to take a step back from the complexity of your full stack and run a quick sanity check. You can use netcat as a simple web server to prove to yourself that the network infrastructure is … Ubuntu: Simulating a Web Server using Netcat
Although virtualization has pushed a self-service culture for infrastructure, it is still common in production environments to need your Network Operations team to open the required ports necessary for any new application deployment. So, while you may be able to create the base virtualized host, you can’t go much further without the network connectivity. And … Ubuntu: Pre-Validate Network ACL and Firewall Connectivity with Netcat
When troubleshooting basic connectivity from your SaltStack minions to your Salt master, the first thing to remember is the basic flow – the minions initiate the connection to port 4505/4506 on the Salt master. With this in mind, if you have modified /etc/salt/minion so that the master is explicitly set and logs are set to … SaltStack: Troubleshooting Basic Network Connectivity of Minion on Ubuntu
Before running state.apply against a minion, especially in a production environment, a good sanity test can be to list the states that will be executed without actually running those states. This can be done by adding tests=True to the end of the state command. For example, to check all the states that will be applied … SaltStack: Validating States of Minion without Execution
In production data centers, it is not uncommon to have limited public internet access due to security policies. So while running ‘apt-get’ or adding a repository to sources.list is easy in your development lab, you have to figure out an alternative installation strategy because you need a process that looks the same across both development … Ubuntu: Installing Packages without Public Internet Access
Content Packs are plugins that allow you you to create pre-packaged knowledge about specific event types. For example, you can create a content pack that knows how to extract fields from one of your custom log sources. Beyond extracted fields, you can also add saved queries, aggregations, alerts, dashboards, and visualizations. Incoming Events from Agent … vRealize Log Insight: Creating your own content pack for field extraction
It is common for a virtualized Guest OS base image to have a generic minimal storage capacity. But this capacity can easily be exceeded by production scenarios, performance testing, logging, or even the general cruft of running a machine 24×7. For this reason, extending a virtualized disk can be extremely helpful. Here is a walk … Ubuntu: Extending a virtualized disk using fdisk when not using LVM
When building your logstash filter, you would often like to validate your assumptions on a large sampling of input events without sending all the output to ElasticSearch. Using Logstash metrics and conditionals, we can easily show: How many input events were processed successfully How many input events had errors An error file containing each event … Logstash: Using metrics to debug the filtering process
In my previous posts, I have shown how to test grok patterns locally using Ruby on Linux and Windows. This works well when your VM do not have full internet access, or only have console access, or any reason that you want to test it locally. If you have access to a graphical web browser … Logstash: Testing Logstash grok patterns online
If the logs you are shipping to Logstash are from a Windows OS, it makes it even more difficult to quickly troubleshoot a grok pattern being sent to the Logstash service. It can be beneficial to quickly validate your grok patterns directly on the Windows host. Here is an easy way to test a log … Logstash: Testing Logstash grok patterns locally on Windows