python

GCP: publishing and reading from Google PubSub Topic using Python client libraries


Categories: Hyperscaler, Python

Google Pub/Sub is a managed messaging platform providing a scalable, asynchronous, loosely-coupled solution for communication between application entities. It centers around the concept of a Topic (queue).  A Publisher can put messages on the Topic, and a Subscriber can read messages from the Subscription on a Topic. In this article, I will first use the … GCP: publishing and reading from Google PubSub Topic using Python client libraries

Mac: multiple Python versions/virtualenv with brew and pyenv


Categories: Mac, Python

Although you could use brew to install Python directly, the cleaner way to manage Python versions and isolate Python virtual environments is by using pyenv and pyenv-virtualenv. pyenv allows you to install and switch between different versions of Python, while pyenv-virtualenv provides isolation of pip modules, for independence between projects.

Python: TreeMap visualization of hierarchical Pandas DataFrame


Categories: Python

The Plotly graphing library has a wide array of visualizations for datasets.  And it has native support for Pandas DataFrame, which makes it convenient for datasets coming from a wide range of sources. One visualization that I find particularly useful for hierarchical data is the TreeMap, which can group on data lineage, uses area as … Python: TreeMap visualization of hierarchical Pandas DataFrame

Python: migrating pip modules to newer Python version on Ubuntu


Categories: Python

Migrating from one Python 3.x version to a newer 3.x minor version seems like it would just be a simple ‘apt install’ of the latest Python package.  But you most likely have pip modules installed at a version specific ‘dist-packages’ or ‘site-packages’ directory, and those modules have to be freshly installed into the newer version … Python: migrating pip modules to newer Python version on Ubuntu

Python: Flask-OIDC protecting Client App and Resource Server using Windows 2019 ADFS


Categories: Authentication, Python

Flask OIDC is an extension to the popular Flask web framework that enables OAuth2/OIDC for your application.  The base project does not support ADFS, but I have create a personal fork of this module that supports Windows 2019 ADFS as the OAuth2 Authentication Server. In this article, we will exercise the OAuth2 Authorization Code flow.  … Python: Flask-OIDC protecting Client App and Resource Server using Windows 2019 ADFS

Python: New Relic Agent for Gunicorn app deployed on Kubernetes


Categories: Containers, Kubernetes, Python

Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads on a production Kubernetes cluster, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a series of previous articles, we … Python: New Relic Agent for Gunicorn app deployed on Kubernetes

Python: New Relic instrumentation for Flask app deployed with Gunicorn


Categories: Containers, Logging, Python

Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production.  If you are running these types of workloads in production, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a previous article, we created a Docker image of … Python: New Relic instrumentation for Flask app deployed with Gunicorn

GCP: Deploying a 2nd gen Python Cloud Function and exposing from an HTTPS LB


Categories: Hyperscaler

GCP Cloud Functions have taken a step forward with the 2nd generation release.  One of the biggest architectural differences is that now multiple request can run concurrently on a single instance, enabling large traffic loads. In this article, I will show you how to deploy a simple Python Flask web server as a 2nd gen … GCP: Deploying a 2nd gen Python Cloud Function and exposing from an HTTPS LB

Python: printing in color using ANSI color codes


Categories: Python

Although there are Python modules [1,2,3] specially suited for displaying text to the console in color, if you want a quick no-dependency method then you can use ANSI color codes directly. Here is an example of printing a line in green, then red. print(“\033[0;32mOK this is green\033[00m”) print(“\033[0;31mERROR this is red\033[00m”) Additional color codes can … Python: printing in color using ANSI color codes

Python: converting JSON to dot notation for easier path determination


Categories: Python, Scripting

Most of the modern cloud platforms and utilities have us manipulate either JSON or YAML configuration files.  And when you start dealing with real world scenarios with hundreds of lines of embedded data structures it is too difficult and error-prone to manually inspect indentation levels to determine the exact dotted or json path to an … Python: converting JSON to dot notation for easier path determination

Kubernetes: Using Downward API metadata from a Python application


Categories: Containers

In a previous post, I described the Kubernetes Downward API and how it allows us to inject pod/container metadata into our runtime container. In this article, I’ll show how you can read the environment variables and mounted files from inside a containerized Python based application.

Python: exploring the use of startswith against a list: tuple, regex, list comprehension, lambda


Categories: Python

As with any programming language, Python has a multitude of ways to accomplish the same task.  In this article, I will explore the idea of taking a string and checking if it ‘startswith’ any of the strings from a predetermined list. As context for the example code, RFC1918 sets out several IPv4 ranges that can … Python: exploring the use of startswith against a list: tuple, regex, list comprehension, lambda

Python: Publishing and Consuming from RabbitMQ using Python


Categories: Python

The pika module for Python provides an easy interface for creating exchanges and queues as well as producers/consumers for RabbitMQ . In this article, I will provide examples of a producer and consumer written in Python3.  All source code is available on github.

Python: Examples of list comprehension


Categories: Python

Python list comprehension provides a concise and pythonic way of generating lists.  It allows you to select a subset based on a filter and apply a function to each element. Because although navigating every list entry, filtering certain records, and applying functions to specific fields is doable using basic ‘for’ statements, ‘if’ logic, and temporary … Python: Examples of list comprehension

Python: Using Flask to stream chunked dynamic content to end users


Categories: Python

If you are using Flask to generate dynamic content of significant size, such as large binary images/pdf or large text-based datasets, then you need to consider streaming to minimize the memory footprint of Flask and preserve scalability. Using an inner generate function and a yield allows Flask to return chunks of data back to the … Python: Using Flask to stream chunked dynamic content to end users

Python: Using a custom decorator to inspect function arguments


Categories: Python

The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. Building on my earlier article on the basics of introspection, I will now show how to inject custom … Python: Using a custom decorator to inspect function arguments

Python: Using inspection to view the parameters of a function


Categories: Python

The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type.  I will show a simple example of its usage in this article.

Python: Getting live output from subprocess using poll


Categories: Python

Using subprocess.Popen, subprocess.call, or subprocess.check_output will all invoke a process using Python, but if you want live output coming from stdout you need use subprocess.Popen in tandem with the Popen.poll method. In this article I will show how to invoke a process from Python and show  stdout live without waiting for the process to complete.  … Python: Getting live output from subprocess using poll

Python: Parsing command line arguments with argparse


Categories: Python, Scripting

Especially when writing small Python utility programs, it can be tempting to use sys.argv to retrieve a small list of positional parameters from the command line.  But small utilities turn into successful tools over time, and manually parsing optional parameters and switches will quickly overwhelm your code logic. In this article I will show how … Python: Parsing command line arguments with argparse

Python: Using pip with a squid proxy


Categories: Development

In a production datacenter it would not be uncommon for internet access to be limited to domains whitelisted on a web proxy such as Squid.  If this is the case, and you are using pip to install packages, then you will need to: Have your Squid administrators whitelist pypi.python.org Add the “–proxy” switch when invoking … Python: Using pip with a squid proxy

PuTTy: Bulk import PuTTy session definitions into the registry using Powershell


Categories: Python, Scripting

Putty is one of the first tools I install on any host or jumpbox.  And creating a saved session definition is extremely helpful so I can get the right window size, scrollback, keep alives, color scheme, etc. but creating each session definition by hand is time consuming. In this article, I will show you how … PuTTy: Bulk import PuTTy session definitions into the registry using Powershell

Zabbix: Accessing Zabbix using the py-zabbix Python module


Categories: Monitoring, Python

The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems. This fosters development of community-driven modules like the py-zabbix Python module, which is an easy way to automate Zabbix as well as send/retrieve metrics.

ELK: ElasticDump and Python to create a data warehouse job


Categories: Logging, Python

By nature, the amount of data collected in your ElasticSearch instance will continue to grow and at some point you will need to prune or warehouse indexes so that your active collections are prioritized. ElasticDump can assist in moving your indexes either to a distinct ElasticSearch instance that is setup specifically for long term data, or exporting … ELK: ElasticDump and Python to create a data warehouse job

SaltStack: Creating a ZooKeeper External Pillar using Python


Categories: DevOps, Python, Scripting

SaltStack has the ability to create custom states, grains, and external pillars.  There is a long list of standard external pillars ranging from those which read from local JSON files, to those that pull from EC2, MongoDB, etcd, and MySQL. In this article, we will use Apache ZooKeeper as the storage facility for our SaltStack … SaltStack: Creating a ZooKeeper External Pillar using Python

Python: Using Python, JSON, and Jinja2 to construct a set of Logstash filters


Categories: DevOps, Python, Scripting

Python is a language whose advantages are well documented, and the fact that it has become ubiquitous on most Linux distributions  makes it well suited for quick scripting duties. In this article I’ll go through an example of using Python to read entries from a JSON file, and from each of those entries create a … Python: Using Python, JSON, and Jinja2 to construct a set of Logstash filters