Python

GCP: publishing and reading from Google PubSub Topic using Python client libraries


Categories: Hyperscaler, Python

Google Pub/Sub is a managed messaging platform providing a scalable, asynchronous, loosely-coupled solution for communication between application entities. It centers around the concept of a Topic (queue).  A Publisher can put messages on the Topic, and a Subscriber can read messages from the Subscription on a Topic. In this article, I will first use the … GCP: publishing and reading from Google PubSub Topic using Python client libraries

Ubuntu: pyenv for managing multiple Python versions and environments


Categories: Python

Keeping the Ubuntu system-level Python version and modules independent from those desired at each project level is a difficult task best managed by a purpose-built tool. There are many solutions in the Python ecosystem, but one that stands out for simplicity is pyenv and pyenv-virtualenv. pyenv allows you to install and switch between different versions … Ubuntu: pyenv for managing multiple Python versions and environments

Mac: multiple Python versions/virtualenv with brew and pyenv


Categories: Mac, Python

Although you could use brew to install Python directly, the cleaner way to manage Python versions and isolate Python virtual environments is by using pyenv and pyenv-virtualenv. pyenv allows you to install and switch between different versions of Python, while pyenv-virtualenv provides isolation of pip modules, for independence between projects.

Python: fixing ‘CryptographyDeprecationWarning: Blowfish has been deprecated’


Categories: Python

If you are getting a warning similar to below when running a Python3 application: /usr/lib/python3/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated This can be resolved by upgrading to the latest paramiko module. # check current version then upgrade pip3 show paramiko pip3 install paramiko –upgrade # check upgraded version pip3 show paramiko In my case, this … Python: fixing ‘CryptographyDeprecationWarning: Blowfish has been deprecated’

Python: TreeMap visualization of hierarchical Pandas DataFrame


Categories: Python

The Plotly graphing library has a wide array of visualizations for datasets.  And it has native support for Pandas DataFrame, which makes it convenient for datasets coming from a wide range of sources. One visualization that I find particularly useful for hierarchical data is the TreeMap, which can group on data lineage, uses area as … Python: TreeMap visualization of hierarchical Pandas DataFrame

Python: migrating pip modules to newer Python version on Ubuntu


Categories: Python

Migrating from one Python 3.x version to a newer 3.x minor version seems like it would just be a simple ‘apt install’ of the latest Python package.  But you most likely have pip modules installed at a version specific ‘dist-packages’ or ‘site-packages’ directory, and those modules have to be freshly installed into the newer version … Python: migrating pip modules to newer Python version on Ubuntu

Python: Flask-OIDC protecting Client App and Resource Server using Windows 2019 ADFS


Categories: Authentication, Python

Flask OIDC is an extension to the popular Flask web framework that enables OAuth2/OIDC for your application.  The base project does not support ADFS, but I have create a personal fork of this module that supports Windows 2019 ADFS as the OAuth2 Authentication Server. In this article, we will exercise the OAuth2 Authorization Code flow.  … Python: Flask-OIDC protecting Client App and Resource Server using Windows 2019 ADFS

Python: New Relic Agent for Gunicorn app deployed on Kubernetes


Categories: Containers, Kubernetes, Python

Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads on a production Kubernetes cluster, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a series of previous articles, we … Python: New Relic Agent for Gunicorn app deployed on Kubernetes

Python: New Relic instrumentation for Flask app deployed with Gunicorn


Categories: Containers, Logging, Python

Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production.  If you are running these types of workloads in production, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a previous article, we created a Docker image of … Python: New Relic instrumentation for Flask app deployed with Gunicorn

Python: Building an image for a Flask app served from Gunicorn


Categories: Containers, Python

Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production.  Running Flask applications directly is great for development and testing of the basic request/response flow, but you need gunicorn to handle production level loads,  concurrency, logging, and timeouts. In this article, I will show you how to build a Docker image … Python: Building an image for a Flask app served from Gunicorn

Python: constructing a DataFrame from a relational database with pandas


Categories: Python

If the dataset you want to analyze with pandas is coming from a normalized relational database, then you can use ‘pandas.read_sql‘ to pull the data directly. In this article, we will deploy a small MariaDB instance with Docker and show how we can create DataFrame directly from a single table or from a join between … Python: constructing a DataFrame from a relational database with pandas

Python: printing in color using ANSI color codes


Categories: Python

Although there are Python modules [1,2,3] specially suited for displaying text to the console in color, if you want a quick no-dependency method then you can use ANSI color codes directly. Here is an example of printing a line in green, then red. print(“\033[0;32mOK this is green\033[00m”) print(“\033[0;31mERROR this is red\033[00m”) Additional color codes can … Python: printing in color using ANSI color codes

Python: find the most recently modified file matching a pattern


Categories: Python

Whether it is the most recent log file, image, or report –  sometimes you will need to find the most recently modified file in a directory.  The example below finds the latest file in the “/tmp” directory. import os import glob # get list of files that matches pattern pattern=”/tmp/*” files = list(filter(os.path.isfile, glob.glob(pattern))) # … Python: find the most recently modified file matching a pattern

Python: converting JSON to dot notation for easier path determination


Categories: Python, Scripting

Most of the modern cloud platforms and utilities have us manipulate either JSON or YAML configuration files.  And when you start dealing with real world scenarios with hundreds of lines of embedded data structures it is too difficult and error-prone to manually inspect indentation levels to determine the exact dotted or json path to an … Python: converting JSON to dot notation for easier path determination

Python: Setting the preferred Python version on Bionic 18 and Focal 20


Categories: Python

Even though Python2 reached EOL in January 2020, there are still lagging applications and modules that force its continued use on certain systems.  If that is your situation, you need the ability to force preference to Python3 unless a program explicitly invokes Python2. If you are using Ubuntu Focal 20, then you have an easy … Python: Setting the preferred Python version on Bionic 18 and Focal 20

Python: exploring the use of startswith against a list: tuple, regex, list comprehension, lambda


Categories: Python

As with any programming language, Python has a multitude of ways to accomplish the same task.  In this article, I will explore the idea of taking a string and checking if it ‘startswith’ any of the strings from a predetermined list. As context for the example code, RFC1918 sets out several IPv4 ranges that can … Python: exploring the use of startswith against a list: tuple, regex, list comprehension, lambda

Python: Publishing and Consuming from RabbitMQ using Python


Categories: Python

The pika module for Python provides an easy interface for creating exchanges and queues as well as producers/consumers for RabbitMQ . In this article, I will provide examples of a producer and consumer written in Python3.  All source code is available on github.

Python: Examples of list comprehension


Categories: Python

Python list comprehension provides a concise and pythonic way of generating lists.  It allows you to select a subset based on a filter and apply a function to each element. Because although navigating every list entry, filtering certain records, and applying functions to specific fields is doable using basic ‘for’ statements, ‘if’ logic, and temporary … Python: Examples of list comprehension

Python: Using Flask to upload files


Categories: Python

If you are looking for basic HTTP file uploading using Flask, you can use the code in my python-flask-upload-files project on github as an example. This project exercises multiple scenarios: Upload a single file in chunked mode (not forced to save entirety to disk) Upload a single file Upload multiple files Error message shown when … Python: Using Flask to upload files

Python: Using Flask to stream chunked dynamic content to end users


Categories: Python

If you are using Flask to generate dynamic content of significant size, such as large binary images/pdf or large text-based datasets, then you need to consider streaming to minimize the memory footprint of Flask and preserve scalability. Using an inner generate function and a yield allows Flask to return chunks of data back to the … Python: Using Flask to stream chunked dynamic content to end users

Python: Using a custom decorator to inspect function arguments


Categories: Python

The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. Building on my earlier article on the basics of introspection, I will now show how to inject custom … Python: Using a custom decorator to inspect function arguments

Python: Using inspection to view the parameters of a function


Categories: Python

The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type.  I will show a simple example of its usage in this article.

Python: Getting live output from subprocess using poll


Categories: Python

Using subprocess.Popen, subprocess.call, or subprocess.check_output will all invoke a process using Python, but if you want live output coming from stdout you need use subprocess.Popen in tandem with the Popen.poll method. In this article I will show how to invoke a process from Python and show  stdout live without waiting for the process to complete.  … Python: Getting live output from subprocess using poll

Python: Parsing command line arguments with argparse


Categories: Python, Scripting

Especially when writing small Python utility programs, it can be tempting to use sys.argv to retrieve a small list of positional parameters from the command line.  But small utilities turn into successful tools over time, and manually parsing optional parameters and switches will quickly overwhelm your code logic. In this article I will show how … Python: Parsing command line arguments with argparse

Ubuntu: Auditing sudo commands and forwarding audit logs using syslog


Categories: Linux, Python

sudo provides users with temporary elevated privileges to perform operations.  No matter what your security philosophy, sudo is more than likely enabled on your system if even for a limited number of users. And if it is enabled, creating an audit log of exactly what was run (and who ran it) is essential to reporting.  … Ubuntu: Auditing sudo commands and forwarding audit logs using syslog

PuTTy: Bulk import PuTTy session definitions into the registry using Powershell


Categories: Python, Scripting

Putty is one of the first tools I install on any host or jumpbox.  And creating a saved session definition is extremely helpful so I can get the right window size, scrollback, keep alives, color scheme, etc. but creating each session definition by hand is time consuming. In this article, I will show you how … PuTTy: Bulk import PuTTy session definitions into the registry using Powershell

CloudFoundry: Beyond the maintenance page, delivering a response during service unavailability


Categories: Containers, DNE, Python

For most Cloud Foundry applications and services, you can avoid downtime for maintenance with a combination of following best practices for 12 factor app development and taking advantage of Cloud Foundry’s scaling and flexible routing to implement Blue-Green deployment. But there will still be times where an application, service, or shared backend component does not … CloudFoundry: Beyond the maintenance page, delivering a response during service unavailability

Zabbix: Accessing Zabbix using the py-zabbix Python module


Categories: Monitoring, Python

The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems. This fosters development of community-driven modules like the py-zabbix Python module, which is an easy way to automate Zabbix as well as send/retrieve metrics.

ELK: ElasticDump and Python to create a data warehouse job


Categories: Logging, Python

By nature, the amount of data collected in your ElasticSearch instance will continue to grow and at some point you will need to prune or warehouse indexes so that your active collections are prioritized. ElasticDump can assist in moving your indexes either to a distinct ElasticSearch instance that is setup specifically for long term data, or exporting … ELK: ElasticDump and Python to create a data warehouse job

SaltStack: Creating a ZooKeeper External Pillar using Python


Categories: DevOps, Python, Scripting

SaltStack has the ability to create custom states, grains, and external pillars.  There is a long list of standard external pillars ranging from those which read from local JSON files, to those that pull from EC2, MongoDB, etcd, and MySQL. In this article, we will use Apache ZooKeeper as the storage facility for our SaltStack … SaltStack: Creating a ZooKeeper External Pillar using Python

Python: Using Python, JSON, and Jinja2 to construct a set of Logstash filters


Categories: DevOps, Python, Scripting

Python is a language whose advantages are well documented, and the fact that it has become ubiquitous on most Linux distributions  makes it well suited for quick scripting duties. In this article I’ll go through an example of using Python to read entries from a JSON file, and from each of those entries create a … Python: Using Python, JSON, and Jinja2 to construct a set of Logstash filters