Although you could use brew to install Python directly, the cleaner way to manage Python versions and isolate Python virtual environments is by using pyenv and pyenv-virtualenv. pyenv allows you to install and switch between different versions of Python, while pyenv-virtualenv provides isolation of pip modules, for independence between projects.
The Plotly graphing library has a wide array of visualizations for datasets. And it has native support for Pandas DataFrame, which makes it convenient for datasets coming from a wide range of sources. One visualization that I find particularly useful for hierarchical data is the TreeMap, which can group on data lineage, uses area as … Python: TreeMap visualization of hierarchical Pandas DataFrame
Migrating from one Python 3.x version to a newer 3.x minor version seems like it would just be a simple ‘apt install’ of the latest Python package. But you most likely have pip modules installed at a version specific ‘dist-packages’ or ‘site-packages’ directory, and those modules have to be freshly installed into the newer version … Python: migrating pip modules to newer Python version on Ubuntu
Flask OIDC is an extension to the popular Flask web framework that enables OAuth2/OIDC for your application. The base project does not support ADFS, but I have create a personal fork of this module that supports Windows 2019 ADFS as the OAuth2 Authentication Server. In this article, we will exercise the OAuth2 Authorization Code flow. … Python: Flask-OIDC protecting Client App and Resource Server using Windows 2019 ADFS
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads on a production Kubernetes cluster, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a series of previous articles, we … Python: New Relic Agent for Gunicorn app deployed on Kubernetes
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads in production, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a previous article, we created a Docker image of … Python: New Relic instrumentation for Flask app deployed with Gunicorn
GCP Cloud Functions have taken a step forward with the 2nd generation release. One of the biggest architectural differences is that now multiple request can run concurrently on a single instance, enabling large traffic loads. In this article, I will show you how to deploy a simple Python Flask web server as a 2nd gen … GCP: Deploying a 2nd gen Python Cloud Function and exposing from an HTTPS LB
Although there are Python modules [1,2,3] specially suited for displaying text to the console in color, if you want a quick no-dependency method then you can use ANSI color codes directly. Here is an example of printing a line in green, then red. print(“\033[0;32mOK this is green\033[00m”) print(“\033[0;31mERROR this is red\033[00m”) Additional color codes can … Python: printing in color using ANSI color codes
Most of the modern cloud platforms and utilities have us manipulate either JSON or YAML configuration files. And when you start dealing with real world scenarios with hundreds of lines of embedded data structures it is too difficult and error-prone to manually inspect indentation levels to determine the exact dotted or json path to an … Python: converting JSON to dot notation for easier path determination
In a previous post, I described the Kubernetes Downward API and how it allows us to inject pod/container metadata into our runtime container. In this article, I’ll show how you can read the environment variables and mounted files from inside a containerized Python based application.
As with any programming language, Python has a multitude of ways to accomplish the same task. In this article, I will explore the idea of taking a string and checking if it ‘startswith’ any of the strings from a predetermined list. As context for the example code, RFC1918 sets out several IPv4 ranges that can … Python: exploring the use of startswith against a list: tuple, regex, list comprehension, lambda
The pika module for Python provides an easy interface for creating exchanges and queues as well as producers/consumers for RabbitMQ . In this article, I will provide examples of a producer and consumer written in Python3. All source code is available on github.
Python list comprehension provides a concise and pythonic way of generating lists. It allows you to select a subset based on a filter and apply a function to each element. Because although navigating every list entry, filtering certain records, and applying functions to specific fields is doable using basic ‘for’ statements, ‘if’ logic, and temporary … Python: Examples of list comprehension
This may come as a surprise, but Bash only supports integer arithmetic natively. If you need to perform calculations on floating point numbers, you will need to call out to a utility program like bc or Python. As a quick proof to yourself, try multiplying integers, then do the same with floating point numbers. $ … Bash: Performing floating arithmetic using bc
If you are using Flask to generate dynamic content of significant size, such as large binary images/pdf or large text-based datasets, then you need to consider streaming to minimize the memory footprint of Flask and preserve scalability. Using an inner generate function and a yield allows Flask to return chunks of data back to the … Python: Using Flask to stream chunked dynamic content to end users
Although sending a plain text message through an unauthenticated SMTP relay might be easily done with a few lines of Python code, sending rich HTML email through authenticated relays or via the Gmail API using OAuth2 requires a bit of work. In this article, I will provide sample code for sending rich HTML email with … Python: Sending HTML emails via Gmail API or SMTP relay
The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. Building on my earlier article on the basics of introspection, I will now show how to inject custom … Python: Using a custom decorator to inspect function arguments
The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. I will show a simple example of its usage in this article.
Using subprocess.Popen, subprocess.call, or subprocess.check_output will all invoke a process using Python, but if you want live output coming from stdout you need use subprocess.Popen in tandem with the Popen.poll method. In this article I will show how to invoke a process from Python and show stdout live without waiting for the process to complete. … Python: Getting live output from subprocess using poll
Especially when writing small Python utility programs, it can be tempting to use sys.argv to retrieve a small list of positional parameters from the command line. But small utilities turn into successful tools over time, and manually parsing optional parameters and switches will quickly overwhelm your code logic. In this article I will show how … Python: Parsing command line arguments with argparse
In a previous post, I described how to install and use the ‘govc‘ tool – which is a command line tool for working with VMware vCenter. When you do a listing on a virtual machine “govc ls -l -json <vmpath>”, you are returned back a very detailed JSON data structure that contains all the information … Python: JSONPath to extract vCenter information using govc
There are many tools that utilize json, and when it is relatively simple you can use standard modules or even custom coding to pull out the desired portions. But once these data structures reach a certain level of complexity you really should consider a Python module that implements JSONPath (analogous to xPath for XML). The … Python: Querying JSON files with JSONPath using jsonpath_rw_ext
Amazon EC2 provides a web interface for managing IaaS, but for repeatable infrastructure deployment what you really want is the ability to deploy and manage this infrastructure using an API or command line tool. The AWS SDK for Python (Boto3) provides a lower level as well as resource level API for managing and creating infrastructure. This … AWS: Installing the AWS SDK for Python on Ubuntu
The mako template engine is a powerful module for Python used by many large sites including reddit.com. You simply push the variables you want into a context and then merge that with a text template. Additionally, mako allows you to take advantage of the full power of the Python language and libraries by allowing calls … Python: Calling python functions from mako templates
In a production datacenter it would not be uncommon for internet access to be limited to domains whitelisted on a web proxy such as Squid. If this is the case, and you are using pip to install packages, then you will need to: Have your Squid administrators whitelist pypi.python.org Add the “–proxy” switch when invoking … Python: Using pip with a squid proxy
Putty is one of the first tools I install on any host or jumpbox. And creating a saved session definition is extremely helpful so I can get the right window size, scrollback, keep alives, color scheme, etc. but creating each session definition by hand is time consuming. In this article, I will show you how … PuTTy: Bulk import PuTTy session definitions into the registry using Powershell
The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems. This fosters development of community-driven modules like the py-zabbix Python module, which is an easy way to automate Zabbix as well as send/retrieve metrics.
By nature, the amount of data collected in your ElasticSearch instance will continue to grow and at some point you will need to prune or warehouse indexes so that your active collections are prioritized. ElasticDump can assist in moving your indexes either to a distinct ElasticSearch instance that is setup specifically for long term data, or exporting … ELK: ElasticDump and Python to create a data warehouse job
SaltStack has the ability to create custom states, grains, and external pillars. There is a long list of standard external pillars ranging from those which read from local JSON files, to those that pull from EC2, MongoDB, etcd, and MySQL. In this article, we will use Apache ZooKeeper as the storage facility for our SaltStack … SaltStack: Creating a ZooKeeper External Pillar using Python
Python is a language whose advantages are well documented, and the fact that it has become ubiquitous on most Linux distributions makes it well suited for quick scripting duties. In this article I’ll go through an example of using Python to read entries from a JSON file, and from each of those entries create a … Python: Using Python, JSON, and Jinja2 to construct a set of Logstash filters
SaltStack grains are used for relatively static information such as operating system, IP address, and other system properties. They are also useful for targeting minions, for example whether a system is part of dev/test/prod, or a flag on whether it falls under LifeScience or HIPAA regulation. In this article we will implement a custom grain … SaltStack: Creating a Custom Grain using Python