Keeping the Ubuntu system-level Python version and modules independent from those desired at each project level is a difficult task best managed by a purpose-built tool. There are many solutions in the Python ecosystem, but one that stands out for simplicity is pyenv and pyenv-virtualenv. pyenv allows you to install and switch between different versions … Ubuntu: pyenv for managing multiple Python versions and environments
Although you could use brew to install Python directly, the cleaner way to manage Python versions and isolate Python virtual environments is by using pyenv and pyenv-virtualenv. pyenv allows you to install and switch between different versions of Python, while pyenv-virtualenv provides isolation of pip modules, for independence between projects.
Flask is a suitable web server during development, but if you are going to deploy in a production environment, a Python WSGI server such as Gunicorn should be used. This also applies to Python Flask apps deployed to GCP Cloud Run. Gunicorn is necessary to tune the worker and thread count of each instance to … GCP: deploying a Python WSGI Gunicorn app on Cloud Run
If you are getting a warning similar to below when running a Python3 application: /usr/lib/python3/dist-packages/paramiko/transport.py:219: CryptographyDeprecationWarning: Blowfish has been deprecated This can be resolved by upgrading to the latest paramiko module. # check current version then upgrade pip3 show paramiko pip3 install paramiko –upgrade # check upgraded version pip3 show paramiko In my case, this … Python: fixing ‘CryptographyDeprecationWarning: Blowfish has been deprecated’
The Plotly graphing library has a wide array of visualizations for datasets. And it has native support for Pandas DataFrame, which makes it convenient for datasets coming from a wide range of sources. One visualization that I find particularly useful for hierarchical data is the TreeMap, which can group on data lineage, uses area as … Python: TreeMap visualization of hierarchical Pandas DataFrame
Python issues warning messages to users when an exception or halting the program is not warranted but there is still useful information that should be relayed to the end user. Instead of following your first instincts to suppress these warnings you should instead resolve the root issue. However, if you already understand the problem and … Python: suppressing warnings from Python applications
Migrating from one Python 3.x version to a newer 3.x minor version seems like it would just be a simple ‘apt install’ of the latest Python package. But you most likely have pip modules installed at a version specific ‘dist-packages’ or ‘site-packages’ directory, and those modules have to be freshly installed into the newer version … Python: migrating pip modules to newer Python version on Ubuntu
Flask OIDC is an extension to the popular Flask web framework that enables OAuth2/OIDC for your application. The base project does not support ADFS, but I have create a personal fork of this module that supports Windows 2019 ADFS as the OAuth2 Authentication Server. In this article, we will exercise the OAuth2 Authorization Code flow. … Python: Flask-OIDC protecting Client App and Resource Server using Windows 2019 ADFS
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads on a production Kubernetes cluster, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a series of previous articles, we … Python: New Relic Agent for Gunicorn app deployed on Kubernetes
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. If you are running these types of workloads in production, you should consider an observability platform such a New Relic to ensure availability, service levels, and visibility into transactions and logging. In a previous article, we created a Docker image of … Python: New Relic instrumentation for Flask app deployed with Gunicorn
Gunicorn is a WSGI HTTP server commonly used to run Flask applications in production. Running Flask applications directly is great for development and testing of the basic request/response flow, but you need gunicorn to handle production level loads, concurrency, logging, and timeouts. In this article, I will show you how to build a Docker image … Python: Building an image for a Flask app served from Gunicorn
If the dataset you want to analyze with pandas is coming from a normalized relational database, then you can use ‘pandas.read_sql‘ to pull the data directly. In this article, we will deploy a small MariaDB instance with Docker and show how we can create DataFrame directly from a single table or from a join between … Python: constructing a DataFrame from a relational database with pandas
If your data is coming from CSV dumps of relational database tables, then you probably need to flatten the data before starting the training or analysis. It requires an understanding of the relational schema so you can specify the foreign key lookups, but from that pandas can reconstruct the relationships and produce a single flattened … Python: flattening exported csv relational data with pandas
Although there are Python modules [1,2,3] specially suited for displaying text to the console in color, if you want a quick no-dependency method then you can use ANSI color codes directly. Here is an example of printing a line in green, then red. print(“\033[0;32mOK this is green\033[00m”) print(“\033[0;31mERROR this is red\033[00m”) Additional color codes can … Python: printing in color using ANSI color codes
Whether it is the most recent log file, image, or report – sometimes you will need to find the most recently modified file in a directory. The example below finds the latest file in the “/tmp” directory. import os import glob # get list of files that matches pattern pattern=”/tmp/*” files = list(filter(os.path.isfile, glob.glob(pattern))) # … Python: find the most recently modified file matching a pattern
Most of the modern cloud platforms and utilities have us manipulate either JSON or YAML configuration files. And when you start dealing with real world scenarios with hundreds of lines of embedded data structures it is too difficult and error-prone to manually inspect indentation levels to determine the exact dotted or json path to an … Python: converting JSON to dot notation for easier path determination
Even though Python2 reached EOL in January 2020, there are still lagging applications and modules that force its continued use on certain systems. If that is your situation, you need the ability to force preference to Python3 unless a program explicitly invokes Python2. If you are using Ubuntu Focal 20, then you have an easy … Python: Setting the preferred Python version on Bionic 18 and Focal 20
As with any programming language, Python has a multitude of ways to accomplish the same task. In this article, I will explore the idea of taking a string and checking if it ‘startswith’ any of the strings from a predetermined list. As context for the example code, RFC1918 sets out several IPv4 ranges that can … Python: exploring the use of startswith against a list: tuple, regex, list comprehension, lambda
The pika module for Python provides an easy interface for creating exchanges and queues as well as producers/consumers for RabbitMQ . In this article, I will provide examples of a producer and consumer written in Python3. All source code is available on github.
Python list comprehension provides a concise and pythonic way of generating lists. It allows you to select a subset based on a filter and apply a function to each element. Because although navigating every list entry, filtering certain records, and applying functions to specific fields is doable using basic ‘for’ statements, ‘if’ logic, and temporary … Python: Examples of list comprehension
If you are looking for basic HTTP file uploading using Flask, you can use the code in my python-flask-upload-files project on github as an example. This project exercises multiple scenarios: Upload a single file in chunked mode (not forced to save entirety to disk) Upload a single file Upload multiple files Error message shown when … Python: Using Flask to upload files
If you are using Flask to generate dynamic content of significant size, such as large binary images/pdf or large text-based datasets, then you need to consider streaming to minimize the memory footprint of Flask and preserve scalability. Using an inner generate function and a yield allows Flask to return chunks of data back to the … Python: Using Flask to stream chunked dynamic content to end users
Although sending a plain text message through an unauthenticated SMTP relay might be easily done with a few lines of Python code, sending rich HTML email through authenticated relays or via the Gmail API using OAuth2 requires a bit of work. In this article, I will provide sample code for sending rich HTML email with … Python: Sending HTML emails via Gmail API or SMTP relay
The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. Building on my earlier article on the basics of introspection, I will now show how to inject custom … Python: Using a custom decorator to inspect function arguments
The ability to use introspection to dynamically determine the current code context and frame/stack can be a powerful tool for a developer. Introspection can be applied for debugging, logging, metrics collection, or method overloading based on type. I will show a simple example of its usage in this article.
Using subprocess.Popen, subprocess.call, or subprocess.check_output will all invoke a process using Python, but if you want live output coming from stdout you need use subprocess.Popen in tandem with the Popen.poll method. In this article I will show how to invoke a process from Python and show stdout live without waiting for the process to complete. … Python: Getting live output from subprocess using poll
Especially when writing small Python utility programs, it can be tempting to use sys.argv to retrieve a small list of positional parameters from the command line. But small utilities turn into successful tools over time, and manually parsing optional parameters and switches will quickly overwhelm your code logic. In this article I will show how … Python: Parsing command line arguments with argparse
In a previous post, I described how to install and use the ‘govc‘ tool – which is a command line tool for working with VMware vCenter. When you do a listing on a virtual machine “govc ls -l -json <vmpath>”, you are returned back a very detailed JSON data structure that contains all the information … Python: JSONPath to extract vCenter information using govc
There are many tools that utilize json, and when it is relatively simple you can use standard modules or even custom coding to pull out the desired portions. But once these data structures reach a certain level of complexity you really should consider a Python module that implements JSONPath (analogous to xPath for XML). The … Python: Querying JSON files with JSONPath using jsonpath_rw_ext
Amazon EC2 provides a web interface for managing IaaS, but for repeatable infrastructure deployment what you really want is the ability to deploy and manage this infrastructure using an API or command line tool. The AWS SDK for Python (Boto3) provides a lower level as well as resource level API for managing and creating infrastructure. This … AWS: Installing the AWS SDK for Python on Ubuntu
sudo provides users with temporary elevated privileges to perform operations. No matter what your security philosophy, sudo is more than likely enabled on your system if even for a limited number of users. And if it is enabled, creating an audit log of exactly what was run (and who ran it) is essential to reporting. … Ubuntu: Auditing sudo commands and forwarding audit logs using syslog
The mako template engine is a powerful module for Python used by many large sites including reddit.com. You simply push the variables you want into a context and then merge that with a text template. Additionally, mako allows you to take advantage of the full power of the Python language and libraries by allowing calls … Python: Calling python functions from mako templates
Putty is one of the first tools I install on any host or jumpbox. And creating a saved session definition is extremely helpful so I can get the right window size, scrollback, keep alives, color scheme, etc. but creating each session definition by hand is time consuming. In this article, I will show you how … PuTTy: Bulk import PuTTy session definitions into the registry using Powershell
For most Cloud Foundry applications and services, you can avoid downtime for maintenance with a combination of following best practices for 12 factor app development and taking advantage of Cloud Foundry’s scaling and flexible routing to implement Blue-Green deployment. But there will still be times where an application, service, or shared backend component does not … CloudFoundry: Beyond the maintenance page, delivering a response during service unavailability
The open-source Zabbix monitoring solution has a REST API that provides the ability for deep integrations with your existing monitoring, logging, and alerting systems. This fosters development of community-driven modules like the py-zabbix Python module, which is an easy way to automate Zabbix as well as send/retrieve metrics.
By nature, the amount of data collected in your ElasticSearch instance will continue to grow and at some point you will need to prune or warehouse indexes so that your active collections are prioritized. ElasticDump can assist in moving your indexes either to a distinct ElasticSearch instance that is setup specifically for long term data, or exporting … ELK: ElasticDump and Python to create a data warehouse job
SaltStack has the ability to create custom states, grains, and external pillars. There is a long list of standard external pillars ranging from those which read from local JSON files, to those that pull from EC2, MongoDB, etcd, and MySQL. In this article, we will use Apache ZooKeeper as the storage facility for our SaltStack … SaltStack: Creating a ZooKeeper External Pillar using Python
Python is a language whose advantages are well documented, and the fact that it has become ubiquitous on most Linux distributions makes it well suited for quick scripting duties. In this article I’ll go through an example of using Python to read entries from a JSON file, and from each of those entries create a … Python: Using Python, JSON, and Jinja2 to construct a set of Logstash filters
SaltStack grains are used for relatively static information such as operating system, IP address, and other system properties. They are also useful for targeting minions, for example whether a system is part of dev/test/prod, or a flag on whether it falls under LifeScience or HIPAA regulation. In this article we will implement a custom grain … SaltStack: Creating a Custom Grain using Python