Migrating from one Python 3.x version to a newer 3.x minor version seems like it would just be a simple ‘apt install’ of the latest Python package. But you most likely have pip modules installed at a version specific ‘dist-packages’ or ‘site-packages’ directory, and those modules have to be freshly installed into the newer version if you want the same behavior.
Hopefully you are using venv at the individual project level, which will avoid most problems since a specific Python version and set of modules is explicit. But there is still the matter of Python modules installed at the System level (dist-packages) as well as User level (site-packages) that can be affected by a Python upgrade.
This article will show you how to capture the module name and version used by the older Python version, and use that to install matching modules to the newer version’s dist-packages/site-packages directory.
Note that ‘dist-packages’ is a Debian/Ubuntu convention.
Install newer Python version
For this article, let’s assume that we started with Python 3.8 (/usr/bin/python3.8) installed as the default on Ubuntu 22.04, and then we installed a newer Python version 3.9 as well (/usr/bin/python3.9). This does not wipe out or remove the older version, they will both exist in parallel.
# check existing Python 3.8 /usr/bin/python3.8 --version # system default 'python3' is Python 3.8 python3 --version # verify that 3.9 is available for install sudo apt update sudo apt-cache policy python3.9 # install Python 3.9 (3.8 will still remain) sudo apt install python3.9 -y # validate newer version now available /usr/bin/python3.9 --version
Install System and User level modules for testing migration
Using the default older Python version, let’s first install some small test modules at the System and User level that can serve as markers of our success later.
Install test module at System level
# install System level module as test
sudo pip3 install python-string-utils
# verify it is installed into older python 'dist-packages'
pip3 list -v | grep python-string-utils
# should run successfully
python3 -c "import string_utils; print(string_utils.is_string('hello'))"
Install test module at User level
# install User level module as test
pip3 install --user simple_env
# verify it is installed into older python 'site-packages'
pip3 list --user -v | grep simple-env
# should run successfully
foo=BAR python3 -c "import simple_env as se; print(se.get('foo'))"
Use ‘update-alternatives’ to manage preferred version
Now that we have two different versions of Python on the host, what we need is a way to assign a default ‘python3’ to point to the newer one. You could create your own symbolic links, but on Debian/Ubuntu the ‘update-alternatives‘ is the preferred method of administration.
# check for existing alternatives update-alternatives --query python3 # does system already have link for 'python3'? ls -l /usr/bin/python3 # If so, remove default symbolic link so we can use alternatives instead sudo unlink /usr/bin/python3 # create 3.9 with highest preference sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.8 10 sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.9 20 # check priorities update-alternatives --query python3 # we can see managed link now points to /etc/alternatives/python3 ls -l /usr/bin/python3 # verify that python3 points to newer version python3 --version
Migrating the System level Python modules
Although venv and user level modules are the preferred way of installing Python modules, it is not uncommon for some modules (e.g. those installed by apt) to be installed at the system level in the shared “/usr/local/lib/python<version>/dist-packages” directory.
old_version="python3.8"
new_version="python3.9"
# verify we are using the preferred newer version
python3 --version
# verify that pip is using the newer version
pip3 -V
# test module no longer works with newer Python (ModuleNotFoundError)
python3 -c "import string_utils; print(string_utils.is_string('hello'))"
# show list of system modules, notice no modules coming from newer python
pip3 list -v | grep "$new_version"
# capture name and version of modules used by older version
/usr/bin/$old_version -m pip list -v | grep -v "$HOME" | grep $old_version | awk -F' ' '{ printf "%s==%s\n", $1, $2 }' | tee /tmp/system-${old_version}-freeze.txt
Then we use this file of exact module names and versions from the older Python 3.8 to do a fresh install into Python 3.9.
# install modules into newer 'dist-packages'
sudo pip3 install -r /tmp/system-${old_version}-freeze.txt
# now module present in newer 'dist-packages'
pip3 list -v | grep "$new_version"
# now successful using newer Python
python3 -c "import string_utils; print(string_utils.is_string('hello'))"
All the Python modules that were loaded for the older version are now available to the newer version as well.
Migrating the user level Python modules
User level Python modules are the recommended way for normal users to deploy required Python modules, but that does not make it immune to issues of being tied to the Python version.
User level modules are installed at “~/.local/lib/python<version>/site-packages”, so if the ‘python3’ version and alias is updated at the system level, it can mean your previously downloaded modules are not picked up anymore.
old_version="python3.8"
new_version="python3.9"
# verify we are using the preferred newer version
python3 --version
# verify that pip is using the newer version
pip3 -V
# test of User level module no longer works (ModuleNotFoundError)
foo=BAR python3 -c 'import simple_env as se; print(se.get("foo"))'
# show list of user modules, notice no modules coming from newer python
pip3 list --user -v | grep $new_version
# capture name and version of modules in older version
/usr/bin/$old_version -m pip list --user -v | grep "$old_version" | awk -F' ' '{ printf "%s==%s\n", $1, $2 }' | tee ~/${old_version}-freeze.txt
Then we use this file of exact User module names and versions from the older Python 3.8 to do a fresh install for Python 3.9.
# install modules into newer 'site-packages'
pip3 install --user -r ~/${old_version}-freeze.txt
# now modules present in newer 'site-packages'
pip3 list --user -v | grep "$new_version"
# now successful using the newer Python
foo=BAR python3 -c 'import simple_env as se; print(se.get("foo"))'
REFERENCES
stackoverflow, how to move all modules to new version of python
stackoverflow,com, migrate site package modules installed with pip from older to newer version
fabianlee.org, python setting the preferred version using update-alternatives
github pyenv-pip-migrate, project for migrating pip modules
stackoverflow, how ‘dist-packages’ is a Debian specific convention
NOTES
If you want to remove alternatives
sudo update-alternatives --remove python3 /usr/bin/python3.8 sudo update-alternatives --remove python3 /usr/bin/python3.9 # OR sudo update-alternatives --remove-all python3
checking library directories
python3 -c "import sysconfig; print(sysconfig.get_paths()['purelib'])" python3 -c "import sysconfig; print(sysconfig.get_paths()['platlib'])"
where python will look for packages, link
python3 -c "import sys; print('\n'.join(sys.path))"