If you have warning messages coming from apt about an invalid signature verification and EXPKEYSIG, then it is likely that a signing key for one of the remote apt repo has expired.
Below is an example coming from an expired podman package at the “download.opensuse.org” repo.
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04 InRelease: The following signatures were invalid: EXPKEYSIG 4D64390375060AA4 devel:kubic OBS Project <devel:kubic@build.opensuse.org> W: Failed to fetch http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/InRelease The following signatures were invalid: EXPKEYSIG 4D64390375060AA4 devel:kubic OBS Project <devel:kubic@build.opensuse.org>
Using the hex value from the warning message, prepare a variable for our upcoming commands.
# value pulled from warning message EXPKEYSIG hexid=4D64390375060AA4 # install package for decoding PGP sudo apt install pgpdump -y
Deprecation of centralized system keyring
Before you follow this article, it is important to understand that the centralized system keyring was deprecated starting in Ubuntu 21 (but is still available in Ubuntu 22).
The legacy implementation had the trusted system keyring file at “/apt/trusted.gpg” or multiple files in “/apt/trusted.gpg.d/”. The primary issue was that a PGP key added to this file or directory would mean that apt trusted any package coming from any location, as long as it referenced any of these keys. This was a security risk.
The new implementation is that individual keys are placed into their own file in the “/usr/share/keyrings/” directory. The apt repo definition files (in /etc/apt) each use a ‘signed-by’ attribute to reference the local file path pointing to a specific key. This makes the trust explicit, with each remote repo trusting a specific individual key.
As a concrete example of the recommended implementation, below is the apt repo definition found at “/etc/apt/sources.list.d/github-cli.list” for the github CLI apt repo. Notice the ‘signed-by’ attribute referring to a specific local key.
deb [arch=amd64 signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main
Verify local expired PGP key
Option #1: For deprecated trusted keyring
If your system is still using the deprecated system keyring, then apt-key can be used to view the expiration.
# show ownership and expiration date from local keyring
apt-key export $hexid 2>/dev/null | pgpdump | grep -E "User ID -|expiration" -A1
# will also confirm owner and expiration date from local keyring
apt-key list | grep -i "${hexid: -8:4} ${hexid: -4}" -C3
Option #2: For independent key files
If the apt repo file definition is using the recommended ‘signed-by’, then you can view each expiration.
# view path locations of PGP keys, ignore '.save' which are backup files sudo grep -sr "signed-by" /etc/apt | grep -v save # get all PGP path locations pgp_paths=$(sudo grep -srl "signed-by" /etc/apt | grep -v save | xargs cat | grep -Po 'signed-by=\K([^\]" ])*') # view expiration dates of PGP files sudo apt install pgpdump -y for f in $pgp_paths; do echo;echo "==$f"; pgpdump $f | grep -E "User ID -|expiration" -A1; done
Find newest PGP key
The developers of the package should have provided the newest PGP key by either:
- Uploading the new PGP key to the web-accessible apt repo; you can perform a manual web search for the apt key
- Uploading the PGP key to a central keyserver such as keyserver.ubuntu.com; you can query the keyserver for the apt key
Option #1: New PGP key available on package web site
The naming of the PGP keys for each web-accessible apt repo is not standardized, so I find the easiest discovery method is to manually point your browser at the search engine duckduckgo.com and search the site mentioned in the warning for a PGP public key marker (Google does not find these files as accurately).
Here is an example using our original example of the podman key from opensuse.
site:download.opensuse.org apt “BEGIN PGP PUBLIC KEY BLOCK”
Copy and paste this location into a variable value like below to prepare for the next section.
# define variable based on only search result remote_key_file=https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_22.04/Release.key
As another example, here is the search expression to find the Google gcloud and kubectl keys.
site:packages.cloud.google.com apt “BEGIN PGP PUBLIC KEY BLOCK”
# define variable based on top search result from above remote_key_file=https://packages.cloud.google.com/apt/doc/apt-key.gpg.asc
Option #2: New PGP key available on central keyserver
The developers of the package may have uploaded the newest PGP key to a central keyserver such as keyserver.ubuntu.com. You can use the ‘hexid’ value from the sections above to search the keyserver site.
echo "going to search for: $hexid"
curl -fs "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x${hexid}" | pgpdump | grep -E "User ID -|expiration" -A1
If the expiration date is exceeded, then you cannot use this key. You are probably not the first to hit this problem, search the issues logs of the specific project and log an issue if necessary.
However, if the PGP key’s expiration is newer (or expiration does not exist), then you can define the variable value like below to prepare for the next section.
remote_key_file="https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x${hexid}"
Have apt trust the PGP key
Now that you have correctly identified the ‘remote_key_file’ location, you can have apt trust this key in one of two ways.
Option #1: Legacy method of adding to system trusted keyring
This method is deprecated as of Ubuntu 21, but ‘apt-key’ can still be used to add the key to your system’s trusted keyring. This actually makes the key trusted by any and all apt repository, which is a driving factor in its deprecation.
# validate same ownership as original, and newer expiration date curl -s $remote_key_file | pgpdump | grep -E "User ID -|expiration" -A1 # add new key curl -s $remote_key_file | sudo apt-key add - # validate warning message is no longer thrown sudo apt update
Option #2: Recommended method of ‘signed-by’ for each apt repo
The recommended method of trust is to add a ‘signed-by’ parameter in the apt repo definition file pointing at a local key path. This ties a specific repo to a specific key and is therefore more secure.
The commands below use our original example from ‘download.opensuse.org’
# validate same ownership as original, and newer expiration date curl -s $remote_key_file | pgpdump | grep -E "User ID -|expiration" -A1 # example apt repo we are adding domain=download.opensuse.org # save key file locally sudo curl -fsSL $remote_key_file -o /usr/share/keyrings/$domain.asc # MUST have 644 at least or apt cannot find key sudo chmod 644 /usr/share/keyrings/$domain.asc # find apt repo definition file repo_file=$(sudo grep -srl $domain /etc/apt | grep -v save) echo "apt repo for $domain found at: $repo_file" # manually add '[signed-by=<path>]' parameter which is path to pgp file (.asc) sudo vi $repo_file # show new apt repo definition cat $repo_file deb [signed-by=/usr/share/keyrings/download.opensuse.org.asc] http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04/ / # optional step, deleting key from deprecated system keyring sudo apt-key del $hexid # warning message should no longer be thrown sudo apt update
REFERENCES
debian.org, current scheme for package signature checks
debian.org, apt PGP key not in standard location
askubuntu.com, script for finding all legacy keys and converting to new format
askubuntu.com, finding the signing key for a ppa launchpad
itfoss.com, explanation of legacy keyrings, dearmor, deprecation of ‘apt-key’
omgubuntu.co.uk – fix apt-key deprecation error with export and dearmor
ubuntu man pages, apt-key deprecated in Ubuntu 21
jeffmotoretta.ca – why apt-key is deprecated
NOTES
Alternate central keysites
pgpkeys.mit.edu, pool.sks-keyservers.net
Check if the developers have loaded newer key into keyserver
curl -s "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x${hexid}" | pgpdump | grep -E "User ID -|expiration" -A1