When automating software and infrastructure, it is not uncommon to need to supply a user id and password for installation or other operations. While it is certainly possible to pass these plaintext credentials directly in the state, this is not best practice.
# not best practice!!!
testdb_user:
mysql_user.present:
- name: frank
- password: "test3rdb"
- host: localhost
There are several issues with this approach.
States are accessible to all minions, so using the example state file above, not only does your mysql host have access to the password, but all your salt managed hosts. Additionally, in the world of infrastructure-as-code, we have to assume that it is checked into source control, and the plaintext password is now accessible to anyone with repository access. As a last point, even if all this was acceptable, a pillar is a more appropriate place to store this information because it provides flexibility to do things like have a different password for frank in dev/test/prod environments.
Luckily we can address all these issues by using the SaltStack GPG renderer. It provides secure encryption/decryption of pillar data, limited to only those minions that absolutely require it.
Generate key pair on Salt Master
Generating a key pair takes a certain amount of entropy, or randomness. Especially on a virtualized machine, you can find that not enough operations have been executed to generate GPG keys. Installing the ‘rng-tools’ package can address this issue [1,2]. For Ubuntu, run the following:
# apt-get install python-gnupg -y # apt-get install rng-tools # rngd -r /dev/urandom
Then generate the key pair:
# mkdir -p /etc/salt/gpgkeys # chmod 0700 /etc/salt/gpgkeys # gpg --gen-key --homedir /etc/salt/gpgkeys
The gpg utility will ask several questions, you can customize or accept the default values for RSA key size of 2048 with no expiration. Then when it asks for real name, this is the identifier for your key and it is common to use the hostname. You can leave email address and comment blank by just pressing <ENTER>. Then when it asks for a passphrase, just press <ENTER> again twice. If you specify a passphrase, it won’t work with Salt because Salt works in non-interactive mode.
At this point you will have the following files in your /etc/salt/gpgkeys directory: pubring.gpg, pubring.gpg~, random_seed, secring.gpg, and trustdb.gpg.
The list of keys stored in the general GPG keyring as well as our new directory can be see with the commands below.
# gpg --list-keys # gpg --homedir /etc/salt/gpgkeys --list-keys
When you list the keys in /etc/salt/gpgkeys, you should see that one of them has the identifier you typed as the ‘Real name’ when generating the key pair. This is the identifier used in future operations.
Key Distribution
At this stage, it is a good idea to export your private key for safe keeping and export your public key for general distribution.
# gpg --homedir /etc/salt/gpgkeys --export-secret-keys --armor > exported_private.key
You will want to put this secret key into the best secure, encrypted-at-rest solution you have (whether this is a USB stick in a vault or Thycotic). And then export the public key, which you can check into source control or distribute in any way you choose.
# gpg --homedir /etc/salt/gpgkeys --armor --export > /etc/salt/gpgkeys/exported_pubkey.gpg
Now import the public key we just generated into the general public key keyring:
# gpg --import /etc/salt/gpgkeys/exported_pubkey.gpg
The resulting output will tell you that one key was imported, and then it will show you the identifier used, which corresponds to the ‘Real name’ used when generating the key pair.
gpg: /root/.gnupg/trustdb.gpg: trustdb created gpg: key BA792F0B: public key "saltserver" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1)
Encrypt a Secret
To encrypt a secret, run gpg, being sure to pass the identifier (i.e. Real name) as a parameter so it knows which key to use. My identifier is ‘saltserver’, obviously you need to replace this with yours.
# echo -n "supersecret" | gpg --armor --batch --trust-model always --encrypt -r "saltserver"
The resulting output will be a Base64 encoded message that can be put into a pillar sls file. If instead of a string, you have a file such as a private ssh key that needs encoding the command would look like this:
# cat ssh-private.key | gpg --armor --encrypt -r "saltserver"
Encoded Secret in Pillar
When creating the pillar .sls file, you need to do a couple of things. First, specify the rendering order in the first shebang line so that gpg is done last. And then when you specify a multiline string using the pipe, make sure you add the proper number of spaces so that yaml indentation is honored (the line below are truncated because long lines messed up the blog formatting).
#!jinja|yaml|gpg a-secret: | -----BEGIN PGP MESSAGE----- Version: GnuPG hQEMAweRHKaPCfNeAQf9GLTN16hCfXAbPw... QuetdvQVLFO/HkrC4lgeNQdM6D9E8PKonM... cnEfmVexS7o/U1VOVjoyUeliMCJlAz/30R... RhkhC0S22zNxOXQ38TBkmtJcqxnqT6YWKT... m4wBkfCAd8Eyo2jEnWQcM4TcXiF01XPL4z... Gr9v2DTF7HNigIMl4ivMIn9fp+EZurJNiQ... FKlwHiJt5yA8X2dDtfk8/Ph1Jx2TwGS+lG... skqmFTbOiA===Eqsm -----END PGP MESSAGE-----
Now, if you assign this pillar to a minion (e.g. ‘myminion’), you should be able to see the unencrypted value if you list the pillar values:
salt 'myminion' pillar.items
Or directly from the salt minion, you can see the unencrypted value.
salt-call pillar.items salt-call pillar.get a-secret
Note that only the minions that are assigned this pillar would have access to the secret.
Using the secret in a state file
To use this encrypted value in a state, you need to pull it from the pillar, as shown below.
test-gpg:
cmd.run:
- name: echo {{ salt['pillar.get']('a-secret') }}
As stated earlier, all minions have access to state files, but now the secret is no longer divulged in the .sls, and unless a minion is assigned the pillar, the value will be empty.
REFERENCES
https://docs.saltstack.com/en/latest/topics/best_practices.html#storing-secure-data
https://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.gpg.html
https://clinta.github.io/random-local-passwords/
https://help.ubuntu.com/community/GnuPrivacyGuardHowto
http://serverfault.com/questions/214605/gpg-not-enough-entropy
http://www.clausconrad.com/blog/using-the-gpg-renderer-to-protect-salt-pillar-items
http://irtfweb.ifa.hawaii.edu/~lockhart/gpg/gpg-cs.html
https://lwn.net/Articles/525459/
https://www.2uo.de/myths-about-urandom/
If you get error messages in your Salt execution that GPG cannot be found, make sure you install the python-gnupg package.
NOTES
GPG for encryption/decryption without SaltStack
sudo apt-get install python-gnupg rng-tools -y sudo rngd -r /dev/urandom mkdir gpgkeys chmod 700 gpgkeys gpg --gen-key --homedir gpgkeys (accept all default, "testing" for real name) (no passphrase, <ENTER> twice with empty string) gpg --homedir gpgkeys --list-keys to encrypt: echo -n "MyP4ss!" | gpg --armor --batch --trust-model always --encrypt --homedir gpgkeys -r "testing" > test.enc to decrypt: gpg --homedir gpgkeys -d test.crypt 2>/dev/null