Configuration Management tools like SaltStack are invaluable for managing infrastructure at scale. Even in the growing world of containerization, there is the need for bulk automation.
This article will detail installation of Salt SSH which leverages the power of SaltStack without the requirements for an agent install.
Installation
The first step is to add the SaltStack repository and key:
$ wget -qO - https://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest/SALTSTACK-GPG-KEY.pub | sudo apt-key add - $ sudo apt-get install apt-transport-https ca-certificates -y $ echo "deb http://repo.saltstack.com/apt/ubuntu/16.04/amd64/latest $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/saltstack.list
Then refresh the repository and install SaltStack SSH:
$ sudo apt-get update $ sudo apt-cache policy salt-ssh $ sudo apt-get install salt-ssh -y
Now create default directories to support the state and pillar files:
$ sudo mkdir -p /srv/salt $ sudo mkdir -p /srv/pillar
SSH Connection to remote hosts
salt-ssh makes ssh connections to remote hosts either through username/password authentication or the use of a public/private key pair.
In the case of username/password authentication, the values can be provided in the salt-ssh roster or prompted.
A remote host can also use a public/private key pair for authentication. The remote host will have the public side of the key on its local file system. You will need the private side of the key on the salt-ssh server to present as the authentication token.
salt-ssh via user/password login
The most basic method of connecting to a remote host via ssh is by specifying the username and password. This is assuming that password based authentication is allowed.
The first thing you should do is validate that you can use a regular ssh client to establish a connection.
$ ssh <user>@<host>
In my case, I’m hitting an Ubuntu 14.04 host named ‘trusty1’, so I use the following command, and after accepting the fingerprint of the remote host and entering in the password manually, I gain ssh access to the system.
$ ssh vagrant@trusty1
This tells me that salt-ssh should be able to connect to the host also, so now I add this host to the “/etc/salt/roster” salt-ssh configuration file.
trusty1: host: 192.168.1.124 user: vagrant passwd: vagrant timeout: 5
And now run a quick connectivity test against the host which should return output similar to below.
$ sudo salt-ssh -i trusty1 test.version trusty1: 2018.3.2
The -i switch tells salt-ssh to ignore the hostkey prompt, it actually adds the entry to /root/.ssh/known_hosts (because we are running it as sudo). It won’t be necessary to specify the switch for this host anymore.
Note that if your target host does not have python2 installed (such as Ubuntu 16.04 Xenial), then you will see an error like below.
trusty1:
----------
retcode:
10
stderr:
ERROR: Unable to locate appropriate python command
stdout:
ERROR: salt requires python 2.6 or newer on target hosts, must have same major version as origin host
The easiest way to address this is to use a salt-ssh raw ssh connection to install the ‘python-minimal’ package, which satisfies this requirement.
$ sudo salt-ssh trusty1 -i -r 'sudo apt-get update; sudo apt-get install python-minimal -y'
If you do not want to put the plaintext password into the roster file, you can also use the ‘-askpass’ switch at the console.
salt-ssh via public/private key pair
The other way that you can authenticate is with a public/private keypair. The private key is kept secure on the ssh client side, and the public side is stored on the remote host in “~/.ssh/authorized_keys”.
The first thing you should do is validate that you can use a regular ssh client to establish a connection using a certificate.
$ ssh -i <privatekey> <user>@<host>
In my case, I’m hitting a vagrant built Ubuntu 14.04 host named ‘trusty1’, so I use the following command and gain ssh access to the system.
$ ssh -i .vagrant/machines/default/virtualbox/private_key vagrant@trusty1
This tells me that salt-ssh should be able to connect to the host also using the “private_key” file as authentication.
Now add this host entry to the “/etc/salt/roster” salt-ssh configuration file. I’ve also copied the “private_key” file to a local file available to salt-ssh named “trusty1.rsa” that has permissions of 400.
trusty1: host: 192.168.1.124 user: vagrant priv: /home/vagrant/trusty1.rsa passwd: vagrant timeout: 5
And now run a quick connectivity test against the host which should return output similar to below.
$ sudo salt-ssh -i trusty1 test.version trusty1: 2018.3.2
The -i switch tells salt-ssh to ignore the hostkey prompt, it actually adds the entry to /root/.ssh/known_hosts (because we are running it as sudo). It won’t be necessary to specify the switch for this host anymore.
Default salt-ssh key pair
The first time you use salt-ssh, it automatically creates a default public/private key pair in the “/etc/salt/pki/master/ssh” directory. Although you don’t have to use this default key pair, it does make roster definition simpler because salt-ssh will assume you want to use this default file for authentication.
Also, by using the salt-ssh specific key, and not sharing a pair provided by an IaaS for generic purposes, it can be revoked or updated independent of your other services/users.
If you have password access to the remote host, and want to introduce the ability to use a public/private key pair for authentication, an easy way to do that is the use of ‘ssh-copy-id’. The ssh-copy-id tool can take this public key and append it to the remote host’s “~/.ssh/authorized_keys”.
$ sudo ssh-copy-id -i /etc/salt/pki/master/ssh/salt-ssh.rsa.pub <user>@<host>
In the “/etc/salt/roster” file, you don’t even need to specify the ‘priv’ anymore since salt-ssh will assume you want to use the default value.
If you don’t have password based access to the remote host and only have PKI based authentication, you can’t use ssh-copy-id and can instead use a raw ssh command to append the salt-ssh default public key to the remote host’s authorized_keys file.
$ sudo cat /etc/salt/pki/master/ssh/salt-ssh.rsa.pub | ssh -i trusty1.rsa vagrant@trusty1 "cat - >> ~/.ssh/authorized_keys"
With these changes in place, you should now be able to run the following command without the use of ‘passwd’ or ‘priv’ in the salt-ssh roster file.
$ sudo salt-ssh trusty1 test.version
Basic SaltStack functionality using salt-ssh
With connectivity tests now complete, we can move on to checking some basic Salt functionality using the agentless salt-ssh.
Arbitrary commands
To run an arbitrary command against the remote host using salt.states.cmd module.
$ sudo salt-ssh trusty1 cmd.run 'ls /var/log'
Grains
To see a list of all the grain values, and then just the exact OS version of the remote host:
$ sudo salt-ssh trusty1 grains.items $ sudo salt-ssh trusty1 grains.item osfinger
Pillars
To view a list of pillar values, first we need to create a pillar definition for the ‘trusty1’ remote host. Create “/srv/pillar/top.sls” with the content below.
base: 'trusty1': - trusty1
Then create “/srv/pillar/trust1.sls” with the content below.
trusty1:
lookup:
mystringdata: this is for trusty1
Now listing all the pillar values of trusty1 shows the following.
$ sudo salt-ssh trusty1 pillar.items
trusty1:
----------
trusty1:
----------
lookup:
----------
mystringdata:
this is for trusty1
States
To execute a simple state, first we need to create a state definitions for the ‘trusty1’ remote host. Create “/srv/salt/top.sls” with the content below.
base: 'trusty1': - simplestate1
Then create “/srv/salt/simplestate1.sls” with the content below.
{% set tstamp = salt["cmd.run"]("date +%Y.%M.%dT%H:%M:%S%z") %}
marker file to show simplestate1 was run:
file:
- append
- name: /tmp/simplestate1.log
- text: this proves that simplestate1 was run at {{tstamp}}
Now applying high state to the host results in the following output to the console and a file on the remote host named “/tmp/simplestate1.log”.
$ sudo salt-ssh trusty1 state.apply
trusty1:
----------
ID: marker file to show simplestate1 was run
Function: file.append
Name: /tmp/simplestate1.log
Result: True
Comment: Appended 1 lines
Started: 14:53:54.549171
Duration: 5.233 ms
Changes:
----------
diff:
---
+++
@@ -0,0 +1 @@
+this proves that simplestate1 was run at 2018.53.01T14:53:53+0000
Summary for trusty1
------------
Succeeded: 1 (changed=1)
Failed: 0
------------
Total states run: 1
Total run time: 5.233 ms
salt-ssh and targeting
Per the documentation, note that targeting via grains is not possible with salt-ssh. Only glob and regex tarets are supported.
This means that targeting based on a grain values such as OS version like “salt-ssh -G “os:Ubuntu” is not possible [1,2,3].
salt-ssh and sudo
If you issue commands that need sudo, or specify ‘sudo’ in the roster, you must be sure that the connecting user has paswordless sudo access. If this is not set in the sudoers file, the salt-ssh will be interactively prompted which breaks the automation.
sudo works only if NOPASSWD is set for user in ‘/etc/sudoers’ or ‘/etc/sudoers.d’. For example, the line below would give the ‘vagrant’ user passwordless sudo.
vagrant ALL=(ALL) NOPASSWD: ALL
REFERENCES
https://docs.saltstack.com/en/latest/topics/ssh/
https://docs.saltstack.com/en/latest/topics/ssh/roster.html#ssh-roster
https://github.com/saltstack/salt/issues/42267 (salt-ssh must be run once to create /etc/salt/pki/maser/ssh/salt-ssh.rsa.pub)
https://github.com/saltstack/salt/issues/8723 (need saltssh install, sudo apt-get install sshpass -y)
https://blog.sandra-parsick.de/2014/11/01/salt-ssh-installation-on-centos-5-5/
https://groups.google.com/forum/#!topic/salt-users/-1DxtMDC8r8 (salt-ssh hung on needing passphrase for private key)
https://www.vagrantup.com/docs/vagrantfile/ssh_settings.html (set ssh password in vagrant config)
https://docs.saltstack.com/en/latest/topics/ssh/#targeting-with-salt-ssh (targeting with grains not possible with salt-ssh, only glob and regex; grains not persistent across remote host reboots)
https://docs.saltstack.com/en/latest/topics/ssh/#calling-salt-ssh (salt-ssh requires python 2.6+, on remote systems with python3 must use salt-ssh 2017.7.0+)
https://security.stackexchange.com/questions/69407/why-is-using-an-ssh-key-more-secure-than-using-passwords (certs safer than password)
https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Host-based_Authentication (/etc/ssh/ssh_config options)
https://github.com/saltstack/salt/pull/46684 (salt-ssh python cross-major version support in 2018.3.1)
https://stackoverflow.com/questions/41986507/unable-to-set-default-python-version-to-python3-in-ubuntu/41986556 (default python on ubuntu)
NOTES
Vagrant shows default ssh private key
vagrant ssh-config
Vagrant box xenial64 and ‘ubuntu’ user
Vagrant ubuntu/xenial64 needs ‘ubuntu’ user password reset because no default
Check if ssh private/public key are pair
ssh-keygen -y -e -f private.rsa (private side, ssh client)
ssh-keygen -y -e -f ~/.ssh/authorized_keys (remote host, look at single public entry on target host)
Password authentication disabled, need private key for ssh client
/etc/ssh/ssh_config
PermitRootLogin prohibit-password
RSAAuthentication yes
PubkeyAuthentication yes
#AuthorizedKeysFile %h/.ssh/authorized_keys
IgnoreRhosts yes
RhostsRSAAuthentication no
HostbasedAuthentication no
PasswordAuthentication no
debug mode set to TRACE
sudo salt-ssh –no-host-key -v -l trace xenial1 cmd.run hostname
remove passphrase from private key
ssh-keygen -p
deploy salt-ssh private key to password based systems
–key-deploy
convert putty ppk to openssh private key
puttygen passwordless.ppk -O private-openssh -o passwordless.pem
does private rsa key have passphrase
ssh-keygen -yf xenial1.rsa
Report full versions of salt components
salt –versions-report
Look at salt-call being made by salt-ssh by SALT_ARGV
sudo salt-ssh -l trace xenial1 cmd.run ‘ls /var’ 2>&1 | grep SALT_ARGV
Default python3 on Ubuntu
update-alternatives –list python
update-alternatives –install /usr/bin/python python /usr/bin/python3 10
sudo update-alternatives –config python