SaltStack has the ability to create custom states, grains, and external pillars. There is a long list of standard external pillars ranging from those which read from local JSON files, to those that pull from EC2, MongoDB, etcd, and MySQL.
In this article, we will use Apache ZooKeeper as the storage facility for our SaltStack pillar data. ZooKeeper is used extensively for configuration management and synchronization of distributed applications, so it makes sense that it could serve as a central repository for pillar data.
ZooKeeper basics
ZooKeeper data is stored by name in a hierarchical namespace where each node in the tree is called a znode. As an example, this allows us to place generic data at “/app1”, and then more specific information about app1 at “/app1/p_1” as illustrated in the diagram below.
Solution Design
For this exercise, we will have a hierarchy where each SaltStack minion has its own znode at “/pillar/<minionid>”. For example, “/pillar/dbhost” and “/pillar/appserver1”.
At each znode, we will store a JSON string representing a flat list of key/value pairs. For example, the data found at “/pillar/dbhost” will be:
{ "therole": "database", "tmpdir": "/tmp" }
Installing ZooKeeper
For this example, we will use a very simple ZooKeeper deployment with the understanding that real production deployments provide impressive replication and fault tolerance when deployed across a set of hosts in an ‘ensemble‘.
The instructions provided are for Ubuntu.
> sudo apt-get install openjdk-7-jre-headless -y > sudo apt-get install zookeeperd -y > sudo ufw allow 2181
Now a quick smoke test to check the ZooKeeper port:
> telnet localhost 2181
Type ‘ruok’ when telnet connects, and you should get a response back ‘imok’ and the session will end.
Create ZooKeeper znodes with JSON data
Now let’s create our znode with the proper data for our hosts. First, we connect with the command line interface:
> cd /usr/share/zookeeper > bin/zkCli.sh -server localhost:2181
Create the znodes and set their JSON content:
create /pillar this_is_my_pillar_data
create /pillar/dbhost {"therole":"database","tmpdir":"/tmp"}
create /pillar/apphost1 {"therole":"app","tmpdir":"/tmp/app"}
You may notice that the strings values do not have any padding spaces. This is because zkCLI.sh is a very basic client and does not support values that have any spaces; there is no concept of enclosing the value in quotes. The value after a space would be interpreted incorrectly as the third parameter to the create method, which is the ACL.
We will live with this limitation since we are doing a proof of concept, but there are other clients such as zookeepercli and zk_shell available if you need this functionality.
And now for a quick validation of the values stored in each znode:
ls /pillar get /pillar get /pillar/dbhost get /pillar/apphost1
Python Client
Now that we have seen the data coming back from ZooKeeper using the basic client, let’s look at what it takes to get a custom Python script pulling the same data.
SaltStack external pillars are evaluated on the master (not the minions), so run this python script on the Salt master. We start by installing the Kazoo python library for ZooKeeper.
> sudo apt-get install python-pip -y > sudo pip install kazoo > mkdir -p /srv/ext/pillar
And then creating ‘/srv/ext/pillar/zookeeper.py’:
#!/usr/bin/python # # prereq: # sudo apt-get install python-pip -y # sudo pip install kazoo # import sys import json import logging from kazoo.client import KazooClient log = logging.getLogger(__name__) def createKazooClient(host, port): zk = KazooClient(hosts=host + ':' + str(port)) zk.start() return zk def destroyKazooClient(zk): zk.stop() return True def getJSON(zk, name): if zk.exists(name): return zk.get(name)[0] else: return None def makeJSONIntoData(jsonString): return json.loads(jsonString) if __name__ == '__main__': # parse command line args if len(sys.argv) >= 4: zkhost = sys.argv[1] zkport = sys.argv[2] zkpath = sys.argv[3] else: print 'Usage: host port /zkpath' print 'Example: localhost 2181 /pillar/dbhost' sys.exit(1) # get JSON from ZooKeeper zk = createKazooClient(zkhost, zkport) jsonstring = getJSON(zk, zkpath) destroyKazooClient(zk) # turn JSON into data structure and show keys datastruct = makeJSONIntoData(jsonstring) for key in datastruct.keys(): print key + ' -> ' + datastruct[key]
And now validate the data from ZooKeeper:
> chmod ugo+r+x zookeeper.py > ./zookeeper.py localhost 2181 /pillar/dbhost
Which should return back:
therole -> database tmpdir -> /tmp
SaltStack External Pillar
Finally, it’s time to hook this into SaltStack as an external pillar. The first step is to modify the ‘/etc/salt/master’ so that it picks up the custom external pillar directory and the host/port parameters of our ZooKeeper installation:
extension_modules: /srv/ext
ext_pillar:
- zookeeper:
- host: localhost
- port: 2181
Then restart the salt master:
> sudo service salt-master restart
Then we need to add a method called ‘ext_pillar’ to our python script which is the method that SaltStack will be looking for to evaluate the pillar. Here is a full listing of /srv/ext/pillar/zookeeper.py with the ext_pillar() method added:
#!/usr/bin/python
#
# prereq:
# sudo apt-get install python-pip -y
# sudo pip install kazoo
#
import sys
import json
import logging
from kazoo.client import KazooClient
log = logging.getLogger(__name__)
def createKazooClient(host, port):
zk = KazooClient(hosts=host + ':' + str(port))
zk.start()
return zk
def destroyKazooClient(zk):
zk.stop()
return True
def getJSON(zk, name):
if zk.exists(name):
return zk.get(name)[0]
else:
return None
def makeJSONIntoData(jsonString):
return json.loads(jsonString)
# proof that grain values are accessible to ext pillar
def showGrainValues():
for grain in __grains__:
log.debug(grain + "->" + str(__grains__[grain]))
# ext_pillar method is hook into SaltStack
def ext_pillar(minion_id, pillar,host='localhost',port=2181):
# pull arg values
host = host['host']
port = port['port']
log.debug('testing from SaltStack ' + minion_id + ' ' + host + ':' + str(port))
# get JSON from ZooKeeper
try:
zk = createKazooClient(host, port)
jsonstring = getJSON(zk, '/pillar/' + minion_id)
destroyKazooClient(zk)
except Error:
log.exception("ERROR while using Kazoo Client")
jsonstring = None
# turn JSON into datastructure
if jsonstring is not None:
datastruct = makeJSONIntoData(jsonstring)
else:
datastruct = {}
return datastruct
if __name__ == '__main__':
# parse command line args
if len(sys.argv) >= 4:
zkhost = sys.argv[1]
zkport = sys.argv[2]
zkpath = sys.argv[3]
else:
print 'Usage: host port /zkpath'
print 'Example: localhost 2181 /pillar/dbhost'
sys.exit(1)
# get JSON from ZooKeeper
zk = createKazooClient(zkhost, zkport)
jsonstring = getJSON(zk, zkpath)
destroyKazooClient(zk)
# turn JSON into data structure and show keys
datastruct = makeJSONIntoData(jsonstring)
for key in datastruct.keys():
print key + ' -> ' + datastruct[key]
Then have salt evaluate the pillar data for ‘dbhost’:
> salt 'dbhost' saltutil.refresh_pillar > salt 'dbhost' pillar.items
And the results should look like below, if there is already pillar data for this host, then it will be merged:
dbhost:
----------
therole:
database
tmpdir:
/tmp
With those values now set on the pillar, your states can now use this data managed centrally in ZooKeeper.
REFERENCES
https://docs.saltstack.com/en/latest/topics/development/external_pillars.html
https://zookeeper.apache.org/doc/trunk/zookeeperOver.html
https://medium.com/@Drew_Stokes/saltstack-extending-the-pillar-494d41ee156d
https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-14-04
http://kazoo.readthedocs.io/en/latest/install.html
https://docs.python.org/3/library/json.html