An issue that keeps coming up on the mailing lists as well as Stackoverflow[1,2] is how to merge multiple pillar files for use with a single state. The problem is that pillars using the same key overwrite each other, and there is no easy way to express the desire to merge instead.
There are various workarounds, but all of these expect the human operator to know about these disparate sources and manually mend them together with a unifying sls file (using includes or anchors/references).
The state and pillar files in this article can be downloaded from my github page.
Pillar Problem
To make the problem statement concrete, let’s pretend we have minions with a defined grain of ‘role’ that determines whether the MatLab Engineering/Math suite or Maya 3dStudio graphics are installed. Both these applications need to have their logs rotated on a daily basis so that the size does not grow too large.
# /srv/pillar/top.sls 'G@role:matlab': - match: compound - logrotate-matlab 'G@role:maya3ds': - match:compound - logrotate-maya3ds
# /srv/pillar/logrotate-matlab.sls
logrotate:
lookup:
rotatejobs:
matlab:
path: /var/log/matlab/matlab.log
# /srv/pillar/logrotate-maya3ds.sls
logrotate:
lookup:
rotatejobs:
maya3ds:
path: /var/log/3ds/3ds.log
As long as a minion was never a participant in both the ‘matlab’ and ‘maya3ds’ role, the above pillar definitions would work. But if a minion ever had both roles, now one would overwrite the other and log rotation would not function properly for one of the applications.
You may react to this by saying you would simply create another sls pillar that unified both these pillars into one file like below:
# /srv/pillar/logrotate-matlab-and-maya3ds.sls
logrotate:
lookup:
rotatejobs:
matlab:
path: /var/log/matlab/matlab.log
maya3ds:
path: /var/log/3ds/3ds.log
That would resolve the immediate problem, but now let’s pretend that instead of only two applications, you had tens of applications deployed, and there were many different role combinations. Now you would have to manually create an sls for minions that had applications (A,B,C) and another sls for minions that had applications (X,Y,Z), etc…
This manual stitching together of pillars is manual and error-prone. And every time a minion changes roles, all these pillars would have to revisited.
Pillar Solution
The better solution is to recognize that the pillar data should be adjacent to each application it supports, and then merged for use in the single generic state.
For example, imagine if the pillars were instead defined like this:
# /srv/pillar/top.sls base: 'G@role:matlab': - match: compound - logrotate-matlab 'G@role:maya3ds': - match: compound - logrotate-maya3ds
# /srv/pillar/logrotate-matlab.sls
logrotate-matlab:
lookup:
rotatejobs:
matlab:
path: /var/log/matlab/matlab.log
# /srv/pillar/logrotate-maya3ds.sls
logrotate-maya3ds:
lookup:
rotatejobs:
matlab:
path: /var/log/maya3ds/maya3ds.log
Notice the very subtle difference where the keys are no longer colliding because they are now both unique and no longer using the same ‘logrotate’ key and are instead are prefixed with ‘logrotate-‘.
But, now there is a problem because if you have a single minion that has both roles and you check the pillar, then you can see you indeed have two pillar keys (they did not overwrite each other), but neither contains the ‘logrotate’ key that the state is expecting.
$ sudo salt 'trusty1' pillar.items
trusty1:
----------
logrotate-matlab:
----------
lookup:
----------
rotatejobs:
----------
matlab:
----------
path:
/var/log/matlab/matlab.log
logrotate-maya3ds:
----------
lookup:
----------
rotatejobs:
----------
maya3ds:
----------
path:
/var/log/3ds/3ds.log
So, the question becomes how can you mend these together, and then have them presented to the state as a single combined entity with the correct name?
Simple State
So far we have only looked at the pillar files, but let’s take a quick look at the state files for the ‘logrotate’ formula.
# /srv/salt/top.sls
base:
'G@role:matlab':
- match: compound
- logrotate
'G@role:maya3ds':
- match:compound
- logrotate
# /srv/salt/logrotate/init.sls
{% from "logrotate/map.jinja" import logrotate with context %}
{% for key,value in logrotate['rotatejobs'].items() %}
{% set thisjob = key %}
{% set thislog = value.path %}
logrotate-task-{{thisjob}}:
cmd.run:
- name: echo asked to logrotate {{thislog}}
{% endfor %}
# /srv/salt/logrotate/map.jinja
{% import_yaml 'logrotate/defaults.yaml' as default_settings %}
# standard OS family customization
{% set os_family_map = salt['grains.filter_by']({
'RedHat': {
},
'Arch': {
},
'Debian': {
},
'Suse': {
},
}, merge=salt['pillar.get']('logrotate:lookup'))
%}
{% do default_settings.logrotate.update(os_family_map) %}
# merge pillar lookup on top of default settings
{% set logrotate = salt['pillar.get'](
'logrotate:lookup',
default=default_settings.logrotate,
merge=True
)
%}
# /srv/salt/logrotate/defaults.yaml
logrotate:
rotatejobs:
# placeholder:
# path: /var/log/placeholder/dummy.log
The top.sls state file says to apply the ‘logrotate’ formula to any minion in the role of ‘matlab’ or ‘maya3ds’, and the logrotate/map.jinja and logrotate/default.yaml are boiler plate best practice for allowing pillar overrides.
The only file of real interest here is ‘logrotate/init.sls’ which goes through all the items in the logrotate.rotatejobs hash and prints the log path for debugging purposes. Clearly, there is no actual log rotation logic here, this is just placeholder logic for our example.
{% for key,value in logrotate['rotatejobs'].items() %}
The only problem is, this formula is getting its data from the pillar named ‘logrotate:lookup’, and as we saw from the pillar section, we are being provided ‘logrotate-matlab:lookup’ and ‘logrotate-maya3ds:lookup’. So if we want this state to operate on our data, we are going to need an enhancement.
State Solution
As we talked about above, now that the pillar data is present (but in two unique keys), the challenge is to get the state logic to combine these and use them as a single pillar in its work.
You could hack this into the main state sls file, but even cleaner is to enhance map.jinja which by best practice is already loading in pillar defaults and overrides.
# /srv/salt/logrotate/map.jinja
{% import_yaml 'logrotate/defaults.yaml' as default_settings %}
# standard OS family customization
{% set os_family_map = salt['grains.filter_by']({
'RedHat': {
},
'Arch': {
},
'Debian': {
},
'Suse': {
},
}, merge=salt['pillar.get']('logrotate:lookup'))
%}
{% do default_settings.logrotate.update(os_family_map) %}
# merge pillar lookup on top of default settings
{% set logrotate = salt['pillar.get'](
'logrotate:lookup',
default=default_settings.logrotate,
merge=True
)
%}
# inner loop variables don't have scope, so have to use hash trick
# https://fabianlee.org/2016/10/18/saltstack-setting-a-jinja2-variable-from-an-inner-block-scope/
#
{% set mergedresult = { 'logrotate': {} } %}
{% for key in pillar.keys() %}
{% if "logrotate-" in key %}
{% set val = pillar.get(key) %}
{% set keylookup = key~":lookup" %}
{% set custjobs = salt['pillar.get'](keylookup,{}) %}
{% set merged = salt['slsutil.merge'](
mergedresult['logrotate'],
custjobs,
merge_lists=True
)
%}
{% do mergedresult.update({'logrotate': merged}) %}
{% endif %}
{% endfor %}
# do final merge, adding to main variable in outer scope
{% set logrotate = salt['slsutil.merge'](
logrotate,
mergedresult.logrotate,
merge_lists=True
)
%}
Take note that top half was presented earlier and is basic boilerplate that is part of almost every formula. Our custom logic takes up the latter half, so let’s go through its key sections in more detail.
Remember, the pillar that the state wants to work on is called “logrotate”, but the pillars we provided it to use are “logrotate-matlab” and “logrotate-maya3ds”. This means we need to iterate through all the pillar keys, and if it starts with “logrotate-” then we should put it into the “custjobs” working variable.
{% for key in pillar.keys() %}
{% if "logrotate-" in key %}
{% set val = pillar.get(key) %}
{% set keylookup = key~":lookup" %}
{% set custjobs = salt['pillar.get'](keylookup,{}) %}
Now we will use ‘slsutil.merge’ to merge together the dictionary in “customjobs” with the final result list.
{% set merged = salt['slsutil.merge'](
mergedresult['logrotate'],
custjobs,
merge_lists=True
)
%}
{% do mergedresult.update({'logrotate': merged}) %}
The reason we don’t directly update ‘logrotate’ is because of the strange scoping rules in jinja2 that don’t allow variables to change inside an inner scope. If we directly changed the outer scoped ‘logrotate’ variable, as soon as we left the inner block the changes would be lost!
Finally, when we do exit the inner block scope, we need to update the ‘logrotate’ hash with the results built in the inner block.
# do final merge, adding to main variable in outer scope
{% set logrotate = salt['slsutil.merge'](
logrotate,
mergedresult.logrotate,
merge_lists=True
)
%}
Now when ‘/srv/salt/logrotate/init.sls’ loads in the ‘logrotate’ variable, it will come with any pillar variants already loaded into the ‘logrotate.rotatejobs’ dictionary key.
{% from "logrotate/map.jinja" import logrotate with context %}
Running state against our single minion that has both ‘matlab’ and ‘maya3ds’ roles will result in output like below, which shows that both pillar definitions were identified and merged for use by this single state.
$ salt 'trusty1' state.apply
trusty1:
----------
ID: logrotate-task-maya3ds
Function: cmd.run
Name: echo asked to logrotate /var/log/3ds/3ds.log
Result: True
Comment: Command "echo asked to logrotate /var/log/3ds/3ds.log" run
Started: 17:49:20.328995
Duration: 9.622 ms
Changes:
----------
pid:
21930
retcode:
0
stderr:
stdout:
asked to logrotate /var/log/3ds/3ds.log
----------
ID: logrotate-task-matlab
Function: cmd.run
Name: echo asked to logrotate /var/log/matlab/matlab.log
Result: True
Comment: Command "echo asked to logrotate /var/log/matlab/matlab.log" run
Started: 17:49:20.338738
Duration: 4.591 ms
Changes:
----------
pid:
21932
retcode:
0
stderr:
stdout:
asked to logrotate /var/log/matlab/matlab.log
Summary for trusty1
------------
Succeeded: 2 (changed=2)
Failed: 0
------------
Total states run: 2
Total run time: 14.213 ms
The state and pillar files in this article can be downloaded from my github page.
If you have downloaded the state and pillar files for testing in your own environment, you can add one one of your minions to both the ‘matlab’ and ‘maya3ds’ role with the following command:
$ sudo salt 'myminion' grains.set role ['matlab','maya3ds'] $ sudo salt 'myminion' grains.get role
If you want to install a SaltStack master and test minion on Ubuntu, you can read my article here.
REFERENCES
https://serverfault.com/questions/587756/saltstack-pillar-includes-under-same-key
http://stackoverflow.com/questions/24631231/how-to-join-two-salt-pillar-files-and-merge-data
https://groups.google.com/forum/#!topic/salt-users/e4O_r8KQR0g
https://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.yamlex.html
https://docs.saltstack.com/en/latest/topics/development/external_pillars.html
https://github.com/saltstack/salt/issues/3991