Configuration Management in ORBIT and WINLAB

We are currently using saltstack to push configuration changes to servers. This is a server/client model, with the salt-minion running on clients, and salt-master on the central servr.

Clients find the master via looking for 'salt' in DNS, by default, and can be otherwise configured in a multitude of ways. Currently, 'salt' resolves to salt.winlab.rutgers.edu or salt.orbit-lab.org, and both are CNAMES for interfaces of remote-l, which exists in both networks.

After finding the master, the minion performs a key exchange. Further communication is done via encrypted message broker traffic over tcp, using the ZMQ protocol.

File structure and layout

We are using salt 'states' and 'pillars'. States describe a desired configuration, e.g., this package is installed, these files are present, this user has these properties, etc. Pillars contain more specific configurations in key-value pairs, or YAML more generally.

States reference pillars, so instead of configuring PAM to refer to ldap.orbit, the specific string would be pulled in as needed. TODO Syntax reference.

Importantly, information in the 'state' files are available to ALL clients, as it represents potential commands to run. Conversely, information in the 'pillar' files are ONLY available to the clients targeted. Other clients will see either nothing, or a default value.

States and pillars are each targeted to clients via a 'top' file. This contains a list of matching rules, and state IDs. For example:

base:
  external*.orbit-lab.org:
    - ssh.present

This top file would target all clients with a name matching external*.orbit-lab.org, and have them execute the state ssh.present, from the tree for enviroment 'base'

Our current structure is as follows:

/srv
    /pillar
        /base
            top.sls
        /orbit
        /winlab
    /salt
        /base
            top.sls
        /orbit
        /winlab

This is a slightly more complex set up with multiple environments. The roots for each environment are set in /etc/salt/master, under file_roots and pillar_roots.

When referring to a state or pillar, salt looks for the matching id in the corresponding environment tree, here base, orbit, or winlab. However, TOP FILES ARE GLOBAL. There is nothing stopping you from targeting an orbit machine with a winlab state, or both via sloppy use of wildcards.

This can be used to benefit by putting common configuration under the base tree, and only keeping environment specific states under their respective trees. Code reuse means we fix problems fewer times. Environment specific configuration will in most cases be in pillars, not states anyway.

It is possible to have multiple top files, but salt merges them upon execution, so it is better to manually configure it.

Usage

The simplest use case is remote execution, with the format: salt 'client pattern match' cmd.run 'command' This will match all clients in the pattern match, and execute that command, returning the result.

To apply a specifc state: salt 'client pattern match' state.apply environment:state_id

To apply the configuration from the top file: salt 'client pattern match' state.apply

To do a dry run, use the form: salt 'client pattern match' state.apply test=True I have configured all clients to default to this, to overcome this set test=False.

Resources

We are running version 2018.3. A helpful list of best practices are ​here