SaltStack is easily one of my favorite DevOps tools. I always like to compare it to a Leatherman, it's a multi purpose tool that you can always reach for when you need to accomplish something.

One extremely useful yet somewhat underutilized features are Salt Engines. Engines are really just long running Python processes that run on either the master or the minion that perform whatever tasks you need to do. There are a few built in engines, but, I don't think they are terribly useful in most situations. The real power of engines is providing you with a baseline foundation for deploying and running tasks, so that you only have to focus on the components specific to your business.

In most operations based tasks like this, writing code is rarely the most time consuming part. Instead you spend more time:

• Building out CI and other pipelines to distribute your tooling across a multitude of hosts
• Writing solutions to monitor and maintain that your service is up and running
• Developing a way to communicate with the daemons you deploy

Salt Engines solve all of these issues by providing you with a robust set of tooling.

• Salt is able to easy distribute custom engines to all minions without any external tooling
• Salt engines are just simple python scripts and salt provides common python libraries on Linux, Windows and more
• Salt provides a low latency message bus to provide two-way communications with all minions

To provide a few examples of how we have leveraged engines in our real world environment:

1. We have a special engine that lives on the master that is able to ingest Salt Beacons, like memory usage and disk space, and emit the data to Datadog. This saves us from having to install Datadog agents on every single piece of infrastructure we manage which can be a headache.
2. We install a system at client sites that consists of a server with a couple VMs and sometimes up to 1000s of small camera based devices. We need to be able to control and manage these devices with networks we have no control over so we have a Salt Engine we deploy that is able to access these devices and report a multitude of information back to us.
3. We we have an engine that runs on the master that pushes all the events into SNS, this let's us process events with standard cloud tooling as well as produce audit reports on actions that happened against the minions.

The great thing about these engines is how easy it is to build and deploy it because Salt gives us a pre-configured environment to write and publish them to every minion we have in our stack. By utilizing the existing tools that Salt provides we were able to build and deploy all of these tools with very little effort, which results in a very high level of value for a very small amount of time.

A very simple starting point for a Salt engine looks like this, which is (mostly) straight from the Salt git repository:

import logging

import salt.utils.event
import salt.utils.json

log = logging.getLogger(__name__)

# This function is just selecting the correct context depending on if this engine is running on the master or the minion
def event_bus_context(opts):
    if opts["__role"] == "master":
        event_bus = salt.utils.event.get_master_event(
            opts, opts["sock_dir"], listen=True
        )
    else:
        event_bus = salt.utils.event.get_event(
            "minion",
            transport=opts["transport"],
            opts=opts,
            sock_dir=opts["sock_dir"],
            listen=True,
        )
        log.debug("test engine started")
    return event_bus

# This is where the real magic starts happening
def start():
    """
    Listen to events and write them to a log file
    """
    with event_bus_context(__opts__) as event_bus:
        while True:
            event = event_bus.get_event()
            jevent = salt.utils.json.dumps(event)
            if event:
                log.debug(jevent)

Deploying this custom engine is as easy as including it in your Salt fileserver and running a single command:

salt '*' saltutil.sync_engines

That's it, your engine is now available to every single piece of infrastructure in your fleet.