Automating SaltStack Tasks with Webhooks

SaltStack is a fantastic tool for provisioning and configuring machines across all sorts of infrastructure setups. It’s been a staple for me for a long time when it comes to configuration management. One of the struggles that I’ve seen with people is how to utilize it though in a more event driven system. This is one place Salt really shines but it needs a little help from an outside tool

Before I begin I just want to take a minute to note that SaltStack itself has built in support for ingesting webhooks, but I will say it’s one place that is somewhat lacking in my opinion. The only real options you have are enabling the webhooks engine which is unauthenticated or using the salt net-api to ingest them. Both of these then result in the event going through the salt event reactor which can be, well, quirky.

It can also make ingesting some of the weird data schemes that things use to fit them into the shape that you need.

Thankfully there is a solution – Webhook. This is a very lightweight Go webhook engine for ingesting, parsing, and executing commands. It comes included in Debian, Ubuntu and Arch for easy install simply

apt-get install -y webhook

Once installed you will now need to configure a webhook. For this example I’ll configure a very simple webhook which will update the local fileserver.

To start we will need to configure the webhook, in Debian based repositories the service is setup to load the config file located in /etc/webhook.conf. It will accept json or yaml configuration and I usually use yaml as I find it easier to read.

- id: update-fileserver
  execute-command: "/etc/webhook/scripts/fileserver-update.py"
  command-working-directory: "/etc/webhook/scripts"
  trigger-rule:
    and:
    - match:
        type: value
        value: some-secret-value-here
        parameter:
          source: url
          name: token
    - match:
        type: value
        value: refs/heads/master
        parameter:
          source: payload
          name: ref

To break down the important parts here:

  • id: This is the name of the webhook and will be the URL required to activate it
  • execute-command: This is telling the webhook what command to execute
  • trigger-rule: This is telling the webhook what it needs to match to activate this webhook. This is where to set filters and most important a secret.

Once this is configured you can startup the webhook service with

systemctl enable --now webhook

Now we need to configure where the magic happens. Most examples I see tend to use the salt reactor or bash scripts to execute salt commands, but why limit yourself? Salt provides a very convenient to use so you can combine the power of Python and Salt to do your dark bidding

First thing is first we’ll create the location to store all our excellent scripts

mkdir -p /etc/webhook/scripts/

And now we’ll create our first script. This one is fairly simple, it just uses the RunnerClient to update the Salt fileserver

#!/usr/bin/python3
import salt.runner

opts = salt.config.master_config("/etc/salt/master")
runner = salt.runner.RunnerClient(opts)

runner.cmd(
  fun="fileserver.update",
)

print("Filerserver updated")

Update the script file to mark it executable

chmod +x /etc/webhook/scripts/fileserver-update.py

Then all you need to do is configure your Git host to submit a webhook on push events to your webhook URL:

http://you-host.com:9000/hooks/update-filserver?token=some-secret-value-here

Now simply make a push to your git host’s Salt repository and see it in action

In my next post I’ll cover some basics of using Datadog to do event driven infrastructure tasks.