This guide is for two audiences:

You want to learn Salt.
You plan on having a few DigitalOcean droplets and want an easy way to manage them.

We’re going to combine the simplicity of Salt with DigitalOcean’s snapshot and image feature. For great justice.

Why bother?

Sooner or later you’re going to have a bunch of DigitalOcean droplets doing excitingly different things.

Wouldn’t it be cool if you could run sudo apt-get dist-upgrade -y on all of them in one shot? Yes it would. Let’s learn config management.

Why not salt-cloud?

You can do this with another tool called salt-cloud. It uses the DigitalOcean API to provision droplets. It runs a complex script called salt-bootstrap on all the new machines.

We’re not going to do that.

Let’s be clever with our own template, because:

It’s pretty simple. Salt-cloud is overkill for a small number of servers.
You will have hands-on control over what’s happening.
You will learn about Salt and Images! Learning is fun.

Here’s the plan

Create a master server. You could use an existing machine to save a few beer tokens.
Create a minion - thank you Salt developers for using this word - and test it.
Tweak and snapshot the minion as a template for future droplets.
Execute a complex victory dance.

The Salt chain of command

You need a single Salt master - a package called salt-master - in order to issue imperious commands to the unwashed, quivering minions.

Your minions have a client installed - salt-minion - and are told where the master is. The minions ask for permission to join up, the master says yes, and off we go.

You can install a minion client on the master as well, if you want to be perverse. Implications of doing this:

The master server can be managed along with everyone else. This is handy.
You run a small risk of screwing the master when you accidentally send a sudo rm -rf /* to all minions. Hint: Don’t do this.

1. Create the master

Create a new Ubuntu droplet called configmaster and log in as root. At create time, specify an SSH key because we are adults.

Let’s set up a new user and give them sudo privileges:

# adduser salt
# # (set a password and default all the other questions)
# visudo

Make the sudoers file look like this and save it out:

# User privilege specification
root    ALL=(ALL:ALL) ALL
salt    ALL=(ALL:ALL) ALL

That’s our dirty root-werk done. Log out and SSH back in as salt using your new password.

Install the good stuff

As user salt:

$ sudo echo "sudo is working, I feel like an internet hero"
$ sudo add-apt-repository -y ppa:saltstack/salt
$ sudo apt-get update
$ sudo apt-get install -y salt-master
$ service --status-all 2>&1 | grep salt
 [ + ]  salt-master
$ sudo salt-key -L
Accepted Keys:
Unaccepted Keys:
Rejected Keys:

Your master is now done. Told you it was simple.

What have I created?

Minions ask for permission to join the network by sending over a key. The salt-key -L command means ‘(L)ist all the minions currently connected’.

Minions who are connected but remain patiently waiting for a decision are listed as Unaccepted.
Notice that a master can choose to Reject a minion that has the wrong sort of haircut.

For debug purposes, the salt-master log is at e.g. cat /var/log/salt/master. With default settings, the log will be empty unless something is broken.

By default the Salt master listens on ports 4505 and 4506 on all interfaces (0.0.0.0).

2. Set up a minion

We’re going to do the absolute minimum to bootstrap this new server. Once it’s hooked up, all work can be done remotely from the master.

Create a new droplet called salt01. Log in as root and:

# add-apt-repository -y ppa:saltstack/salt
# apt-get update
# apt-get install -y salt-minion
# service --status-all 2>&1 | grep salt

There’s one item of config business. We need to set the location of the master (an IP address for the moment). Edit the file as below and set it:

# vi /etc/salt/minion
# # set the appropriate line to read 'master: <your configmaster IP>'
# service salt-minion restart

You can understand what the minion is doing by watching the log:

# cat /var/log/salt/minion

Have a look now. You can see that at first, the minion couldn’t find a master at the default hostname salt. Once this was fixed and restarted, we see

[salt.crypt       ][ERROR   ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

This means the minion is connected but the master hasn’t yet Accepted the key.

I for one welcome our new minion

Back on the master (as user salt), you can see a minion wants to register:

$ sudo salt-key -L
Accepted Keys:
Unaccepted Keys:
salt01
Rejected Keys:

So let’s be a nice doorman and let them in:

$ sudo salt-key -y -a salt01
The following keys are going to be accepted:
Unaccepted Keys:
salt01
Key for minion salt01 accepted.

$ sudo salt-key -L
Accepted Keys:
salt01
Unaccepted Keys:
Rejected Keys:

Note that salt-key -A will accept all keys currently waiting for a decision.

Test your powers

Minions can execute functions from execution modules that are defined by Salt. They have names like test.ping.

The full module list is pretty exhaustive. Naturally you can add your own stuff.

On the master:

$ # You can address a specific minion:
$ sudo salt salt01 test.ping
salt01:
    True

# or address all minions, to e.g. dump their IPs:
$ sudo salt '*' network.ip_addrs
salt01:
    - 123.456.78.123

You can get away with not escaping the * asterisk but it’s probably not a great habit.

The cmd.run function allows arbitrary shell commands:

$ sudo salt '*' cmd.run 'uname -a'
salt01:
    Linux salt01 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

So we’ve created a minion. Do we want to do this crap every time we add a new one? Nope. Let’s create a DigitalOcean image.

Now we understand what’s going on, we’re going to make a template out of salt01. This requires a spot of minor surgery to remove its identity.

I am not a number, I am a free man

When the salt-minion service starts up, it generates an identity based on the hostname. This minion_id, along with a new key are cached to disk.

We need to remove that for our template - credit to this post for the method. On your salt01 minion:

# service salt-minion stop
# rm /etc/salt/pki/minion/minion_master.pub
# rm /etc/salt/pki/minion/minion.*
# cat /dev/null >/etc/salt/minion_id

If we didn’t do this, new machines created from template would all have a cached identity of salt01. We would see this on startup:

# salt-minion -l debug
[DEBUG   ] Reading configuration from /etc/salt/minion
[INFO    ] Using cached minion ID from /etc/salt/minion_id: salt01
...

Oh shit! Identity theft. An army of clones worked for The Empire, but not for us.

Don’t confuse the master

Since we just blew away salt01’s identity, we need to tell the master not to expect that key any more:

$ sudo salt-key -D -y

The -D means ‘(D)elete all minion keys’. The salty D.

Make an image!

DigitalOcean lets you take a snapshot, which creates an image. Images can be used to create new droplets just like the old one, except for a brand new hostname.

First we need to power off salt01:

# poweroff

And in your DigitalOcean web console, snapshot it. Call the image something like ‘Salt minion template’.

When the snapshot finishes, create a new droplet salt02 from the image.

When salt02 starts up everything will happen by magic:

It already has the salt-minion package installed and config ready to go! Great.
It will generate its new identity.
It will connect to the master.

Get up salt01, you’re fine

After the snapshot, salt01 will automatically be powered on, and will rejoin exactly as above using its own hostname.

4. With two minions I am unstoppable

We’re all done. We have two fresh minions asking to join the master, so let them in:

$ sudo salt-key -L
Accepted Keys:
Unaccepted Keys:
salt01
salt02
Rejected Keys:

$ sudo salt-key -A -y
The following keys are going to be accepted:
Unaccepted Keys:
salt01
salt02
Key for minion salt02 accepted.
Key for minion salt02 accepted.

$ sudo salt '*' cmd.run 'uname -a; uptime'
salt02:
    Linux salt02 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
     18:37:07 up 5 min,  1 user,  load average: 0.01, 0.03, 0.02
salt01:
    Linux salt01 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
     18:37:07 up 8 min,  1 user,  load average: 0.00, 0.01, 0.01

Whenever you need a new droplet, use your custom image and watch it automatically come under control of your config management network.

You’re free to update the image and make a new snapshot whenever you feel like it - just remember to clean the identity beforehand.

But you have not mentioned splutter

Networking

DigitalOcean takes care of the hostname and MAC address on new servers. You can confirm this with e.g.

$ sudo salt '*' network.hw_addr eth0
salt02:
    04:06:28:41:19:01
salt01:
    04:04:29:22:fe:11

Security

Don’t forget security! You’ll want to take basic security steps like hardening your SSH settings, maybe install ufw and all that good stuff. The good news: you can set it up once in Salt and use the master to push to all new servers…

Happy minioning!