DevOps on OVH's cloud with Saltstack

Introduction

OVH is a french company that provides cheap virtual instances. I've selected a VPS, a Virtual Private Server, on Ubuntu 13.10. Its price is less than 2€ per month.

In this tutorial, I'll expose my way of using these instances using a powerful DevOps tool named Saltstack. With a single command, I can ping all the instances that I manage, ensure that all the servers run my latest NGINX configuration, etc, ... Everything is done from my Mac at home. It is reproducible. Therefore on risky configuration settings, I can recreate a virtual machine reproducing an existing environment (as described in my previous articles: Virtualize your servers and Fill up your servers automatically with goodies), checking my new configuration before applying it on a production server.

Note : For the story, I'm a simple customer of OVH. I don't own any share in this company. This is not an advertisement for their service. I only share it as they provide a good service and as VPS may be used to deploy almost any kind of software that you need. This tutorial could be applied to other cloud service.

Installing Saltstack master on my personal computer

Installation of Saltstack is done on OSX with Homebrew:

brew install saltstack

Now, we are going to increase the opened sockets capacity of OSX:

sudo launchctl limit maxfiles 4096 8192

When installed from Homebrew, Saltstack doesn't come with the default configuration files. Theses files are described in the Saltstack documentation pages: 21.6. Configuration file examples. Saltstack expects to see at least 2 files, master and minion into the uncreated /etc/salt directory. Let's fix that:

sudo mkdir /etc/salt

And copy the content of master and minion into this freshly created directory.

Before customizing your master file, you should know your IP address in the LAN. Here's a simple command that analyse your current configuration (note that I've setup it as an alias in my personal dotfiles as ips command).

ifconfig -a | perl -nle'/(\d+\.\d+\.\d+\.\d+)/ && print $1'

Now edit the /etc/master file to reflects the opened sockets capacity and as I dont' like sudoing each time I have to launch a command, add your username to the allowed users (here it's PEM, of course):

max_open_files: 8192
user: PEM
interface: 192.168.1.30
file_roots:
  base:
    - /Users/PEM/Projects/SolutionsM3/DevOps/states
pillar_roots:
  base:
    - /Users/PEM/Projects/SolutionsM3/DevOps/pillar

Note that I've setup my Saltstack master so that its all the formulas that I deploy are stored in my personal directory. This allows me to modify every deployed configuration and save them with Git once I've finished working on them. This is what DevOps is for: your infrastructure and administration as simple script files with formulas reproducible, idempotent, evolving, without connecting manually to every servers each time you need to adjust a simple variable.

Prepare your SSH configuration

First, I ensure that I can connect myself to my VPS without password. I don't like sudoing nor I don't like being asked for password when my security has been tightened. OVH sets up an OpenSSH server on all the instances.  OSX comes already bundled with OpenSSH, the client and the server. Though, there is one step that you need to do, if you haven't done it before : generating your personal SSH keychain. This is done with a simple bundled command:

ssh-keygen

This command generate the following files in your home directory:

/Users/PEM/.ssh
├── id_rsa
├── id_rsa.pub
└── known_hosts

We are going to authorize ourselves on our VPS with your public key:

cat ~/.ssh/id_rsa.pub | ssh root@vpsXXXXX.ovh.neta 'cat >> .ssh/authorized_keys'

Where XXXXX is the VPS's number that OVH has provided you.
Now connect yourself to your VPS using a simple ssh command and without password anymore:

ssh root@vpsXXXXX.ovh.net

Declaring your VPS as a minion

Normally, your Saltstack master should be visible from your minions, the distant servers that you need to manage. Using this configuration, Saltstack is able to handle thousands of servers in the blink of an eye by relying on secured AES tunnel relying on ZeroMQ.

But in my case, I'm on my personal Mac, at home, thus, in my LAN. Even if I modify my gateway, my ISP may change my IP whenever it wants it. I could setup a DynDNS service but each time I'll travel to another one location, I would be forced to set it up again. Thankfully, last year, Saltstack added an SSH transportation capabilities. It's a bit slower but it is as powerful as a real master / minion configuration using ZeroMQ. All you have to do is to create a list (a roster) of the servers that you want to manage in your /etc/salt/roster file with this info:

vpsXXXXX:
  host: vpsXXXXX.ovh.net

Now, whenever I want to check by a ping all my servers, I use a single command:

salt-ssh '*' test.ping

If I want to target a specific one:

salt-ssh 'vpsXXXXX' test.ping

With simple naming scheme, I'm able to achieve deployment of a specific package on a specific group of servers. Nifty.

We install remotely the appropriate Saltstack packages on every server:

salt-ssh '*' -r 'apt-get install -y salt-minion'

A simple example

Deploying the tree command on all my servers or checking that it has already been deployed from the comfort of my coach is done like this. In my DevOps project, I've set up 2 files:

├── pillar
└── states
    ├── top.sls
    └── tree.sls

The states/top.sls file declare all the available formulas that I want to apply on every servers. In this simple example it contains only a basic rule to install the tree command:

base:
  '*':
    - tree

And for the states/tree.sls file, just a simple call to the Saltstack's module pkg, which is able to handle almost every Linux packaging tools that I've been playing with:

tree:
  pkg:
    - installed

Time for the installation. As the Saltstack's state are idempotent, I can run this command every time I want. It will only execute it where it is required:

tree:
salt-ssh '*' -c /etc/salt state.highstate

With this installation, I'm capable of checking the tree of files exposed by NGINX on all my servers with a single command:

salt-ssh '*' -r 'tree /var/www'

Color your logs

Log files are a pain to read. When colored a bit, you start having a better understanding of what it going on under the hood.

This is achieved via a simple tail replacement: colortail.

On OSX, hit the following command:

brew install colortail

On Ubuntu, use your favorite package system command:

sudo apt-get install colortail

Now, in your ~/.bashrc or your ~/.zshrc, add a simple aliases:

alias tail='colortail -k ~/.colortail/conf.default '

Create a ~/.colortail dir where you can put your colored theme files:

mkdir ~/. colortail

Here is my ~/.colortail/conf.default file:

COLOR brightred
{
# matches the word ERROR
^.*(ERROR|error).*$
}
COLOR yellow
{
# matches the word WARNING
^.*(WARNING|warning).*$
}
COLOR green
{
# matches the word INFO
^.*(INFO|info).*$
}
COLOR grey
{
# matches the word DEBUG
^.*(DEBUG|debug).*$
}
COLOR brightblue
{
# matches the time
^.*([0-9]{2}:[0-9]{2}:[0-9]{2}).*$
}

Tips: Debug Saltstack States

Just hop under your current Vagrant VM and launch the SLS file locally:

vagrant ssh
sudo salt-call state.highstate --log-level=debug

Source files of the 2 tutorials on Vagrant and SaltStack available on Github

I've uploaded my source file on Github: VagrantSaltstackNginxSample for the Vagrant and Saltstack tutorials:

Virtualize your servers

Fill up your servers automatically with goodies

Fill up your servers automatically with goodies

Introduction

In my former article "Virtualize your servers", we created a virtualized server in few command lines. But... It's empty as the void in space (or so we think...). Still, you can log in easily with:

vagrant ssh

It is now time to fill it with some goodies: NGINX with all the bells and whistles.

Saltstack to the rescue

There are many provisionner available for Vagrant: Chef, Puppet, Docker, Saltstack, ... even simple bash scripts. A provisionner acts as framework for creating scripts that will fill your server wether they are physical or virtual. Note that as the servers may be virtual or physical, we will simply refers them as nodes. Nodes are the processing unit of your private cloud, wether it be a single to thousands machine.

Depending on the provisionner you choose, you may have more power for addressing use case scenarios such as:

• Managing a common configuration for each node with specialization for some of them.
• Configuring IP addresses and routes for each node.
• Managing states for each node: a state is an expected running configuration.
• Upgrading each node and letting the other knows about it while limiting the impacts on service unavailability.

I've chosen Saltstack as it covers these capabilities and the configuration files, the minions, that you write are simplistic and well organized.

Make Vagrant knows about Saltstack

Vagrant speaks Saltstack out-of-the-box. You only put in your little Vagrantfile where are stored your Saltstack files. Here, I put them in a directory named salt.

# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # Name of the node
  config.vm.box = "UbuntuServer"

  # Import a preinstalled Ubuntu Server 12.04 LTS
  config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box"

  # Add port forwarding to access service deployed in the node
  config.vm.network "forwarded_port", host: 8080, guest: 80

  # Synchronized folders
  config.vm.synced_folder "salt/srv/salt/", "/srv/salt/"
  
  # Use Saltstack as provisioner
  config.vm.provision :salt do |salt|
    # Set the directory where is stored your minion
    salt.minion_config = "salt/minion"
    # Maintain states
    salt.run_highstate = true
  end
end

Note the option salt.run_highstate = true. It tells  Vagrant to ensure the node state whenever it is executing it.

Here comes the minions

There are 2 ways of using Saltstack: locally or as a server which acts as a repository of minions. In our simple example, we use the most basic way, the local incarnation: masterless. Therefore, the salt/minion file is kept as its bare minimum:

# We run in masterless mode
master: localhost
# We use the local file directory
file_client: local

We use the default Saltstack directory structure to store our minion that will install NGINX:

.
├── Vagrantfile
└── salt
    ├── minion
    └── srv
        └── salt
            ├── top.sls
            └── webserver.sls

The top.sls file is your entry file to the installation. Here, there is only one formula. Thus, it is pretty straightforward:

base:
  '*':
    - webserver

The webserver.sls is called by the top.sls. It contains the formula to install and run NGINX as a service:

nginx:
  pkg:
    - installed
  service:
    - running
    - require:
      - pkg: nginx

Get ready!

Now, it's time to fire up everything. If you have just followed my previous tutorial "Virtualize your servers", you should already have downloaded the node's OS. We will destroy it. Yes. Destroy it. This will not re-download the node's OS. It will simply destroy its current customized image. Then, we will rerun Vagrant with our new and fresh configuration. This will do the following:

• Re-configure the node.
• Bootstrapping Saltstack into it.
• Launch Saltstack so that:
• It install NGINX using the node's OS package manager, apt-get in this case.
• Launch NGINX as a service.

vagrant destroy
vagrant up

Now, fire up your browser of choice and use the following URL: http://localhost:8080. This should display the NGINX default page just as if you had install it on a regular Ubuntu server.

Next steps

Vagrant and Saltstack are an incredibly powerful combo. In this example, I've made you destroy and reinstalled the node. It was only to ensure a clean state. While creating your server, you don't need to recreate everything and restart your node. Using the states provided by Saltstack, you just modify your SLS files and reprovision the states as you develop them. Iteration are done in a matter of seconds with the command:

vagrant provision

You can install other services as easily. There are some already prepared formulas that you can directly import or take inspiration from in the Saltstack formulas repository.