5 avr. 2014

Fill up your servers automatically with goodies

Introduction

In my former article "Virtualize your servers", we created a virtualized server in few command lines. But... It's empty as the void in space (or so we think...). Still, you can log in easily with:
vagrant ssh
It is now time to fill it with some goodies: NGINX with all the bells and whistles.

Saltstack to the rescue

There are many provisionner available for Vagrant: Chef, Puppet, Docker, Saltstack, ... even simple bash scripts. A provisionner acts as framework for creating scripts that will fill your server wether they are physical or virtual. Note that as the servers may be virtual or physical, we will simply refers them as nodes. Nodes are the processing unit of your private cloud, wether it be a single to thousands machine.

Depending on the provisionner you choose, you may have more power for addressing use case scenarios such as:
  • Managing a common configuration for each node with specialization for some of them.
  • Configuring IP addresses and routes for each node.
  • Managing states for each node: a state is an expected running configuration.
  • Upgrading each node and letting the other knows about it while limiting the impacts on service unavailability.
I've chosen Saltstack as it covers these capabilities and the configuration files, the minions, that you write are simplistic and well organized.

Make Vagrant knows about Saltstack

Vagrant speaks Saltstack out-of-the-box. You only put in your little Vagrantfile where are stored your Saltstack files. Here, I put them in a directory named salt.
# Vagrantfile API/syntax version. Don't touch unless you know what you're doing!
VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  # Name of the node
  config.vm.box = "UbuntuServer"

  # Import a preinstalled Ubuntu Server 12.04 LTS
  config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box"

  # Add port forwarding to access service deployed in the node
  config.vm.network "forwarded_port", host: 8080, guest: 80

  # Synchronized folders
  config.vm.synced_folder "salt/srv/salt/", "/srv/salt/"
  
  # Use Saltstack as provisioner
  config.vm.provision :salt do |salt|
    # Set the directory where is stored your minion
    salt.minion_config = "salt/minion"
    # Maintain states
    salt.run_highstate = true
  end
end
Note the option salt.run_highstate = true. It tells  Vagrant to ensure the node state whenever it is executing it.

Here comes the minions

There are 2 ways of using Saltstack: locally or as a server which acts as a repository of minions. In our simple example, we use the most basic way, the local incarnation: masterless. Therefore, the salt/minion file is kept as its bare minimum:
# We run in masterless mode
master: localhost
# We use the local file directory
file_client: local
We use the default Saltstack directory structure to store our minion that will install NGINX:
.
├── Vagrantfile
└── salt
    ├── minion
    └── srv
        └── salt
            ├── top.sls
            └── webserver.sls
The top.sls file is your entry file to the installation. Here, there is only one formula. Thus, it is pretty straightforward:
base:
  '*':
    - webserver
The webserver.sls is called by the top.sls. It contains the formula to install and run NGINX as a service:
nginx:
  pkg:
    - installed
  service:
    - running
    - require:
      - pkg: nginx

Get ready!

Now, it's time to fire up everything. If you have just followed my previous tutorial "Virtualize your servers", you should already have downloaded the node's OS. We will destroy it. Yes. Destroy it. This will not re-download the node's OS. It will simply destroy its current customized image. Then, we will rerun Vagrant with our new and fresh configuration. This will do the following:
  • Re-configure the node.
  • Bootstrapping Saltstack into it.
  • Launch Saltstack so that:
    • It install NGINX using the node's OS package manager, apt-get in this case.
    • Launch NGINX as a service.
vagrant destroy
vagrant up
Now, fire up your browser of choice and use the following URL: http://localhost:8080. This should display the NGINX default page just as if you had install it on a regular Ubuntu server.

Next steps

Vagrant and Saltstack are an incredibly powerful combo. In this example, I've made you destroy and reinstalled the node. It was only to ensure a clean state. While creating your server, you don't need to recreate everything and restart your node. Using the states provided by Saltstack, you just modify your SLS files and reprovision the states as you develop them. Iteration are done in a matter of seconds with the command:
vagrant provision
You can install other services as easily. There are some already prepared formulas that you can directly import or take inspiration from in the Saltstack formulas repository.

3 avr. 2014

Virtualize your servers

Introduction

Virtualizing a server has many advantages over installing real servers. First, you can have many installments for testing or replicating existing ones. Second, it helps you develop installation scripts which will help you on your next projects.

OSX optional installations

OSX comes with no package manager except the AppStore. Though, it does the required common uses for casual users, it fall short when you need hundreds of tools. Thankfully, Brew is up to the task... if you are not too reluctant using the command line. Its installation is really simple:
ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
You have now access to almost every free and open source available packages.

Now, you can install an access to some already build packages as well as some proprietary software using Cask:
brew tap phinze/cask
brew install brew-cask
Here, you are. No more search attempts for finding a software. Almost every installations are available in a single command. If you don't know the name of the package, you simply search it on the command line. Here's an example for Firefox:
brew cask search firefox
==> Exact match
firefox
==> Partial matches
multifirefox

Installing VirtualBox

VirtualBox is a free and well maintained virtual machine, sometimes called hypervisor. You install it using the following command:
brew cask install virtualbox
Stay alert your password is required. Done! Nice.

Installing Vagrant

Vagrant is an helper installation program that will create for you a complete configured virtualized server. It uses a simple Ruby file as its configuration in order to produce reproducible images of your server. When you want to send a server to your coworkers, you only send this file. When you want to create a new server with some preinstalled tools, you just edit the configuration file. The same step as before is required:
brew cask install vagrant
After a required password, you got it now as an available new command.

Now your system is almost a Cloud controller (well, almost) :)

Installing and running your server OS of choice

With your new artillery, you are able to install every virtual machine image. Here, I depict the details for installing an Ubuntu server. Sure, you can choose every other OSes you may like (just Googles it :) ).
mkdir ~/VirtualBoxes
cd ~/VirtualBoxes
vagrant init UbuntuServer
This step create a simple configuration file named Vagrantfile that we will edit. Choose your favorite editor. Mine is Vim and you can turn on the syntax using the nice vim-vagrant plugin (which comes with a supercharge Vagrant command). As the Vagrantfile is a simple Ruby file, you should be able to activate the appropriate syntax highlighting.

Ready? Just edit the Vagrantfile and replace the line that starts with # config.vm.bow_url = "... with:
  config.vm.box_url = "https://cloud-images.ubuntu.com/vagrant/precise/current/precise-server-cloudimg-i386-vagrant-disk1.box"
That's all. Your server is running. Note that the installation is only done once :)

Check you virtual server

Launch VirtualBox in your Launcher and you can see it running.
My screenshot is in french but you should see where we get.

Turning off the server

Still in your command line? Vagrant helps you out also in this field:
vagrant halt

30 mars 2014

Scrum, quality focused

Lately, I had to explain the Scrum methodology used in software development. I created a little infography on this topic.


To ensure that no surprise arise at the beginning of the project, I secure it with an initial RUN-0 solely concentrated on architecture and infrastructure. This avoids multiple refactoring during the course of the project and the surprise of the product owner to have so few implemented feature at the early stage of the product.

Identically, the last RUN-final is secured so that no feature or request is taken into account. It's the project closing. Only bug fixing is allowed. This avoids any new feature that could risk the whole project delivery. This is one of the key to success and quality for Scrum projects. An other one is to take into account bug fixing during the RUN-N as high level priorities entries into the Sprint backlog.

11 mars 2014

SPDY: Shorten your page load time like no other

SPDY (say 'speedy') is a mature protocol from Google which supersets HTTP 1.1. It is available on NGINX, Apache2, NodeJS and more.

For browsers capable of talking SPDY (Chrome, Opera, Firefox), this protocol provides:
  1. Prevention of the browsers to resend information like user-agent, browser properties. In a way, it creates a session state on a normally stateless protocol.
  2. Avoidance of the HTTP pipelining mechanisms and an ability to prioritize the sent assets to its clients.
  3. A push mechanism that can pre-fill the browsers cache just like it was doing some page prefetching. If you prefer to lower the outgoing bandwidth of your servers, there are server hints that informs the browsers of the prefetching they could if they wish it.
With a default Gzip extension in case your clients does not support this protocol, coupled with PageSpeed (see : PageSpeed: Turbo ON for your sites), SPDY will transform your users experience.

Guy Podjamy, CTO of Akamai, presents a nice introductory video on SPDY. A must watch.

9 mars 2014

Debugging CoffeeScript with ease

node-inspector 0.7 has landed. It opens up a simpler solution for debugging CoffeeScript without having to use the generated JS. This post from StrongLoop explains you how: Debugging CoffeeScript with the Node Inspector Debugger.

Integrate your responsive images in one line in your HTML: the Clowncar's cookbook

Introduction

This is just a post from a cookbook that I've integrated into grunt-clowncar plugin a few month ago.

This technique for handling responsive images without polluting your HTML's code has been first mentioned by Estelle Weyl. It's like an abstract factory pattern for responsive image.

The goal is to create automagically an SVG file (the clowncar) which acts as a container. You just include the SVG file whenever you need responsive images. The SVG files contains all the necessary redirection to all the images and it loads only the appropriate ones.

Here is my original cookbook that I simply repost hereafter.

Preparing the circus

First thing first, we import the necessary tooling.
This cookbook has been created using OSX. But there's no hassle doing it with your operating system of choice.
With your favorite terminal, hop into an empty directory and create a package.json file for your project, if it's not already done.
npm init
We are using the following directory structure:
.
├── Gruntfile.coffee
├── dist
├── package.json
├── src
│   ├── img
│   │   └── photo001.jpg
│   └── index.jade
└── tmp

With :
  • Gruntfile.coffee: Our Grunt file in coffee.
  • dist: A directory for our generated web files.
  • tmp: A directory for temporary files (unminified stuff, for instance).
  • src: A directory containing all our source files that need treatment.
  • src/index.jade: The main jade file that generates our index.html file.
  • src/img: A directory containing all the image that we are about to clown.
Note that the folder strucure is easily adapted in the grunt file provided hereafter.

Now, we import our grunt plugins and a some other neat stuff:
npm install --save-dev grunt grunt-clowncar \
  grunt-contrib-jade grunt-svgmin grunt-contrib-clean \
  grunt-contrib-watch grunt-express grunt-contrib-copy \
  grunt-open matchdep

And, if you haven't done it already, install GraphicsMagick system wide. Here's the command on OSX for convenience:
brew install graphicsmagick

OK. Now, we can proceed with the 1st recipe.

Embedded SVG in HTML code

So, we start with our Grunt file Gruntfile.coffee which should look something like:
# Grunt tasks
module.exports = (grunt) ->
  # Load all plugings from our package.json files
  require('matchdep').filterDev('grunt-*').forEach grunt.loadNpmTasks
  # Project configuration
  grunt.initConfig
    # CLOWNCAR
    # Here comes the fun part. Your image are going to be clowned!
    # Every image stored in 'src/img' produces an SVG file and all
    #  the resolution required by responsive image depending on the
    #  screens that you are targeting.
    # Hereafter, the sizes matches the ones from the default
    #  Twitter's Bootstrap CSS framework.
    # Feel free to adapt them to the screens that you are targeting.
    # All our produced stuff go in the 'tmp' dir as some additional
    #  and optimizing steps are required for our beloved mobile users.
    clowncar:
      options: sizes: [1280, 992, 768, 400]
      all: files: [{
        expand: true
        cwd: 'src/img/'
        src: ['*.jpg']
        dest: 'tmp/'
        ext: '.svg'
      }]
    # Minify the produced SVG (our clown cars).
    svgmin:
      options: datauri: 'base64'
      clowned: files: [{
        expand: true
        cwd: 'tmp/'
        src: ['*.svg']
        dest: 'tmp/minified/'
        ext: '.min.svg'
      }]
    # Copy our multi-resolution images (the clowns under the car)
    #  to our production dir 'dist'.
    copy: clowned:
      expand: true
      cwd: 'tmp/'
      src: ['*-*.jpg']
      dest: 'dist/'
    # Use jade to produce HTML.
    jade: compile:
      options:
        # Minify our produced HTML file automatically.
        pretty: false
      files: 'dist/index.html': ['src/index.jade']
    # Remove everything created so far.
    clean: [ 'dist', 'tmp' ]
    # Create a custom Express server on the fly.
    express: all: options:
      port: 9000
      hostname: '0.0.0.0'
      bases: ['dist/']
      livereload: true
    # Open a browser when site is ready.
    open: all: path: 'http://localhost:<%= express.all.options.port %>'
    # Watch every changes in the 'src' dir and fire up the buuild task.
    watch:
      options:
        # Note: Livereload is not set in this task as it's already provided
        #  by the express server (the grunt task).
        livereload: false
      all: files: 'src/**', tasks: ['build']
  # Build tasks.
  grunt.registerTask 'build', ['clowncar', 'svgmin', 'copy', 'jade']
  # Default task.
  grunt.registerTask 'default', [
      'clean', 'copy', 'build', 'express', 'open', 'watch'
    ]

Pfeww, that was long. But, we've provided you with a real world example with comments. Comments that should be worth reading.

Note that there's no need in minifying our produced image with grunt-contrib-imagemin. grunt-clowncar does it for you. It uses GraphicsMagick's 'convert & thumbnail' feature that does the job pretty well.

So, the produced SVG (the clown car) are in the tmp/minified dir. On the other hand, the responsive images (the clowns or the reduced JPEGs) are directly provided in dist, the production dir. Let's target these with a neat HTML file written in Jade:
- var pageTitle = 'Check the clowns'
!!!5
html
  head
    title= pageTitle
  body
    h1= pageTitle
    include ../tmp/minified/photo001.min.svg

That's it! A oneliner for every included image.

Now, just hit the grunt command and watch your image being clowned:
grunt

Check the dev tools to see the image downloaded from the server while redimensioning your browser.

8 mars 2014

Modern HTML5 assembly line


Just a little infography that I've done for identifying tools and libraries used for developing modern HTML5 apps.