Debugging server side REST calls with mitmproxy in Meteor

Introduction

Isomorphic javascript offers a great flexibility for developper letting them choose where the code will get executed. On the clients or on the servers. While the Chrome DevTools have evolved at an impressive pace allowing to easily debug and see what is going on with your client's REST calls, the state of server side debugging may be a little bit deceptive in this field. Hence, when you need to put data fetching within your server, checking the flows of requests / responses could be a bit cumbersome if you solely rely on transaction logs or node-inspector. You could use technics like packet sniffing using Wireshark but setting a proper environment when using TLS with your REST call is a huge configuration process.

This is where mitmproxy comes to the rescue. Its name says it all: Man In The Middle PROXY. This incredible tool relies on the same technics as the infamous security attack, this time, for a greater purpose. For this article, I'm using mitmproxy as a local proxy server allowing me to trace all outgoing and incoming transactions from my server side Meteor app. Note that it is not the only use case that mitmproxy offers. Actually, this tool is incredibly agile allowing you to debug your mobile apps with great ease. Let us put our white hat on.

Installation on OSX

Using homebrew for installing mitmproxy is a one command line call:

brew install mitmproxy

Note that this steps actually installs 2 tools mitmproxy and mitmdump as well as a set of certificates in ~/.mitmproxy. We only use the first tool in this article but the second one could be handy when you need a lightweight display of your transaction flow and an easy way to save them in a replayable log. Note also that if you also intend to use mitmproxy for debugging other applications like native client ones, you can accept the certificate ~/.mitmproxy/mitmproxy-ca-cert.pemcertificate in OSX that mitmproxy has created just for you. Just a caveat here. When you set your proxy settings in OSX, be sure to remove the local filters that Apple gently put for you if you want to see your native apps talking to your local servers.

Here's a little screenshot of a debugging session using mitmproxy:

Not that isomorphic

For REST calls, Meteor comes with its own isomorphic library: HTTP. On the client, it acts as basic AJAX calls, while on the server, it relies on the request module. Like all isomorphic libraries, this allows you to put your code logic in your clients or in your server with a nice and uniform API. But this comes at a price, it only covers the features that makes sense for both the clients and the server.

A server is not a browser. Indeed, while having cookies on the browser make sense for retaining states in your application for each user, cookies on the server starts to be a weird concept. Are these cookies used for the clients that you will serve or should the server could be seen as a client to another server? The same goes for the proxy settings on a server. Why would you need a proxy on a production server? Therefore, what is available on the client side of your app may not be available on the server side. 

Unfortunately, you could have use cases where the REST API that you are using within your server may need cookies (which is a bit weird when speaking of REpresentational State Transfer which should rely on state-less operations ☺).  Additionally, for debugging our flows, we also need the proxy capabilities that are offered out of the box by our browsers. This is where comes the nice package http-more from dandv. It exposes some already baked in feature of the request module for your server side REST calls.

You install it by replacing Meteor's default package like so:

meteor remove http
meteor add dandv:http-more

Proxyfy your REST calls in your server

Now that our toolbox is setup, it is time to orient our server so that it takes advantages of mitmproxy. This is achieved via the freshly exposed options in the HTTP library.

For orienting your request to the proxy, play nice with your TLS layer and setting a cookie jar:

HTTP.get url,
  proxy: 'http://127.0.0.1:8080'
  strictSSL: false
  jar: true
, (e, r) ->
  console.log 'GET sent through the proxy :-)'

Now, you should see all your flows of your server on mitmproxy.

Some nice mitmproxy shortcuts and moves

mitmproxy's documentation is clear and very helpful. But before let you dive in, I'm sharing some of my most used shortcuts:

• C to clear the list of flows.
• ↑ and ↓ for selecting a flow.
• ↵ for seeing the details of an exchange and q to go back to the list.
• ⇥ in detail of a flow to switch between the request and the response.

When you need to save a mitmproxy session for comparing it with other ones, you can use the save feature:

mitmproxy -w FILENAME

Once saved, you can reread it and modify it to only spare the relevant parts of your flows:

mitmproxy -nr FILENAME

Where:

• n prevents to launch a proxy server.
• r read the previous saved file.

For the editing your flows, use the following shortcuts:

• d for deleting flows.
• e for editing a request or a response.
• w then a for saving your all your modifications of the flows for the session.
• q then y for quitting edition mode.

Another nice move is mitmproxy's capabilities to replay a session. For instance, you can replay the client part of your spared session or the server part just by specifying -c or -s respectively. Very handy for debugging without assaulting the REST service that your implementing or using:

mitmproxy -c FILENAME

Scrum, quality focused

Lately, I had to explain the Scrum methodology used in software development. I created a little infography on this topic.

To ensure that no surprise arise at the beginning of the project, I secure it with an initial RUN-0 solely concentrated on architecture and infrastructure. This avoids multiple refactoring during the course of the project and the surprise of the product owner to have so few implemented feature at the early stage of the product.

Identically, the last RUN-final is secured so that no feature or request is taken into account. It's the project closing. Only bug fixing is allowed. This avoids any new feature that could risk the whole project delivery. This is one of the key to success and quality for Scrum projects. An other one is to take into account bug fixing during the RUN-N as high level priorities entries into the Sprint backlog.

Modern HTML5 assembly line

Just a little infography that I've done for identifying tools and libraries used for developing modern HTML5 apps.

Git & GitHub Foundations • Forks and Pull Requests

A simple and nice tutorial to refresh my memory when I contribute on Github.

Tutoriel Javascript : Bower

Grafikart nous propose une petite introduction très bien faite à l'utilisation de Bower, le gestionnaire de paquets du Web client :

Preprocessing the web

What is it all about?

Suppose that you need to build a complete website or webapp, despite all the simplifications provided these last years, there are still a lot of things to write. Even if you are using an intelligent editor, you will probably take hour, if not days, writing enclosing brackets, tags or things alike : You are repeating yourself and that is not agile at all. One of the main principle teached by agility is stay DRY. These repetition being solely required owing to the verbosity of HTML, JavaScript and CSS. So, it's time to put this principle even into codes.

When you move to preprocessor, you can choose a different set of languages that will keep you away from these repetitions. Therefore, you can focus more on your programs than on their grammar.

Removing the hassle on HTML : Jade

My favorite preprocessor for authoring HTML, XML*, SVG* is Jade. A little example in this field is worth a thousand words:

helloworld.html

<!DOCTYPE html>
<html lang="fr">
  <head>
    <meta charset="utf-8">
    <title>Hello World</title>
  </head>
  <body>
    <p>Hi, there</p>
  </body>
</html>

helloworld.jade

doctype html
html(lang='fr')
  head
    meta(charset='utf-8')
    title Hello World
  body
    p Hi, there

* : Yes, you read me well. I even use Jade for writing XML and SVG. It creates nice and readable source file while creating small XML and SVG files.

Lessen the burden of CSS : Sass

Sass is a well known preprocessor used for easing development of CSS. Many of its features are used as a basis of enhancements for future CSS evolutions : it's format is known as SCSS. You can see it as a superset of basic CSS. What you may not know is that Sass provides a second grammar. Actually, it's its initial grammar, much much more concise. It is called Sass. Pretty straightforward, isn't it? Like always, a little example gives you an instant overview of what you will not type anymore:

style.css

body {
  background-repeat: green;
}

.some-class {
  padding-left: 2em;
}
  .some-class .some-inner-class {
    padding-left: 1em;
  }

style.sass

body
  background-repeat: green

.some-class
  padding-left: 2em
  .some-inner-class
    padding-left: 1em

Relieve the pain of JS : CoffeeScript

Sure, asynchronous programming performs well. But in JS, keeping on repeating function and var over and over, adjusting parenthesis and brackets, keeping on doing the same patterns, is a pain. Even with the upcoming promises, vanilla JS is painfully slow to write. Powerful but so demanding on its grammar. Thanks to CoffeeScript, this pain becomes almost unnoticeable. Treated. Another little example of a simple class that prints out a message and you will see the difference:

echoingclass.js

var EchoingClass, inst1;

EchoingClass = (function() {
  function EchoingClass() {
    console.log('Echo World, or Eco me');
  }

  return EchoingClass;
})();

inst1 = new EchoingClass;

echoingclass.coffee

class EchoingClass
  constructor: ->
    console.log 'Echo World'

inst1 = new EchoingClass

Conclusion

These examples are just some of the preprocessors that I like to use. There are plenty of other ones that I haven't mentioned but used depending on the team that I've worked with: HAML, Dart, Typescript, Less, Stylus, ... 

The objective is to choose at least one of them and start being much more productive by focusing on your code, not on its grammar.

Side note
Don't you think that these language look a lot like Python? Charming snake.

Nice book on Backbone.js

If you want to invest some time on HTML5, choosing a nice MVC framework is your first challenge when you will start developing. Backbone is one of the available solutions out there. Its documentation only consists of its API which could be uneasy at first glance.

Thanks to Addy Osmani, a book explains the basics and the philosophy of Backbone.

It's pretty well written and I like to come back to it whenever I'm not sure to take the proper approach of design.

The book is available for free reading on Github : http://addyosmani.github.io/backbone-fundamentals/ and it's definitely worth buying it if you enjoy it as much as I do.

Remplacer Word par HTML5

Je ne suis pas un adepte des outils de bureautique. Dès que l'on passe d'un OS à l'autre, les résultats sont catastrophiques : le formatage est systématiquement mis à mal. S'il existex des solutions déjà éprouvées tels LaTeX, leurs formats est un nouvel apprentissage ralentissant la création.

Ma solution est de passer par HTML5. En effet, depuis CSS 2.1, il est possible d'appliquer des propriétés pour les sauts de pages à l'impression (page-break-after, page-break-inside, ...). Avec CSS 3, les media queries permettent de sélectionner les propriétés qui seront utilisées en fonction du média de rendu (screen et print, dans notre cas). Un seul document, une mise en page adaptée en fonction du rendu. Cela rappelle sérieusement le RWD, non ?

Mais voilà, après, il faut produire son fichier de sortie. Un PDF généralement. Comment faire rentrer cela dans une chaîne de production automatisée pour éviter les multiples combinaisons de touches et d'ajustements spécifiques ?

Ma solution préférée est d'utiliser PhantomJS.

Pour ceux, qui ne le connaisse pas, c'est un navigateur sans fenêtre de rendu. Quoi, mais ça ne sert à rien ? Et bien si, vous avez bien lu. Un navigateur sans fenêtre de rendu. On l'utilise pour tester les développements web ou hybrides en y injectant ses tests sous la forme de directives en JS. On peut alors vérifier si les éléments ont été rendus proprement, en combien de temps, et on peut faire des sorties sous plusieurs formats pour vérifier les régressions d'une version à l'autre. Les sorties sont du type PNG, JPEG, ... et PDF !

Et oui, c'est là que ça devient intéressant. Toute page web peut être rendue sous la forme d'un PDF via une simple ligne de commande :

phantomjs rasterize.js dist/index.html mon_impression.pdf A4

• rasterize.js est le petit script injecté permettant l'impression, disponible ici avec quelques explications supplémentaires.
• dist/index.html est votre page à imprimer.
• mon_impression.pdf est le fichier PDF qui sera créé.
• A4 est le format sélectionné et vous pouvez en changer comme bon vous semble.

Et voilà. Une petite intégration avec votre gestionnaire de tâches préférés, Grunt pour ma part, et vos fichiers sont créés à la volée dès que vous changez une virgule ou une couleur de fond.

Un livre de chevet sur le métier WPO (Web Performance Optimization)

Pour tout architecte se respectant, quelques livres viennent marquer votre vie. "High Performance Browser Networking" fait parti de cette catégorie.

Orienté Web, il est aussi utile à toute personne désirant améliorer son infrastructure, comprendre les mécanismes et implications des protocoles utilisés. Il est notamment particulièrement intéressant pour les applications mobiles, natives comme hybrides. Il reste très didactique et comme j'ai l'habitude de le dire, il se lit comme un roman de plage. Bref, un 'must have' de la part d'Ilya Grigorik.

PS : Je vous recommande la lecture de son site ainsi que ses vidéos disponibles sur YouTube.