[tarek] Fun with AngularJS & Cornice

Busting the myth about Python & async programming

Before I talk about the topic, I want to make a little digression and bust a myth about Python vs Javascript.

I have heard this sentence several times in Javascript and Node.js developers circles:

Python doesn't support async programming

I've even heard some people explaining that Python couldn't be used as an async language because of the GIL !

Talking about the GIL in this context is completely out of topic. Like Python, Javascript has something similar to the GIL (locks in "Isolates" in V8). But the GIL becomes an issue only when you run several threads in the same process.

And in web programming, we don't really use threads anymore.

Node.js is single-threaded and uses libuv to run an event loop, and feeds the loop with callbacks.

Python has also libraries and frameworks that provide event loops. In fact, Node.js was inspired by Twisted.

There are libuv, libevent & libev bindings in Python. And frameworks and libraries that use those bindings to provide event loops.

That said, I am not a huge fan of callback programming, I find it quite hard to read and debug. I like it better when I am working with thread-like objects.

Fortunately in Python we have gevent that will provide greenlets, which are pseudo-threads that wrap callbacks and an event loop - and you don't have to do callbacks anymore.

Gevent takes care of the gory details and let you do linear, functional programming that can be traced, debugged, without falling into an horrible maze of callbacks.

A code like this...

# from https://github.com/SiteSupport/gevent/blob/master/examples/concurrent_download.py
import gevent
from gevent import monkey
monkey.patch_all()

import urllib2

urls = ['http://www.google.com', 'http://www.yandex.ru', 'http://www.python.org']

def print_head(url):
    print ('Starting %s' % url)
    data = urllib2.urlopen(url).read()
    print ('%s: %s bytes: %r' % (url, len(data), data[:50]))

jobs = [gevent.spawn(print_head, url) for url in urls]

gevent.wait(jobs)

...will asynchronously download all URL pages, and let you work with greenlets.

Some people don't like this approach and use tools like Tornado, that will let you start an event loop and define callbacks, like Node.js.

But enough said - my point is made:

Yes, the Python ecosystem has the same tools than the Javascript/Node.js ecosystem to do async programming. And it has much more to offer in fact.

Cornice for the JSON APIs

For the Marketplace, we're working on building a separated application to provide a metrics dashboard.

The goal is to display interactive dashboards filled with some Highcharts based charts.

The primary goal is to replace the ones we have in the Marketplace application, that give to the users info like the number of downloads for their web apps.

For this, we're going to provide a set of JSON APIs on the top of an Elastic Search server, using Cornice.

Cornice acts as a proxy in front of the Elastic Search server, but also provides a few extra APIs and features we need - it also integrates the excellent pyelasticsearch library.

To give you an idea of how hard is to build such an application with Cornice, here's the core of the code:

es = Service(name='elasticsearch', path='/es')

@es.post(validators=(valid_json_body,), renderer='json')
def query_es_time(request):
    try:
        return request.es.search(request.validated['body'], index='time_*')
    except ElasticHttpError as e:
        request.response.status = e.status_code
        return e.error

The Service class provided by Cornice does a lot of automation here, like sending back a clean JSON error message in case the query is malformed. It also checks that we don't return a JSON list - since that can be a security hole. It makes sure the server returns a 405 if it's called with the wrong method, etc.

You get the idea: Cornice takes care of the things we never think about, and don't want to think about.

AngularJS for the client-side

I tried out Ember.js and quickly disliked the way the templating works in it, and the fact that they define objects for every element you want to add in the DOM.

Cedric gave a more detailed comparison of Ember vs Angular, and I really liked how Angular looked, so I gave it a shot and instantly liked it.

Angular will let you define new DOM directives, that get expanded on the client side at runtime.

For the dashboard, it means I can define something like this:

<dasboard server="http://data.marketplace.mozilla.org/">
    <chart title="Downloads" type="series" field="downloads_count"/>
</dashboard>

And have a nice Highchart dashboard that grabs data out of the Cornice server that's behind http://data.marketplace.mozilla.org (Fake URL!)

Defining directives in Angular is done by providing an HTML template and a bit of Javascript glue code.

In our case we also make the chart directive a sub-directive of the dashboard directive - here's an extract of the code so you get an idea:

var app = angular.module('components', []);

app.directive('dashboard', function() {
    return {
        restrict: 'E',
    scope: {},
    transclude: true,
    controller: function($scope, $element, $attrs) {
        this.server = $scope.server = $attrs.server;
        var charts = $scope.charts = [];
        this.addChart = function(chart) {
            charts.push(chart);
        }
    },
    template:
    '<div class="tabbable">' +
    '<h3>Monolith Dashboard</h3>' +
    '<div class="tab-content" ng-transclude></div>' +
    '</div>',
    replace: true
    };
});

Full code: https://github.com/mozilla/monolith/blob/master/monolith/media/app.js#L30

What I like about Angular is that it's easy to build something that's based on a collection of Plain Old Javascript Objects, so I actually made a separate library that takes care of creating a chart and interacting with the server and the user, given a few tags ids: https://github.com/mozilla/monolith/blob/master/monolith/media/lib/monolith.js#L194

On testing Javascript

I had no idea what was the state of the art for testing Javascript applications since I am a backend developer, so I used what the Angular.js team use and partially built: Testacular & Jasmine.

Testacular is a nice command-line too that will spawn a Firefox instance and run your tests in it. It has nice features like auto-running when a JS file changes, and you can remote-controll it because it uses a Node.JS server to provide interactions.

Although, one thing that annoys me in Javascript (as opposed to Python), is the fact that it's not easy to run processes in your tests fixtures.

What I needed to do is:

• run an Elastic Search server & add content in it
• run the Monolith Server
• then, run the JS tests.

In Python-land, all of this can happen in your test classes. In Javascript, unless I missed the obvious, I had to wrap it in a Makefile: https://github.com/mozilla/monolith/blob/master/Makefile#L30

It's still a bit clunky because I cannot garantee the two servers are really stopped. I should do something better. Maybe I'll end-up wrapping testacular in a Python unit tests... :)

Overall, I quite like building this kind of applications - and I think this pattern of having a light Python web service on the server side, and some kind of JS MVC-based tool on the client-side, is soon going to be the norm.