Caffeinated Simpleton

I got in a brief conversation in the node.js IRC channel today about how weird type conversion is in JavaScript. Specifically, this code was suspect:

What this shows is that when a Boolean object is used in an if statement, it always evaluates to true, but when it’s used in a type coercing operator (such as ==), it converts to the value it represents. Why?

The reason is actually pretty straightforward, even though it’s not obvious when looking at it. if statements convert the result of their expression to a boolean primitive, whereas == converts the right side expression into the type of the left side and checks for equivalency. So if statements don’t do type conversion at all until the expression is evaluated, and then it implicitly converts the result of that expression into a boolean primitive.

JavaScript is really just doing this:

Where the weirdness comes in is how type converting an object works. If you look at the JavaScript spec, you can see that converting an Object to a boolean is always true. And guess what? Instances of Boolean are objects:

Since new Boolean instances are objects, they will always be true when converted to a boolean primitive, which is exactly what the if statement is doing.

So then, why does new Boolean(false) == false result in true? Well it’s because the rules are different for ==. Let’s say you’re doing false == new Boolean(false). This will result in true because the left side of the expression is a primitive, so the interpreter will convert the Boolean instance, not to a boolean, but to a primitive. When converting an object to a value, the interpreter will:

Return a default value for the Object. The default value of an object is retrieved by calling the internal [[DefaultValue]] method of the object, passing the optional hint PreferredType. The behaviour of the [[DefaultValue]] method is defined by this specification for all native ECMAScript objects (8.6.2.6).

And, you guessed it, the default value of a Boolean instance is just the boolean primitive that it represents. So, after conversion, we’re testing false against false which is of course true.

The other direction also works. If we’re doing (new Boolean(false)) == false , then the interpreter will convert false to an Object. The spec specifies that to convert a boolean primitive to an object, it should:

Create a new Boolean object whose [[value]] property is set to the value of the boolean.

So then we have two equivalent Boolean instances being compared, which again results in true

Pretty crazy, but it kind of makes sense when you think about it. Maybe.

“Fuck it, let’s do it live”

Today’s complex web apps have a complicated mish-mash of server-side templates combined with client-side JavaScript. Often times there are certain calls to the server that return HTML, and others that return JSON or XML for client-side use. This not only distorts the “view” portion of MVC, but it can also be hard to develop for and, increasingly, slow. At rdio, we thought that we may be able to see substantial performance improvements across the board if we changed our templating engine from Jinja2 on the server-side to a custom, client-side templating engine. We found this to be quite true in the end, and the majority of the pages found on rdio today use Bujagali (the name we gave our system).

Goals

The goals for the project were pretty ambitious.

• Trivial to convert from Jinja to Bujagali
  • Support template inheritance
  • Support importing shared code
  • Support filters (essentially just functions accessible from the template)
• Make heavy use of client’s cache
• Reduce bandwidth usage to being as close as possible to the size of the actual data required to render
• Perform really, really fast on good browsers
• Not have terrible performance in IE
• Optionally render server-side to support googlebot and other limited JS environments

We achieved these goals, and you can learn how to use the engine and play around with the source on github. Be warned, however, that documentation is sparse right now.

If you want to learn how all the insides work, keep reading.

Architecture

The architecture is pretty simple on the surface. It comes down to heavy caching, robust versioning, a laissez-faire syntax, and flexibility in runtime environment.

A quick note about measuring performance: it’s hard. Bujagali is optimized to perform better over time, to get you your data faster by utilizing smaller payloads, and to actually be fast once we were shoving data into strings. To top it off, it performs differently depending on your client. It’s hard to do an apples-to-apples comparison of overall template performance in a case like this, but suffice it to say that we measured performance extensively at rdio and Bujagali is way, way faster in most cases than the previous Jinja2-based system. I hope to write more about performance in the future.

Caching

When it comes to caching, there’s one overriding rule: templates themselves are very cacheable, data is not. I also did not assume that the application JavaScript was very cacheable since in rdio’s case it’s not. After all, we roll out new versions of the site every day and since the application JavaScript is compressed into one file, your browser almost always has to reload the JavaScript whenever we roll out a new version. I wanted to avoid this in the case of templates for a couple of reasons. First of all, most of the templates change quite a bit less frequently than the JavaScript itself, and second, including all markup on the site would increase the size of the rapidly-changing JS file enormously.

To maximize cacheability of the templates, we turn to the browser cache itself. Each template is just a JavaScript file that contains a JavaScript function of the same name. By including the version of the file in the file name and telling the browser to cache it til we’re all long dead, we can assure that the turnaround time for the browser to get the template loaded into the program is really low. In fact in some versions of IE, loading a script that is contained in the cache is a synchronous operation. This is better explained through a bit of code.

As long as we carefully keep track of versions, we can be sure that the function loaded by the above script is the one that we want and that if the template does not change, it will never require another trip to the server to retrieve. In practice, this is a huge win as the full markup of your site really only has to be transferred once.

Versioning

The next overriding principle was strict versioning. This was pretty much necessary as a result of the heavy caching. Every time data is passed into Bujagali to be rendered, it must specify what version of the template it expects to be rendered against. By providing this information every time, we can be sure that the templates always work right in pretty much every possible proxy and caching set-up. A cool side effect of this is hot-swappable templates. If you modify a template server side and an ajax request comes in for data that’s supposed to render against that freshly modified template, Bujagali will reload the template and render it seamlessly, no refresh required.

Laissez-Faire Syntax

The final part of designing Bujagali was figuring out what we would actually allow in the templates. Jinja is very strict about what you can and cannot do. Bujagali is not. You can just write arbitrary JavaScript, and in fact writing a template in Bujagali feels a lot more like writing JS than it does HTML. This isn’t because I think there should be logic in your templates (I don’t), but I think that it’s up to the developer to decide what constitutes “program logic” and what constitutes “view logic”. Having a full-fledged language can be very useful in a template, and so arbitrary JavaScript is perfectly acceptable in Bujagali. It’s a language you already know and it won’t get in your way. An example comes from one of rdio’s own templates:

Flexibility in Runtime Environment

Despite the fact that Bujagali allows you to execute arbitrary javascript, it does not make a lot of guarantees about the environment. Bujagali requires the Underscore.js library, so it will always be available to templates, but beyond that it makes no guarantees (although it is extensible, so you can ensure that your own objects and helpers are available). This restriction gives us the flexibility to render templates either on the client side or on the server side using a server-side JavaScript environment like node.js.

This restriction on environment does have some practical benefits that keep the templates “clean”, despite the arbitrary JS that is permitted. Without access to the DOM or the rest of the running program, side effects from the templates themselves are limited. This restriction is not actually enforced by Bujagali itself, however, instead you’ll just get burned if you don’t stick to the restriction and then try to use your template in a different context.

Try it out

I’ll write more about actually using Bujagali in the future. However, the code is available now on github. The documentation is a bit thrown together right now, but it’s hopefully enough to get you started. There’s some infrastructure required since, unlike a normal templating system, this system spans the boundary between client and server. Luckily the code provided should make it fairly easy to get going if you already know what you’re doing.

It should be stated that this isn’t exactly a templating system for the faint of heart. Best of luck to all who venture on to github.

I did a little work on Cobra today and pushed the code around a little bit. Things that changed:

• Reorganized into a single closure. This helps keep internal details internal.
• Minimize with Closure Compiler
• JSLint now passes
  • I definitely recommend the JSLint VIM plugin if you’re a vimmer.
• Works in node.js (and most likely any server-side JavaScript implementation)
• Works in Internet Explorer.
  • I had never really tested this, but after fixing the things that JSLint found, the tests just passed.
  • Still need to test in IE 6. I don’t have a machine with IE 6 unfortunately.
• Added to documentation to clarify a few things.

At this point, I think the library’s safe for others to use. I tagged a 0.5 version, and I’ll be calling it 1.0 after it gets into a few more projects. Feel free to try it out!

You can find Cobra on github, or you can download version 0.5 now (tgz).

JavaScript is an amazing little language, but it’s got some quirks that turn a lot of people off. One of those quirks is this, and how it’s not necessarily what you expect it to be. this isn’t that complicated, but there are very few explanations of how it works on the internet. I find myself constantly re-explaining the concept to those who are new to JavaScript development. This article is an attempt to explain how this works and how to use it properly.

What is this?!

this is the current object. People that are used to object oriented languages use objects constantly, and this is how you access your object in JavaScript. This differs from Java and C++. This is best demonstrated by example, so let’s take a look.

class HotDog {
public:
     string getCondiments();
private:
     string condiments;
}
 
string HotDog::getCondiments() {
    return condiments; // The compiler knows to find "condiments" in the current HotDog instance
}

Ruby and Python don’t work this way. Instead, you must say “look for this variable in the current instance”

class HotDog:
    condiments
    def getCondiments(self):
         return self.condiments # "self" is a reference to the current instance of HotDog.

  class HotDog
         @condiments
         def getCondiments
              return @condiments # @condiments is an instance variable of an instance of HotDog
         end
    end

JavaScript has more in common with Ruby and Python than with Java and C++. The same thing in JavaScript is:

function HotDog() {
    this.condiments = "mustard, ketchup";
    this.getCondiments = function() {
         return this.condiments; //this is expected to be a reference to the current instance
    }
}

That’s what this is expected to be, anyway. It’s expected to be a reference to the current instance of whatever object it’s defined within.

this Gets Tricky

However, let’s say we wanted to find out what condiments were on our hotdog in 30 seconds. Assuming the HotDog class from above, that code might look like this:

var myHotDog = new HotDog();
// Call the getCondiments function in 3 seconds
setTimeout(myHotDog.getCondiments, 3000);

Many JavaScript beginners are surprised to learn that this code won’t work. It’ll give an error saying that this doesn’t have a member called condiments, even though it clearly does. What happened?!

As it turns out, this in the getCondiments function is set to the global object, window. This is because there is no binding of functions to instances in JavaScript. Whenever the instance isn’t explicitly stated, then this becomes window (at least in the browser). Writing this as myHotDog.getCondiments() indicates that you want this to be myHotDog, so it works correctly. The setTimeout function, however, just has a reference to that function. When it calls it, it’s not aware of myHotDog, so JavaScript sets this to window

“Fixing” this

There are several ways of forcing this to be what you want, and many of them touch on some of the more interesting features available in JavaScript. The first is apply. apply is a member of every function. It says “call this function with this object as this and these arguments. An example might help.

// These two are equivalent
myHotDog.getCondiments();
myHotDog.getCondiments.apply(myHotDog);
 
// You could also do this
var yourHotDog = new HotDog();
myHotDog.getCondiments.apply(yourHotDog);
 
// The above line does the same thing as:
yourHotDog.getCondiments();
 
// The above two calls will return the condiments off of your hot dog, not mine.

That doesn’t solve our setTimeout problem, however. apply actually calls the function, and we just want a reference to the function that uses the correct this. Luckily this is easily done, but let’s talk about some background first.

Lexical Scoping

In many languages (all the good ones, if you ask me), lexical scoping is supported. Lexical scoping basically allows you to access local variables in a parent function. If a function is defined within another function, it can access its own local variables as well as those of the function it was defined within. Time for another example.

function drinkBeer() {
    var beer = "Big Daddy IPA";
    function pour() {
         var glass = "Pint Glass";
         return "Poured " + beer + " into a " + glass;
   }
   function drink() {
         beer = null;
         return "Beer is all gone";
   }
   pour();
   drink();
}
drinkBeer();

This works just fine. drinkBeer cannot access glass, but pour and drink can access beer.

Lexical scoping is extremely powerful. If this small example doesn’t explain it, I encourage you to look around other examples on the internet until it’s more clear. While writing JavaScript, look out for ways this can make your life easier. Once you’re looking, they’re all over the place.

Solving our setTimeout Problem

With lexical scoping, we can easily solve our setTimeout problem.

var myHotDog = new HotDog();
setTimeout(function() {
    myHotDog.getCondiments();
}, 3000);

See what we did there? We created a new function that we passed to setTimeout. Our new function can access myHotDog, so it just applies it to the getCondiments function. Pretty slick.

Defending against this abuse

As somebody writing the HotDog class, it might be upsetting to you that people constantly need to keep this in mind when accessing your class. It would be nicer if there was a way to make your class work all the time. Fortunately, there is!

function HotDog() {
    var self = this; // self references the current this, which is correct.
    self.condiments = "mustard, ketchup";
    self.getCondiments = function() {
         return self.condiments; // self is guaranteed to be a reference to the original "this"
    }
}

HotDog is a constructor. You instantiate a new instance of HotDog by writing “new HotDog()“. In constructors, this is always your instance. So we created a new variable, my, that references the HotDog instance. This allows you to always refer to the HotDog instance, no matter how the getCondiments function is called.

Bind

Another method of “fixing” this is adding a bind attribute to every function. This allows you to pass in an object that will be your this. Many JavaScript libraries, such as Prototype, do this.

var boundFunction = myHotDog.getCondiments.bind(myHotDog);

Now boundFunction will always call getCondiments with this set to myHotDog. If you’ve been paying attention, it should be fairly obvious how bind is written.

function bind(thisObject) {
    var self = this; // this is the function here
    return function() {
        self.apply(thisObject, arguments); // (arguments is a keyword that refers to the passed in arguments)
    }
}

Cobra

My own class library, cobra, solves the this problem in a different way. It passes a reference to the original this to every function, which allows you to use some fancy features like prototypal inheritance while still not worrying about binding and the like. If you’re interested, you can find the code on github..

Whew

That’s pretty much all you need to know about this.

• this will be window whenever your function is indirectly called.
• Use apply to force this to be what you want
• You can use lexical scoping to make sure this behaves in a predictable manner

I hope that’s clear, comment if it’s not!

Palm’s Mojo SDK has just been released to the public this past week. I thought I would take the opportunity to show off some of the awesome things the SDK can do. The Mojo SDK is exceptionally good at tying your life together with the web services you use every day, so my first series will be on building a simple Meetup.com client. At first, we’ll just sync down meetups using the api key that’s associated with every individual account. Another day, we’ll add OAuth authentication to make it generally useful for anybody.

This article assumes that you have some knowledge of how to create an app, new scenes, and how to debug. If you aren’t somewhat comfortable with the SDK yet, check out the Hello World tutorial.

Setup

At first, we’ll just create a big button that puts all our meetups into the calendar. Nothing fancy. To create the scene, run:

$ palm-generate -t new_scene -p "name=sync"

Then in the stage-assistant.js file, put:

this.controller.pushScene("sync");

Creating a Calendar

The first thing that we will do is create a new calender for Meetup.com. This calendar will appear in the calendar app right next to your Google Calendar or Exchange calendar using the Palm Synergy APIs. This is great because it allows you to deliver new data to your users without having to write yet another way of presenting it. All contacts and calendars can be plugged straight into the core webOS applications.

To create a calendar, you first need to make an account:

self.accountServiceId = "palm://com.palm.accounts/crud/";
 
 /* Retrieves account if it exists, otherwise creates it */
setupAccount: function(self, k) {
    self.controller.serviceRequest(self.accountServiceId, {
        method: 'listAccounts',
        parameters: {},
        onSuccess: function(list) {
            Mojo.Log.info("Got account list: %j", list);
            if (list.list && list.list.length > 0) {
                self.account = list.list[0];
                k();
            }
            else {
                self.account = {
                    username: "justin",
                    domain: "meetup.com",
                    displayName: "Meetup.com",
                    dataTypes: ["CALENDAR"],
                    isDataReadOnly: true,
                    icons: {largeIcon: '', smallIcon: ''}
                };
                self.controller.serviceRequest(self.accountServiceId, {
                    method: 'createAccount',
                    parameters: self.account,
                    onSuccess: function(response) {
                        Mojo.Log.info("Got %j for %j", response, self.account);
                        self.account.accountId = response.accountId;
                        k();
                    }
                });
            }
        },
        onFailure: function() {
            Mojo.Controller.errorDialog("Failed to create account");
        },
        onError: function(error) {
            Mojo.Controller.errorDialog("Error creating account");
        }
    })
},

You’ll notice a few different things about this code, the odd things which are just my style. Using self instead of this is an idiom from Cobra, a JavaScript class library I wrote. k is a continuation, or the function that should be called after the account is created. k I believe is an idiom for Continuation Passing Style in Scheme.

Beyond that, this is very standard Mojo code. Services are identified as “palm://com.palm.” and all methods support onSuccess, onFailure, and onError callbacks. This code checks to see if there is already an account associated with this appId, and if there is, calls its continuation. If there is not, it creates the account and calls its continuation.

After the account is created, we can create the calendar.

self.calendarServiceId = "palm://com.palm.calendar/crud/";
 
/* Retrieves calendar if it exists, otherwise creates it */
setupCalendar: function(self, k) {
    self.controller.serviceRequest(self.calendarServiceId, {
        method: 'listCalendars',
        parameters: {
            accountId: self.account.accountId
        },
        onSuccess: function(calList) {
            Mojo.Log.info("Got calendar list");
            if (calList.calendars.length > 0) {
                self.calendar = calList.calendars[0];
                k();
            }
            else {
                self.calendar = {
                    name: "Meetup.com"
                }
                self.controller.serviceRequest(self.calendarServiceId, {
                    method: 'createCalendar',
                    parameters: {
                        accountId: self.account.accountId,
                        calendar: self.calendar
                    },
                    onSuccess: function(response) {
                        self.calendar.calendarId = response.calendarId
                        k();
                    },
                    onFailure: function(error) {
                        Mojo.Log.error("Creating calendar failed: %j", error);
                        Mojo.Controller.errorDialog("Failed to create calendar");
                    },
                    onError: function(error) {
                        Mojo.Log.error("Creating calendar failed: %j", error);
                        Mojo.Controller.errorDialog("Error creating calendar");
                    }
                });
            }
        },
        onFailure: function() {
            Mojo.Controller.errorDialog("Failed to create calendar");
        },
        onError: function(error) {
            Mojo.Controller.errorDialog("Error creating calendar");
        }
    });
},

This is almost identical to the account creation code, except it’s creating a calendar.

Now calling these functions is easy:

self.setupAccount(function() {
    self.setupCalendar(function() {
        self.buttonModel.disabled = false;
        self.controller.modelChanged(self.buttonModel)
    });
});

Pulling down events

To get the events that are associated with the account, I am using the Meetup.com JavaScript client. The client requires jQuery, but since it only requires jQuery to do JSONP, we replace that line with a Mojo call. There is no reason that we couldn’t use jQuery, but jQuery is overkill for just doing JSONP.

jQuery.getJSON(urlprefix + call_type + url, params, function(json){callback(json)})

Becomes:

var query = $H(params).toQueryString();
url = urlprefix + call_type + url + query;
Mojo.loadScriptWithCallback(url, Mojo.doNothing);

After we have the Meetup client library, getting the events is easy, but a little indirect. It takes three API calls, one to get the member id, one to get every group associated with the member id, and one to get every event in all those groups. You can see the details on Meetup’s API page.

self.client = new MeetupApiClient(Meet.Auth.apiKey);
syncCalendar: function(self) {
    // Gets my member id
    Mojo.Log.info("Syncing calendar");
    self.client.get_members({
        relation: "self"
    }, self._getGroups);
},
 
_getGroups: function(self, members) {
    Mojo.Log.info("Got members");
    var memberId = members.results[0].id;
    self.client.get_groups({
        member_id: memberId
    }, self._getEvents);
},
 
_getEvents: function(self, groups) {
    Mojo.Log.info("Got events");
    groups = groups.results;
    var groupString = groups[0].id;
    var today = new Date();
    for (var i = 1; i < groups.length; i++) {
        groupString += "," + groups[i].id;
    }
    self.client.get_events({
        group_id: groupString,
        after: today.getMonth() + today.getDay() + today.getFullYear()
    }, self._saveEvents);
},

Now self._saveEvents will receive a list of events that are coming up after today. All we need to do is loop over the list, format them as Palm calendar events, and pass them to the calendar service.

_saveEvents: function(self, events) {
    Mojo.Log.info("Saving events");
 
    self.numEventsProcessed = 0;
    self.events = events;
 
    events.results.each(function(meetupEvent) {
        var time = new Date(meetupEvent.time).getTime();
        if (meetupEvent.myrsvp != "no") {
            self.controller.serviceRequest(self.calendarServiceId, {
                method: 'createEvent',
                parameters: {
                    calendarId: self.calendar.calendarId,
                    event: {
                        eventId: meetupEvent.id,
                        subject: meetupEvent.name,
                        startTimestamp: time,
                        endTimestamp: time + 3600000, // 1 hour in ms
                        allDay: false,
                        note: self._formatNote(meetupEvent),
                        location: meetupEvent.lat + ", " + meetupEvent.lon,
                        alarm: 'none',
                    }
                },
                onSuccess: self._createdEvent,
                onError: self._errorCreatingEvent,
                onFailure: self._failureCreatingEvent
            });
        }
    });
},
 
_createdEvent: function(self, response) {
    self._checkIfSyncFinished();
},
 
_errorCreatingEvent: function(self, response) {
    Mojo.Log.error("Could not create event: %j", response);
    self._checkIfSyncFinished();
},
 
_failureCreatingEvent: function(self, response) {
    Mojo.Log.error("Failed to create event: %d, %j", self._eventsReturned, response);
    self._checkIfSyncFinished();
},
 
_checkIfSyncFinished: function(self) {
    self.numEventsProcessed++;
    if (self.numEventsProcessed == self.events.meta.count) {
 
        if (self.events.meta.next) {
            Mojo.Log.info("Fetching the next page of results...");
            self.client.nextPage(self._saveEvents);
        }
        else {
            Mojo.Log.info("Fetched all the events");
            self.buttonModel.disabled = false;
            self.controller.modelChanged(self.buttonModel);
            self.controller.get("sync-button").mojo.deactivate();
        }
    }
},

And that’s basically it! With the code above, all your Meetup.com events can be inserted into your webOS calendar. All you need is an event handler for your button.

self.buttonModel = {
    buttonLabel: 'Sync',
    disabled: true
};
 
self.controller.setupWidget('sync-button', {
    type: Mojo.Widget.activityButton
}, self.buttonModel);
 
Mojo.Event.listen($('sync-button'), Mojo.Event.tap, function() {
    self.buttonModel.disabled = true;
    self.controller.modelChanged(self.buttonModel)
    self.controller.get("sync-button").mojo.activate();
    self.syncCalendar();
});

This works assuming a view containing:

<div x-mojo-element="Button" id="sync-button"></div>

Debugging Tips

Creating a file called framework_config.json allows you to change the logging level. That will permit JavaScript messages to be output into /var/log/messages on the device. This is especially valuable if you’re working on a 64-bit linux machine where the inspector is currently not supported.

{
    "logLevel": 99 // 0 means no logging, 99 will max it out
}

Removing /var/luna/data/dbdata/PalmDatabase.db3 removes all data you may have inserted. This allows you to start over fresh, but I wouldn’t recommend doing this on your actual device. You’ll lose all your data!

Todo

This program isn’t really complete, but it’s a good start. A couple of things we still have to do are:

• Add Authentication with OAuth
• Keep track of already inserted appointments so we don’t insert duplicates
• Add automatic synchronization in the background
• Add push updates (this is impossible without server side support)
• Make things look nice

Stay tuned for articles on some or all of these exciting new features! Until then, the code is available on github.