Friday, May 3, 2013

Managing history in Backbone widgets with jQuery BBQ

In my last blog post, I talked about the two different Backbone architectures we're experimenting with at Coursera: 1) single page web apps, where Backbone takes care of serving particular views for given URLs, and 2) JS widgets, where we write DIVs with particular data attributes into our HTML, and a JS module finds all of them on the page and turns them into Backbone views.

The Widget Approach

We are using this approach for our discussion forums. We have widgets for displaying lists of threads, rendering a single thread, displaying an entire forum with multiple thread lists inside it, and more. We can potentially mash up a few widgets on the same page, if we want, because they each encapsulate all of their functionality inside them.

For example, here's the bit of code that creates our forum widget:

$('[data-coursera-forum-widget]').each(function() {
  var forumId = $(this).attr('data-forum-id');

  var forum = new ForumModel({
    id: forumId
  new ForumView({
    el: $(this)[0],
    model: forum

The Problem

There was one problem with this approach, however: users kept losing their state in the widgets. For example, when a TA was paging through a long list of threads, and then they clicked away to visit one and came back, the widget would forget that it was on that page and they'd have to start from the beginning. When the forums were originally written, in the classical Web 1.0 architecture, the state was always stored in the query parameters of the URL, but now, since we are in JS-land and no longer need to change the URL to change the content of the page, we lost the URL-managed state.

That meant that we lost the ability for users to use the back button through the states of one widget, to go forward to a completely different page and back to the previous state, and to bookmark the state. Once I got enough reports from users who missed those abilities (mostly from our super users, who consider the forums to be their inbox), I realized it was time to take on this problem.

Possible Solutions

There were a few solutions that I considered and talked through with my colleagues:

  • Remember their previous state in cookies or localStorage, and always restore it from there. That wouldn't easily get me back button support, however, and it may have been odd for the user to open the forum in a new tab and see that same state.
  • Open all the links in a new tab, so that they never left the page and lost their state. Yes, I admit, this was a non-ideal solution, and I did try it for a few hours but I quickly remembered that I should not be the one deciding that the user wants those links in a new tab. Also, that wouldn't solve the problem of back button through widget state.
  • Move to the single page web app approach, and let the Backbone router manage the history. That would mean losing the mashability of my widgets, the ability to put any combination of widgets on the same page, and I'm not ready to give that up.
  • Use a JS library to store each widget's history in the URL hash, and use the hashchange event (and fallback implementations) to support the back button.

As you can guess from the title of the post, I went with the final approach. It's the only one that solved all of the users problems and let me keep my widget approach - plus it's a tried and true technique.

jQuery BBQ + Backbone

There are a few libraries out there that manage history via the hash, from very simple (js-hash) to a sophisticated polyfill approach (Hasher). The one that I was most familiar with was jQuery BBQ, and its also the one that did everything I wanted and not too much more. Plus, its docs described exactly our scenario:

<Widget> Yo, hash, update my state parameters.
<Hash> No prob, dude, done. And you didn’t even have to know about that other widget’s parameter, I just merged them in there for you.
<Widget> There’s another widget?
<Widget2> Huh? Did someone say my name?

jQuery BBQ has a straightforward API - you can pushState an object that is merged into the current hash values, you can getState on a certain key, and you can listen to the hashchanged event. To make it easy for multiple Backbone views to manage their state independently of each other, I wrote a WidgetView class with functions that can set and get state scoped to just the widget, and can trigger the view with an event whenever the widget's state changed in the hash. Here's what that class looks like:

Once my Backbone view extends that class, it can check the initial state when the view is loaded, it can set the state when the user clicks around (like on the sort or page controls), and it can listen to the state changed event to decide how to change the UI.

Here's a slimmed down version of the ThreadsView that demonstrates that:

We'll see how this works out once we build out more widgets, but so far, it seems to be working well. Let me know in the comments what approach you've taken.

Thursday, May 2, 2013

Server-side HTML vs. JS Widgets vs. Single-Page Web Apps

At the recent GOTO Chicago conference, I gave a talk on "Frontend Architectures: from the prehistoric to the Post-modern." In just my first 10 months at Coursera, I've experienced the joys and woes of many different frontend architectures, and I wanted to share what I learnt. I detail everything in the slides, but I'll summarize my thoughts here as well.

How do you pick an architecture?

We could make our decision by just checking Twitter and seeing what all the cool kids are talking about, but uh, let's pretend that we're more scientific about it than that, and figure out whats important to us as web developers.

To start off with, here are a few things that users care a lot about:

  • Usability: they can do what they want to do, quickly and intuitively
  • Linkability: they can bookmark and link to parts of the site
  • Searchability/Shareability: they can share what they're using on social networks, and find it on Google (this is important for user growth)
  • More features, less bugs: 'nuff said.

Of course, we also should think about what we as developers care about:

  • Developer Productivity: we want to be able to iterate fast, try new things, and implement features in a way that makes us feel code
  • Testability: we want confidence that our code won't break, and that we won't spend most of our time maintaining a decaying, fragile product
  • Performance: we want our servers to respond quickly and to serve minimum number of requests, and we don't want our users to wait unnecessarily.

And when developers are happier, we can more quickly and safely improve the product, and that makes users happy.

Let's compare a few architectures...

So that's what I looked at when I reviewed our different architectures, and as it turns out, there are definite differences between the architectures in those respects.

Server-side HTML ("Web 1.0")

In much of our our legacy codebase on (where instructors create their courses with a combination of lectures, quizzes, assignments, wikis, and forums), we use PHP to output HTML, and it handles much of the interaction via page reloads. A bit of JS is sprinkled (in a not so pretty manner) throughout.

This architecture suffers the most in terms of usability - it's very hard for users to do many interactions in a small amount of time - but it does have definite benefits of easy linkability, shareability, and searchability.

As a developer though, I hate it. All of our data is entangled inside the HTML, and when we want to bring JS into the HTML, it quickly becomes an untestable mess. Plus, it ties us to a particular backend language, and if we want to change languages (which we're doing, to Scala/Play), we have to rewrite the presentation layer. Yes, it's possible to use this approach in a more elegant way, and try to use templates that are portable across languages, but it's just not likely.

JS widgets

For the forums, we are starting to take a different approach: JS widgets. We write DIVs into the HTML with certain attributes (like data-coursera-forum-threads-widget) and then we include a widgets.js that turns those DIVs into Backbone views.

With this approach, we can create very dynamic interfaces with real-time updates, we can have decent linkability if we have server-side URLs and hash state, but we can't easily make the content in the widgets shareable and searchable, since bots don't understand the JS. However, that's not a big concern for us there, since much of the content is behind a login wall anyways.

From a developer perspective, we typically create APIs for the JS widgets to consume, and well-tested APIs mean that we can more easily create new frontends for the same data. After I ported our forums to the widgets approach, I was able to make new widgets for the same data in just a few hours, since I could re-use the same API. On the flip side, I have to spend more time writing tests for the frontend, since the user can change the state via sequences of interactions and many bugs may not surface until after a particular interaction.

Single-page web apps

On, we serve the same HTML file for every URL, and that HTML file contains only require.js and a call to load a routes.js file. That routes file maps URLs to Backbone views, and we rely on the Backbone Router to figure out what view to load up and to manage history using the HTML5 history API (with the fallback hash technique in older browsers).

We get many of the same benefits and drawbacks as JS widgets with single page web apps, but there are a few key differences. We can have a potentially faster user experience because of the complete lack of true page reloads, but it's harder to do simple things like internal links (you'll end up with a double hash on older browsers), or listen to a window.onunload event. We would suffer from bad searchability here, but we decided that it's really important for users to be able to find and share the course descriptions, so we wrote a "Just in time renderer" that uses Selenium to render the HTML and serve that to bots instead.

On the developer side, it's much trickier to test, because our webapp now has state *across* routes, and our tests have to check different sequences of routes.

So which one's the best?

Trick question! None of them have it all, atleast not yet. I would argue that if you're doing anything highly interactive (like an admin interface), you really want to take a heavy-JS approach, and also I'll point out that we're still in the early days of JS-heavy architectures. We will get better at developers at tackling problems like testability, and bots and browsers will likely get better at problems like searchability and performance. So, yes, it's a bit of a risk to do go as far as the single page webapp approach today, but it will be less and less of a risk as time goes on.

In terms of JS widgets vs Single-page web apps, I think they each have their places. On, I hope that we can give instructors increasing amounts of control and flexibility in how they put together their classes, and that's why I'm building out our functionality as widgets that can be combined together however they please. On, where students browse classes, we are the only ones designing that experience, and we know what we want each page to be, so it makes more sense to go the single page web app approach. The only difference is in the routes vs. the widgets file (mapping URLs vs. transforming DIVs), so we can easily create Views and Models that we use in both apps and widgets.

For my full run-down of the good and bad bits about each approach, flip through the slides and click on the images in the slides to see more code, screenshots, or videos.

What's your approach? What do you like or dislike about it? Let me know in the comments!