JavaScript UI Library Design

Preamble: This write-up is inspired by the talk I gave at BackboneConf 2013.

For the last year, I worked as a frontend engineer at Coursera. We constantly found ourselves needing UI widgets to decorate our interfaces, like modals, popups, tooltips, uploaders. You know, the same widgets that 99% of websites need, plus a few niche widgets.

We wanted to use the same UI widget libraries both in the legacy global-variables-roam-free codebase and the shiny-requirified-backboney codebase. We started with what most developers start with, jQuery plugins, but then we ended up coming up with our own way of architecting UI plugins to meet our particular constraints and satisfy our particular desires.

On jQuery Plugins

Now, don't get me wrong; I'm forever grateful for jQuery and its plugins ecosystem. John Resig released jQuery in 2006 with a plugin mechanism from day 1, and the first third-party jQuery plugin came out only a few weeks later. A year later, the community launched the plugin repository, further encouraging developers to create and share their plugins. They also launched jQuery UI soon after, which gave developer more of an architecture and base for UI plugin development. jQuery plugins became the defacto standard for UI library development and thanks to jQuery encouraging developers to share their creations, there are now thousands of plugins for developers to pick from.

But, that doesn't mean that jQuery plugins are perfect. To begin with, many jQuery plugins lack an architecture or vary wildly in their internal architecture. The original jQuery plugin "architecture" was as simple as attaching a method to $.fn and processing the passed in element. jQuery UI eventually introduced a generic widget factory with more of an architecture, and developers like Addy Osmani wrotes articles on best practices for plugin design. But it's hard to know how many plugins actually follow those best practices, particularly the older ones, and from a purely anecdotal perspective, I've gone through the source of enough plugins to suspect that the majority do not. Now, the internal architecture wouldn't matter if plugins were just blackboxes that did what you wanted them to do. But, they're not - they are chunks of JS code that your code depends on, code that you will likely find yourself debugging and patching to better suit your needs. If they follow a set of best practices and standard architecture,then they're more likely to work well and be easier for developers to dive into.

Besides that, jQuery plugins are inherently dependent on jQuery. Yes, most websites use jQuery these days (including all of Coursera), but it'd be better if there was a standard way to write UI libraries that did not depend on any libraries, and the UI library would only bring the libraries in that it truly needed. Plus, some of the best plugins are dependent on jQuery UI, since it includes the widget factory and offers features like draggable/sortable, and jQuery UI is a heavy addition to an app that isn't otherwise using it. It's also often difficult to use the jQuery UI JS without the CSS and its subsequent look & feel.

Rethinking UI Libraries

After trying to write some custom Coursera UI widgets as jQuery plugins, we decided that it might be more useful for us to start over and figure out how to architect our UI libraries for everything that we wanted out of them. We wanted many of the things that developers want in jQuery plugins, like customization and events, but we also wanted zero dependencies, AMD-compatibility, designer-friendly declarative usage, and more. As an example, I'll step through the creation of a library that emulates <marquee> functionality, because, seriously, there are so many times in life that I need a marquee.

The Basic Marquee Library

We'll start off with the basic library. The library user will write this HTML:

<div id="marquee-me">Yo Wassssup!</div>

And write this JavaScript:

var marquee = new Marquee(document.getElementById("marquee-me"));
window.setTimeout(function() { marquee.stop(); }, 5000);

The code that makes it work is in this JSBin. I've used standard OO JS to create a Marquee object, setInterval to start the element moving via CSS positioning, and clearInterval in the stop method.

Hidden Private Functionality

As I learned from my many years on the Google Maps API, developers using a JS API *will* find every single possible un-documented method and use them in their production code, and they'll get mad once that method stops working. After that experience, my approach now with libraries and APIs is that you should try to keep everything private until absolutely necessary; until developers are groveling at your doorstep for the functionality and you feel confident enough that you can test and maintain that functionality. Hiding functionality isn't as crucial for internal libraries, but I still think it's a useful to make it clear which methods a library author originally intended to be used by the outside.

In the original code, library users could easily call marquee.moveItMoveIt(), a method that I intended to be private (I'd never expose such a silly name!), so I want to restrict that. In the new code, I've wrapped everything in a MarqueeModule that includes a _private object with the moveItMoveIt method as a member as well as the Marquee object which now only has the stop method. Then MarqueeModule returns the Marquee object, and window.Marquee is set to that return value. There's no way now for library authors to access _private, since that's a local variable that was never exposed or returned.

Idempotent Constructor

We developers are forgetful. Okay, at least I am, and I want to make sure that if I call a constructor twice on the same element, I only really construct the UI component once. That way, I don't have to worry about checking everywhere that a UI component may have been constructed and I can just call it again for good measure. This is the fancy concept known as "idempotence".

In the previous code, if a library user called Marquee twice on #marquee-me and tried to stop() it, the marquee would go on forever and ever, because it started two setInterval's. Now, we wouldn't want that.

In the new code, I've added a private getOrMakeMarquee method that checks if there's already a Marquee object associated with the element, returns that if so, and constructs a new one if not.

Customization and Defaults

When a developer can customize their usage of a UI library by just specifying a few options, it gives them a way to change the library without having to go into the code or write their own. That's why many popular libraries boast many options (just see the Select2 demo page for an example). Of course, more options means more to maintain and test, so we don't want to add options just because, we want to add them because we have some inkling of how usage will vary of the library.

For Marquee, we might realize that library users will want to vary the distance and direction:

var marquee = new Marquee(document.getElementById("marquee-me"), 
    {direction: 'forwards', distance: 100});

They also might want to re-specify options later, and have the library pick up the new options:

var marquee2 = new Marquee(document.getElementById("marquee-me"), 
       {direction: 'backwards', distance: 1});

To support options in the new code, I added a defaults object at the top that clearly documents every option, its default value, and its possible values. I then added the customizeMarquee method that merges the passed in options with the default values, and I call that both from the constructor and the getOrMakeMarquee method.

Declarative Customization

The Coursera designers like to work in HTML and CSS to craft the interfaces, and they even have Github accounts and local servers running on their machines to make it possible for them to actually check out branches, tweak HTML, and send changes for review. They're not quite at the stage yet of JS, and given the non-linearity of our JS, it's not so easy to poke around for newbies, so the frontend engineers try to make it possible for them to edit as many things about the look and feel as possible in HTML and CSS alone. That means that when it comes to UI libraries, they should be able to customize the widgets without diving into the JS. And hey, as it turns out, developers like being able to customize in the HTML as well.

For example, for the Marquee library, it should be possible to use data attributes to customize it like so:

<div id="marquee-me" 
     data-marquee-direction="backwards" 
     data-marquee-distance="5">
     Yo Wassssup!
</div>

To support that in the new code, I modified the customizeMarquee method to check the data attributes for each option, and use that if none were specified in the JS.

Declarative Construction

Why should developers and designers have to use JS to initialize a UI widget at all? We could just have them add a specific data attribute that indicates an element should be transformed, and as long as the page included the JavaScript, we'd transform all the matching elements.

For the Marquee library, we could add an attribute like data-marquee:

<div id="marquee-me" 
     data-marquee
     data-marquee-direction="backwards" 
     data-marquee-distance="5">
     Yo Wassssup!
</div>

To support that in the new code, I added a new method called start to the public object. That method takes in an element, searches for any elements with the data-marquee attribute, and calls getOrMakeMarquee on them. We call that method on document.body when the library is loaded, so it can pick up anything that exists already. If we're using this library in a Backbone app where we often construct the HTML after the fact, then we'll still need to call that method on the view element once everything's rendered. Theoretically, we could use Mutation Observers to find out about DOM changes that introduce new data-marquee elements, but that's a new and not widely supported browser feature.

Observable

Another way to let developers customize their use of a library is to let them specify callback functions to happen at particular times in the lifecycle of a UI widget. Many libraries still do this via options, letting developers specify things like an onClick callback. That approach isn't the best, though, since it means only one callback can be assigned, and it clutters up the options hash. The preferred approach is to let developers add event listeners to the widget for whatever event they're interested in. Just like with options, events should be tested for and maintained, so we only want to fire those events that we are prepared to support.

In the Marquee library, we might realize we want library users to find out when the marquee reverses direction, like so:

var marquee = new Marquee(document.getElementById('marquee-me'));
marquee.on('reverse', function() {
  document.getElementById('marquee-me').innerHTML = marquee.direction;
});

To support that in the new code, I added an EventTarget object (copied from this article). and made the Marquee object extend it. Then I can call this.fire('reverse') at the appropriate time.

AMD-Compatible

At Coursera, there are some codebases that use RequireJS to manage dependencies, and some codebases that just shove everything into the global namespace and hope for the best. Not that I'm advocating for the latter, but hey, I bet there a lot of codebases out there that do that, especially smaller ones. We want our UI libraries to work equally well in each codebase, and not have to use separate copies for each.

In an AMD environment, we should be able to use Marquee like so:

define("lib/marquee", function(Marquee) {
  var marquee =  new Marquee(document.getElementById('marquee-me'));
});

To support that in the new code, we added a conditional at the end which checks if define is defined in the AMD way, and if so, it wraps the module in define. If not, then it attaches it to window.Marquee like before.

Declared Dependencies

Most UI libraries will indeed depend on additional libraries - and that's understandable. The code in a UI library should be focused on the unique functionality of that library, not on re-inventing the code of libraries past. So we want a way to bring in those dependencies that works in both the AMD and non-AMD environments.

For the Marquee library, I would likely bring in jQuery for my DOM manipulation and data attributes, as my current code is nowhere near being cross-browser compatible. You can see that in this code; all I do is change the define block at the end and pass the $ to MarqueeModule at the top.

I would also use LucidJS for the event emission, as it's a great little library that's easy to mix-in to an object in a non-obtrusive way. You can see that in this code.

Many of the Coursera UI libraries also bring in Underscore for data manipulation, extend to process options, debounce, and some bring in Q for promises.

Testable

As I've mentioned in many of the above points, we should have tests for all publicly exposed functionality, options, and events. In fact, we should probably test UI libraries even more than our normal application JS, because we will be using these libraries multiple times, and it is more likely that a developer from another part of the codebase will accidentally "abuse" library, using it in a way that the author never expected. If there are tests for the library, then it's easy to add regression tests for the new and interesting ways that the libraries gets used, or add a test that ensures it can't get used that way.

For the Marquee library, I wrote a small suite of 7 tests that check construction, options, declarative usage, and events. That suite is built on Mocha and Chai, but the tests could just as easily be written in QUnit, Buster, or Jasmine. The typical Coursera testing stack is Mocha and Chai with JSDom, so that the UI library tests can be quickly run outside of a browser environment as well.

I did run into a problem testing Marquee that I also ran into with Coursera libraries: now that we've hidden the private functionality, how can we test them? We don't usually care about calling them directly (though there are cases where testing that could be useful), but there are times when it would be useful if we could use Sinon to stub out private functions and simply verify that they were called during the execution of a public function. There are various posts like this one the discuss testing private functions in JS, and I don't think there's one technique that's clearly the best yet.

Documented

Developers should not have to read through a library's code to understand how to use it. Even if you've written the most beautifully usable code in the world (at this point, I will remind you how most parents think their babies are beautiful, when in fact they're screaming trolls). At the minimum, add a comment at the top which shows an example of using the library, both the HTML and the JS needed. If you want users of the library to appreciate you even more, add more examples of advanced configuration, write up an explanation of why the library exists and what future work remains, and point to examples of real world usage of the codebase. Remember, you want your colleague to look over at you with an adoring look on their face, not a begrudged glare.

For the Marquee library, I added a readme with a short history and usage examples.

Remember: when in doubt, document.

Real World Examples

Now that we've walked through the Marquee library together, I've shown you many of the design patterns that power the Coursera UI libraries. You might be wondering what these look like in a "real library", as you likely suspect that the Coursera designers never let me get away with Marquee-ing everything. You'd be right...so here are some actual Coursera UI libraries.

ReadMe

The ReadMe library displays a banner at the top of a page, which will be displayed until a particular date or until users close it a certain number of times. The Coursera frontend lead built this after constantly wanting banners like this to announce new functionality to students and admins, and it's now in heavy use.

To use this library, we'd write HTML like this - notice how we can stick data-readme-close on any element to tell the library that clicking it should close the banner:

<div data-readme="watchlist-announcement" data-readme-show-count="1" data-readme-show-until-closed="data-readme-show-until-closed" data-readme-show-expires="Jun 15, 2013" class="hide readme">  We now give students the ability to "watch" classes they're interested in, which replaces the need for TBA sessions.
  <a href="https://class.coursera.org/mooc/forum/thread?thread_id=472" target="_blank" data-readme-close="data-readme-close">Read more here.</a>  <div data-readme-close="data-readme-close" class="readme-close-icon"><span class="icon-remove"></span></div></div>

Then we'd call ReadMe on the element:

new Readme(this.$('.readme'));

You can see the full code here.

Modals

Like any proper webapp, Coursera uses a lot of modals. The modals library was built as a replacement for the Bootstrap modals library (which didn't do enough) and the fancybox library (which did too much and was heavy), so it was designed to let developers use the Bootstrap CSS if desired, but not force it.

To use it with Bootstrap CSS, we'd write HTML like so:

<div data-modal-overlay-class="coursera-overlay-dark" class="modal coursera-course-selfstudy-modal hide"> 
   <div class="modal-header"><h2>What is "self study"?</h2></div>
   <div class="modal-body"><p>Self-Study bla bla bla....</p></div>
   <div class="modal-footer"><button data-modal-close class="btn btn-primary">OK, I got it!</button></div>
</div>

We could then trigger it via an HTML anchor:

<a data-modal=".coursera-course-selfstudy-modal" role="button">?</a>

Or we could programmatically open it:

Modal(this.$('.coursera-course-self-study-modal')).open();

You can see the full code here. The start function is a bit more interesting in the Modals library, because there should only ever be one Modal open at once, so it takes care of closing previous modals and enforcing the singleton nature of this UI widget.

PopUps

The Popups library is similar to the Bootstrap popovers library, a UI component that pops up next to an anchor, and remains there until the user moves away. Coursera uses it for dropdown menus, hover cards, and more.

To use it, we'd write HTML like this for the anchor. Yes, it's a lot of HTML, and that's because we need our UI to be accessible, so we must add the appropriate ARIA attributes that signal both that the anchor is a button and that it is associated with an expanded menu.

<li class="course-topbar-nav-list-item"
    tabindex="0" role="button" aria-haspopup="true"
    aria-expanded="false" aria-owns="course-topbar-aboutus"
    data-popup="#course-topbar-aboutus"
    data-popup-bind-open="mouseenter" data-popup-direction="se">
    <a>About <i class="icon-caret-down"></i></a>
</li>

We'd write this HTML for the actual popup content - no ARIA required here, as the library itself adds what's necessary. We wrote the ARIA roles manually in the anchor HTML, as we don't necessarily run the JS until the anchor is interacted with, and it needs to be accessible from the beginning.

<div id="course-topbar-aboutus" class="course-topbar-sublist">
    <a class="course-topbar-sublist-item" href="/about/jobs">Jobs</a>
    <a class="course-topbar-sublist-item" href="/about/team">Team</a>
</div>

You can see the full code here. A few things to notice: 1) it documents the accessibility requirements clearly, 2) it enforces that only one popup is open at once, like modals, 3) if the specified activation event is mouseenter but the current browser is a touch device, it uses click instead.

And Many More

Those are some of the most frequently used UI libraries at Coursera, but as you can imagine, there are many more: a custom A/B testing framework, a media uploader using Transloadit as a backend, a rich text area with Markdown and HTML support, tooltips, calendar date picker, affix, draggable, sortable, etc. They vary in how much they adhere to the design patterns I've laid out here, but going forward, the frontend lead tries to enforce them in reviews of new libraries and backport them to old libraries. It's important that new developers that join the engineering team are presented with a consistent architecture, because they'll naturally follow that architecture when building new libraries.

Wrapping It Up

You might be looking at everything I just showed and thinking, "wait, couldn't we do all that with jQuery plugins?" I bet that you could, and I bet there are jQuery plugins out there that do all of that and more. We just went down the route of starting from scratch to see what we'd end up with at the end, given our particular constraints and desires, and I'm sharing what we came up with. I encourage you to share your own best practices for UI library design in the comments, or try out some of the ideas here in your own projects and report back.

My Year at Coursera

Just over a year ago, I realized that my passion in life is education and announced here that I would be joining Coursera as a frontend engineer to help them evolve online university-level education. It's been an incredible year, one where I've learnt a massive amount about frontend engineering, startup culture, and online education.

I've made the difficult decision to move on from Coursera engineering for a different role in the education space, but I really appreciate the time that I had at Coursera and I want to take a moment to look back on it here:

The Projects

I got the opportunity to both make new parts of our interface and re-think existing bits, both from the technology and usability point of view:

• Made the first team page (screenshot)
• Rewrote the course catalog and frontpage (screenshot)
• Wrote the interface and APIs for our social profiles (blog post)
• Improved the messaging of quiz deadlines to reduce student confusion
• Re-wrote our Django admin in Backbone/APIs instead (blog post)
• Ported our legacy codebase to Bootstrap 2 (blog post)
• Re-wrote our forums in Backbone/APIs instead (blog post)
• Coded the feature detection and half of the sign-up process for Signature Track (blog post)
• Wrote an iCal feed for deadlines in classes (blog post)
• Made a way for professors to connect forums to lectures
• Improved the performance of our frontend (blog post)
• Created a reporter wizard for students to get help and report bugs
• Wrote hundreds of tests for the frontend (blog post), Django APIs, and legacy PHP code (blog post)

The Technology

I found myself in a stack of many technologies that I'd heard of but never used on a real project, so this was a great opportunity to learn them on a deeper level:

• Frontend tools: Backbone, Underscore, Jade, Stylus
• 3rd party APIs: Transloadit, Badgeville, Google Maps Places API
• Languages: Python, PHP, Scala
• Web frameworks: Django, Play
• Amazon Services: S3, CloudFront, RDS, EC2, CloudSearch
• MySQL and phpMyAdmin

The Culture

We went from 15 to 50 people over the course of the year, and during that time, we cultivated our own unique Coursera culture, one of fun and learning:

• We came up with fun Formal Friday themes, when we got bored of the formal thing - my favorite was when everyone dyed their hair pink to match mine.
• We started a Show & Tell on Mondays, and are still doing it now, with people across the whole company sharing their hacks, their progress, even their family vacation photos.
• We ran in the MudWarriors challenge together and practiced our handstands in the hallways.
• We started an underground Twitter account.
• We invited startup founders, engineers, and our favorite Coursera professors to give tech talks about their area of expertise, always with lively discussions after.

The People

The Coursera team is a bunch of the smartest and most passionate people I know. Many of them used to be teachers or TAs, they come from all over the world, and they're all motivated by the mission to improve education for everyone. They're also a whole lot of fun. :-)

If you want to join Coursera, they are pretty much always hiring, join them!

...And stay tuned to find out about my next adventure.

Rewriting Django Admin in Backbone

Preamble:

This is an internal guide I wrote for Coursera about our site admin app, which is a Backbone "port" of Django admin. I've snapshotted it here in case it's interesting to other folks that are using Django admin and going down the same route, or just generally thinking about making an admin interface in Backbone.

A Bit of History

When Coursera first began advertising courses on www.coursera.org, the process to add a new course was quite manual: our Course Ops team would work with the university admins to draft up HTML describing the course, engineers would paste that HTML into the DB, and make edits to it as requested. Or, that's how the lore goes.

Here's how that data might look on the site:

As you can imagine, that process doesn't scale: it took unnecessary engineering honors to make incremental edits to the course descriptions, and it did not please university admins to have to wait on such a slow cycle. Ideally, they could log in, edit the description, and see the changes live, within a matter of minutes.

So we set about making an admin interface for the data. Since www.coursera.org runs off a Python/Django/MySQL backend, we first went down the expected route: Django Admin, an app that's built into Django for easy editing of database tables, complete with different permission levels, edit logs, and an extensibility mechanism.

Here's what it looked like on our data:

Django Admin was easy to set up and skin, but when we started implementing requests from Course Ops to improve the editing experience and workflow, we found we had to fight against Django Admin, to hack into its core in ways that felt wrong. We wanted a lot of different ways of editing data, which we could do via custom Django Admin widgets, but that often meant writing HTML and JavaScript into our Python code itself. We also wanted different buttons in our forms depending on the state of the model, like "open a session", and to do that, we had to modify the base templates with hacky HTML and javascript.

You can see a few of our widgets and buttons here:

We were making it work, sure, but I wasn't sure how long we could keep making it work, and if I could bring myself to look at the codebase later, knowing how we'd contorted it to meet our needs. Once I realized that I was the likely engineer to be making bug fixes and implementing feature requests, I decided it was time to move away, fast, before we got too deep into it.

Site Admin

So, over the course of the 3-day Labor Day weekend, I wrote "site admin", an admin interface that used our new approach to writing frontends, with a Backbone frontend and a RESTful Django API. It wasn't feature-complete with Django admin after those 3 days (and it still isn't), but it was built to be extendible on the client-side in a way that Django Admin was not, and that has made it much easier to build on.

Let's walk through the API and the frontend.

The API

The site admin API is designed to be exactly the sort of API that Backbone expects, a RESTful JSON API. It's based off an open-source project called Djangbone, but is now heavily modified.

In admin_api/views.py, the RestrictedAdminAPIView extends the generic Django View class, and defines get/post/put/delete methods that respond to the appropriate HTTP verb and understand how to generically fetch/create/edit/delete any model/collection. That file also contains AdminLogsView, which handles creation and retrieval of edit logs, and AdminSearchView, which is used by autocompletes in the frontend for finding models.

To set up an API for a particular set of models, we'd follow these steps, using the categories app and Category model as an example:

• Update categories/models.py: We add a static method to a Model class that returns back all of the models that the given user is allowed to administer.

  class Category(models.Model):
      @staticmethod
      def objects_administerd_by_user(user):
          if user.is_superuser:
              return Category.objects.all()
          else:
            return Category.objects.none()
  

• Create categories/admin_api.py: In it, we create a new view that extends RestrictedAdminAPIView, where we specify the base_queryset and base_model, corresponding to our Model, and we provide a list of fields that should be serialized into the JSON (we do not want to pass down all fields, particularly in the case of models with related fields, like course students). We also define get_add_form and get_edit_form, which return a Django Form subclass based on the request. We often serve different forms to different users, like when we want looser field validation for super users versus university admins.

  CATEGORY_FIELDS = (
                    'name',
                    'description',
                    )
  
  
  class CategoryAdminAPIView(RestrictedAdminAPIView):
      base_queryset = Category.objects.all()
      base_model = Category
  
      serialize_fields = CATEGORY_FIELDS + ('id', 'short_name')
  
      def get_add_form(self, request):
          if request.user.is_superuser:
              return NewCategoryAdminAPIForm
          else:
              return None
  
      def get_edit_form(self, request):
          if request.user.is_superuser:
              return EditCategoryAdminAPIForm
          else:
              return None
  
  
  class EditCategoryAdminAPIForm(AdminAPIForm):
  
      protected_fields = [
                          'name',
                          'short_name',
                          ]
  
      class Meta:
          model = Category
          fields = CATEGORY_FIELDS
  
      def clean(self):
          cleaned_data = super(EditCategoryAdminAPIForm, self).clean()
          short_name = cleaned_data.get('short_name')
          if short_name is not None and len(short_name) > 20:
              err = 'Please limit to 20 chars (currently %d).' % (
                  len(short_name))
              self._errors['short_name'] = self.error_class([err])
              del cleaned_data['short_name']
          if short_name is not None and not re.match('^[a-z0-9-\.]+$', short_name):
              self._errors['short_name'] = \
                  self.error_class(['Please limit to a-z,0-9.'])
              del cleaned_data["short_name"]
          name = cleaned_data.get('name')
          if name is not None and len(name) > 60:
              err = 'Please limit to 60 chars (currently %d).' % (
                  len(name))
              self._errors['name'] = self.error_class([err])
              del cleaned_data['name']
          return cleaned_data
  
  
  class NewCategoryAdminAPIForm(EditCategoryAdminAPIForm):
  
      class Meta:
          model = Category
          fields = CATEGORY_FIELDS + ('short_name',)
  

• Update admin_api/urls.py: We add 2 URL patterns for this model's API, to handle the model and collection verbs:

  url(r'^categories$',
    CategoryAdminAPIView.as_view(),
    name="api_categories"),
  url(r'^categories/(?P\d+)',
    CategoryAdminAPIView.as_view(),
    name="api_category"), 
      

• Update admin_api/tests.py: We add tests for the API, checking permissions and validation.

     def test_create_category(self):
      client = Client()
  
      cats_url = reverse('api_categories')
  
      # Test: super-user can create category
      create_and_login_as_superuser(client)
      data = {
          'name': 'Fake Category',
          'short_name': 'fake-cat',
      }
      response = self.post_json(client, cats_url, data)
      response_json = simplejson.loads(response.content)
      self.assertEqual(response.status_code, 200)
      self.assertEqual(response_json['name'], data['name'])
      self.assertEqual(response_json['short_name'], data['short_name'])
  
      # Test: cant edit short name after its created
      cat_url = reverse('api_category', args=[response_json['id']])
      data['short_name'] = 'fakecat2'
      response = self.put_json(client, cat_url, data)
      response_json = simplejson.loads(response.content)
      self.assertEqual(response.status_code, 200)
      self.assertEqual(response_json['short_name'], 'fake-cat')
  
      # Test: Cant use a shitty short name
      data['short_name'] = 'SoGreat OMG'
      response = self.post_json(client, cats_url, data)
      response_json = simplejson.loads(response.content)
      self.assertEqual(response.status_code, 400)
      self.assertEqual(response_json['short_name'],
          ['Please limit to a-z,0-9.'])
  
      # Test: instructors cant create categories
      create_and_login_as_instructor(client)
      response = self.post_json(client, cats_url, data)
      self.assertEqual(response.status_code, 400)
  

The Frontend

The site admin frontend is designed in a similar way as the backend: generic views that understand models and collections generally, with ways to specify the differences for each model.

ModelAdminPageView is responsible for creating a page with a header, banner, and then nesting a ModelAdminFieldsView which knows how to create a form with fields, buttons, and links to related models. To create that form, ModelAdminFieldsView calls upon a number of views which extend FieldView, like Select2View and HiddenInputView, and renders them depending on the attributes <-> fields mapping in a model.

CollectionAdminPageView is responsible for creating a page with a header, "new model" button, and then nesting a CollectionAdminListView, which knows how to create statistics charts via the nvd3 library and a tabular view of a collection via CollectionAdminTableView.

To add a model to the frontend, we follow these steps, using the category model as an example:

• Create models/CategoryAdminModel.js: This model extends AdminModel, an extension of Backbone.Model. It defines properties that are needed by Backbone, like the API endpoint, as well as custom properties that are needed by the views (and yes, that is not a perfect separation of data and presentation, but life must go on). The custom properties include a mapping of attributes to form field types, any custom buttons and modals, attributes to filter by in the table view, and more.

  define(["jquery",
            "backbone",
            "underscore",
            "js/core/coursera",
            "pages/site-admin/models/AdminModel"
            ],
  function($, Backbone, _, Coursera, AdminModel) {
  
    var model = AdminModel.extend({
  
      url: 'admin/categories',
      webUrlLabel: 'categories',
      label: 'Category',
  
      displayName: function() {
        return this.get('name');
      },
  
      fieldsets: function() {
  
        return [{
          name: 'name',
          type: 'text'
        }, {
          name: 'short_name',
          type: 'text',
          readonly: !this.isNew()
        }];
      }
  
    });
  
    return model;
  
  });
  

• Create collections/CategoriesAdminCollection.js: This extends AdminCollection, and specifies a handful of properties needed by Backbone and our views. The bulk of the custom logic is in the model, not the collection.

  define(["backbone",
            "js/core/coursera",
            "pages/site-admin/collections/AdminCollection",
            "pages/site-admin/models/CategoryAdminModel"
            ],
  
  function(Backbone, Coursera, AdminCollection, CategoryAdminModel) {
  
    var collection = AdminCollection.extend({
  
      url: 'admin/categories',
  
      webUrlLabel: 'categories',
  
      label: 'Categories',
  
      model: CategoryAdminModel
  
    });
  
    return collection;
  });
  

• Update site-admin/routes.js: This routes file extends Backbone.Router and defines generic URLs that can handle any AdminModel or AdminCollection. We add the new models and collections to modelAcls and collectionAcls, respectively, so that we know what collections to link a user to in the dashboard view. We enforce actual ACLs on the server-side, of course.

The future

The site admin API and frontend have served us reasonably well as we have grown to want more editing abilities and workflow improvements, but there is much to be improved upon.

Collaborative Editing

While developing site admin by myself in my living room, there was one thing that never occurred to me: I was building a collaborative editing interface. We have many different sorts of admins that edit course data, everywhere from super users to university admins to instructors to TAs, and sometimes, there could be multiple admins editing at once. I didn't realize this, however, until after we unleashed site admin for a big launch and I got emails at 4am about instructors losing data and freaking out. I quickly realized how easily that could happen, if two staff were working on a course on different machines. As a quick fix, I added a notion of "protected fields" - fields that weren't allowed to go from something to nothing - and that prevented the worst case of admins losing all the text they'd painstakingly inputted. It does add a problem, however, of admins legitimately wanting to clear fields sometimes, and engineers needing to manually make that change. It also doesn't protect against losing incremental data in a field.

To make site admin work better for multiple editors, there are a few approaches we've thought of, which could be combined in some optimal way:

• Partial updates: Currently site admin does a PUT of the full data of the model, and saves it wholly. An approach we take in many of our frontends now is to use Backbone's changedAttributes to track what changed, and only do an HTTP PATCH and partial update of the model. That would mean two admins could edit different fields and not worry about overriding eachother's changes.
• Real-time update: We could poll for updates to the model, updating the fields when we see changes. If the admin is currently editing an updated field, we could alert to confirm override or show them the changed version.
• Notifications: We could keep track of what admins are on a page, and alert them about the possibility of concurrent edits, which might encourage them to consult with eachother about what changes they are making.
• Change confirmations: We could detect that something had changed since the admin started editing, and prompt the admin to confirm that yes, indeed, they are okay with that change.

DRYer Permissions

We currently have code in the frontend that mirrors the permissions on the backend, like to figure out what collections a particular admin has access to at all, and to figure out what fields should be read-only for particular admins. This code is problematic since it means twice the code to change permission, and it's also dangerous because an engineer could fool themself into thinking that they'd secured something properly, if they did not put a test on the backend.

A better approach might be an API that sends down the permissions for an admin, perhaps using the HTTP OPTIONS verb. For example, an HTTP OPTIONS request to 'admin/api' might return the following:

{"collection": ["categories", "universities", "instructors"]}

Once loading the form for a particular model, the HTTP OPTIONS request might return the following, based on inspecting the Django Form instances:

{"fields": {
   "short_name": {"read_only": false, "restriction": "[a-zA-Z]", "max_length": "20"},
}

Those restrictions would also depend on whether the model was new or existing, and that would need to be represented in that API.

Soft Delete

When we do a delete in site admin now, it does a true delete, removing the rows and related rows from the database tables. That is a scary operation, of course, since it means that the only way to get back the data is to find it in an old database backup, provided we still have it around for the time of deletion.

We would prefer to do a soft delete, which would mean adding a "deleted" column to each model, setting that to true upon deletion, and changing all of our APIs to honor that deleted column when fetching data. This also makes it easier to do an undo. The biggest hurdle to this is change would be auditing all of the APIs that call upon that data to make sure they respect the change.

Drafts vs. Master

When an admin edits and saves their changes in site admin now, the changes are immediately live. Admins don't always want this; they often want to be able to preview their changes, feel confident in them, and then make them live. We have this in place for our course admin data, like quizzes and lectures, but it would require significant changes to the database tables, admin APIs, and user-facing APIs. A step in the right direction might be to make it possible for admins to preview their course pages with the unsaved data by sending it through an iframe and postMessage, perhaps. That is made more difficult by the fact that our admin APIs represent the data in a very different form than the user-facing APIs, however.

Improved Logs

We currently record who did what to which model, but the "what" is only whether it was a creation, edit, or delete. Ideally, we would also include exactly what fields were changed, and what the diff was between the fields. That would be made much easier to do by adding HTTP PATCH support. We also need better pagination and searching of the logs, and we may want to expose them to non-super-users.

Workflow-Based vs. Model-Based

Django admin revolves around models, assuming that the admin thinks to themself, "Yes, I'd like to edit such and such today." However, as it turns out, admins more often think in terms of goals or workflows, such as "Today I'd like to release grades." or "Today I'd like to open the course for enrollment and let all the subscribers know." These workflows often involve different models at each step, and it is non intuitive for the user to have to figure out that sequence. For a few of them, we've created "Checklist" views, with steps and links to the relevant models, anchor'ed at the form element that should be edited. But I think it would be worth it to re-think the admin interface from scratch with the idea of workflows being a first class citizen, instead of something we've tacked on at the end.

A Guide to Writing Backbone Apps at Coursera

Preamble:

At Coursera, we made the choice to use the Backbone MVC framework for our frontends, and over the past year, we've evolved a set of best practices for how we use Backbone.

I wrote a guide for internal use that documents those best practices (much of it based on shorter blog posts here), and I've snapshotted it here on my blog to benefit other engineering teams using Backbone and to give potential Coursera engineers an idea of the current stack. This was snapshotted on July 24th, 2013, so please keep in mind that the Coursera frontend stack may change over time as the team figures out new and better ways to do things.

If you're interested in joining Coursera, check out the many job listings here. The frontend team is a really smart and fun bunch, and there are a lot of interesting technical and usability challenges in the future.

The Architecture

There are many different frontend architectures to choose from, and at Coursera, we have made the deliberate decision to opt for a very JavaScript-heavy, JavaScript-dependent approach to our frontend architecture:

We build up the entire DOM in JavaScript, loading the data via calls to RESTful JSON APIs, and handle state changes in the URL via the hash or HTML5 history API.

This approach has several advantages, atleast as compared to a traditional data-rendered-into-HTML approach:

• Usabiity: Our interfaces can easily be dynamic and real-time, enabling users to perform many interactions in a small period of time. This is particularly important for our administrative interfaces, where users want to be able to drag-and-drop, tick things on and off, and generally manipulate many little things that are present on one screen.
• Developer Productivity: Since this architecture relies on the existence of APIs, that makes it easy for us to build new frontends for the same data, which encourages experimentation with new ways of viewing the same data. For example, after porting our forums to this architecture, I was able to create portable sidebar widgets based off the forums API in just a few hours.
• Testability: The APIs and the frontends can both be tested separately and rigorously using the best suite of tools for the job.

It also has a few disadvantages:

• Linkability: We have to go through a bit more work to make the JS-powered interfaces linkable, and previously simple things like internal anchors (page#section) are surprisingly difficult to implement.
• Search/shareability: Since Facebook bots and search bots do not handle JS-rendered webpages as well, we have to go through more work to make our public pages indexable by them, which we've done through our Just-in-time renderer.
• Testability: We have to write far more tests for our JS frontends since the user can change state via sequences of interactions, and some bugs may not surface until a particular sequence. We also now have state across URL routes when we use the HTML5 history API, and may have to test across multiple views.
• Performance: We must be constantly monitoring our JavaScript to make sure we are not pushing the browser to do too much, as JavaScript can still be surprisingly slow at processing data and turning it into DOM.

However, given the usability benefits of the JS-rendered approach, we have elected to stick with it, and we will need to become experts in overcoming the disadvantages of the approach. At the same time, we can hope that the browsers and tools make those disadvantages slowly disappear, as this is an increasingly popular approach.

The APIs

We have APIs coming from Python/Django, PHP, and Scala/Play. We try to be consistent in the API design, and when possible, we opt for a RESTful JSON API.

For example, if I want to retrieve information about a forum, we'd perform an HTTP GET to a RESTful URL and expect a JSON to come back with an "id" attribute and other useful attributes.

Request:

HTTP GET /api/forums/1

Response:

{
    "id": 1,
    "parent_id": -1,
    "name": "Forums",
    "deleted": false,
   "created": "1369400797"
}

To create a new forum, we'd perform an HTTP POST to a RESTful URL with our JSON, and expect JSON to come back with the "id" filled in:

Request:

HTTP POST /api/forums/
{
    "parent_id": -1,
    "name": "Forums"
}

Response:

{
    "id": 1,
    "parent_id": -1,
    "name": "Forums",
    "deleted": false,
   "created": "1369400797"
}

To update an existing forum, we could do an HTTP PUT with the full JSON of the new properties, but when possible, we prefer to do an HTTP PATCH, only sending in the changed properties. That is a safer approach and means we are less likely to change attributes that we did not intend to change, and also makes our interfaces more usable by multiple people at once.

Request:

HTTP PATCH /api/forums/1
{
    "name": "Master Forums"
}

Response:

{
    "id": 1,
    "parent_id": -1,
    "name": "Master Forums",
    "deleted": false,
   "created": "1369400797"
}

To delete a forum, we could do an HTTP DELETE, but we prefer instead to set a deleted flag on the object, and make sure that we respect the flag in all of our APIs. We often have users that accidentally delete things, and it is much easier to restore if the information is still in the database.

Request:

HTTP PATCH /api/forums/1
{
    "deleted": true
}

Response:

{
    "id": 1,
    "parent_id": -1,
    "name": "Master Forums",
    "deleted": true,
   "created": "1369400797"
}

If we are retrieving many resources, we may want a paginated API, to avoid sending too much information down to the user. Here's what that might look like:

Request:

HTTP GET /api/forum/search?start_page=1&page_size=20

Response:

{"start_page": 1,
  "page_size": 20,
  "total_pages": 40,
  "total_results": 800,
  "posts":  [ ... ]
}

The JavaScript

JavaScript is a powerful language, but it can easily become a jumbled mess of global variables and files that are thousands of lines long. To keep our JavaScript sane, reusable, and modularized, we chose to use an MVC framework. There are approximately a million* MVC frameworks to choose from, but we chose Backbone.JS as it has a large community of knowledge built up around it and it is lightweight enough to be used in many different ways.

Backbone

Backbone provides developers with a means of separating presentation and data, by defining Models and Collections for the data, Views for the presentation, and triggering a rich set of events for communication between the models and views. It also provides an optional Router object, which can be used to create a single page web app that triggers particular views based on the current URL route. For a general introduction to Backbone, see these slides.

There is a lot that Backbone does not provide, however, and it's up to the app developer to figure out what else that app needs, and how much of that you'll get from open-source libraries or decide to write yourselves. That's good because Backbone can lend itself to many different sorts of apps, with the right combination of add-ons, but it's bad because it takes longer to find those add-ons and get them working happily together. As a company, it is in our best interest to converge on a recommended set of add-ons and best practices, so that our code is more consistent across the codebase. At the same time, it's also in our best interest to continually challenge our best practices and make sure that we are using the right tool for the job. If we discover a particular add-on is too buggy or slow, we should phase that out of the codebase and document the reasons why.

There is a larger question, of course: Is Backbone the right framework for us, given how many new frameworks have come out recently that may entice us with promises of speed and flexibility? That is not a question that I have an answer for, but I do think that one can spend forever trying out new frameworks to find the perfect one, and time might be better spent building up best practices around a single framework. However, there may be a time at which we become sufficiently convinced that Backbone is no longer working for our codebase and it is worth the cognitive effort and engineering resources to invest in a new framework.

Here's an exploration of the add-ons and best practices that we use in our Backbone stack.

Backbone Models

A basic model might look like this:

define([
  'underscore',
  'backbone',
  "pages/forum/app",
  "js/lib/backbone.api"
],function(_, Backbone, Coursera, BackboneModelAPI) {

  var model = Backbone.Model.extend({
    api: Coursera.api,
    url: 'user/information'
  });

  _.extend(model.prototype, BackboneModelAPI);

  return model;
});

We start off by declaring the JS dependencies for the model:

• underscore: This is a collection of generic utility functions for arrays, objects, and functions, and it's common to find yourself using them, so most models and views will include it.
• backbone: This is necessary for extending Backbone.Model
• pages/forum/app: Every model will depend on an "app.js", which defines a base URL for API calls and a few other details. It adds objects to the Coursera singleton variable, like Coursera.api, which is used by the Model.
• js/lib/backbone.api: This is a Backbone-specific wrapper for api.js that overrides the sync method and adds create/update/read/delete methods. The api.js library is an AJAX API wrapper that takes care of emulating patch requests, triggering events, showing AJAX loading/loaded messages via asyncMessages.js, and creating CSRF tokens in the client.

Then we define an extension of the Backbone.Model object with api and url options that help Backbone figure out where and how to pull the data for the model, and it mixes in the BackboneModelAPI prototype at the end of the file.

Backbone Models: Relational Models

Out of the box, Backbone will take JSON from a RESTful API and automatically turn it into a Model or a Collection of Models. However, we have many APIs that return JSON that really represent multiple models (from multiple tables in our MySQL database), like courses with universities:

[{"name": "Game Theory",
 "id": 2,
 "universities": [{"name": "Stanford"}, {"name": "UBC"}
]

We quickly realized we needed a way to model that on the frontend, if we wanted to be able to use model-specific functionality on the nested models (which we often do).

Backbone-relational is an external library that makes it easier to deal with turning JSON into models/collections with sub collections inside of them, by specifying the relations like so:

var Course = Backbone.RelationalModel.extends({
   relations: [{
      type: Backbone.HasMany,
      key: 'universities',
      relatedModel: University,
      collectionType: Universities
    }],
});

We started use that for many of our Backbone apps, but we've had some performance and caching issues with it, so we've started stripping it out in our model-heavy-apps and manually doing the conversion into nested models.

For example, heres how the Topic model turns nested courses array into a Courses collection:

  var Topic = Backbone.Model.extend({
    defaults: {},

    idAttribute: 'short_name',

    initialize: function() {
      this.bind('change', this.updateComputed, this);
      this.updateComputed();
    },

    updateComputed: function() {
      var self = this;
      if (!this.get('courses') || !(this.get('courses') instanceof Courses)) {
        this.set('courses', new Courses(this.get('courses')), {silent: true});
        this.get('courses').each(function(course) {
          if (!course.get('topic') || !(course.get('topic') instanceof Topic)) {
            course.set('topic', self);
          }
        });
      }
   }
  });

For a trickier example, here's how the Course model sets a nested Topic model. It has to require the Topic file dynamically, to avoid a cyclic dependency in the initial requires which will wreak all sorts of havoc:

  var course = Backbone.Model.extend({
    defaults: {},

    initialize: function() {
      this.bind('change', this.updateComputed, this);
      this.updateComputed();
    },

    updateComputed: function () {
      // We must require it here due to Topic requiring Courses
      var Topic = require("js/models/topic");
      if (this.get('topic') && !(this.get('topic') instanceof Topic)) {
        this.set('topic', new Topic(this.get('topic')), {silent: true});
      }
   });

We could also look into using Backbone.nested, which seems like a more lightweight library than Backbone.Relational, and it may have less performance issues.

Backbone Views

Here's what a basic Backbone view might look like:

define([
  "jquery",
  "underscore",
  "backbone",
  "js/core/coursera",
  "pages/site-admin/views/NoteView.html"
  ],
function($, _, Backbone, Coursera, template) {
  var view = Backbone.View.extend({
    render: function() {
      var field = this.options.field;
      this.$el.html(template({
        config: Coursera.config
        field: field,
      }));
      return this;
    }
  });

  return view;
});

We start off by declaring the JS dependencies for the view:

• jquery: We often use jQuery in our views for DOM manipulation, so we almost always include it.
• underscore: Once again, underscore's utility functions are useful in views as well as models (in particular, debounce and throttle are great for improving performance of repeatedly called functions.)
• backbone: We must include Backbone so that we can extend Backbone.View.
• js/core/coursera: We include this so that we have a handle on the Coursera singleton variable, which contains useful information like "config" that includes the base URL of assets, which we often need in templates.
• pages/site-admin/views/NoteView.html: This is a particular Jade template that's been auto-compiled into an *.html.js file, and we include it so we can render the template to the DOM. We try to keep all of our HTML and text in templates, out of our view JS.

Then we create the view and define the render function, which passes in Coursera.config and a view configuration option into a template, and renders that template into the DOM.

Backbone Views: Templating

Backbone requires Underscore as a dependency, and since Underscore includes a basic templating library, that's the one you'll see in the Backbone docs. However, we wanted a bit more out of our templating library.

Jade is a whitespace-significant, bracket-less HTML templating library. It's clean to look at because of the lack of brackets and the enforced indenting (like Python and Stylus), but one of it's best features is that it auto-closes HTML tags. We've dealt with too many strange bugs from un-closed tags, and it's one more thing we don't have to worry about when using Jade. Here's an example:

div
    h1 {#book.get('title')}
    p
    each author in book.get('authors')
        a(href=author.get('url')) {#author.get('name')}
    if book.get('published')
        a.btn.btn-large(href="/buy") Buy now!

We could also consider using Handlebars, Mustache, or many other options.

Backbone Views: Referencing DOM

Inside a view, we find ourselves referencing the DOM from the templates repeated times, like to set up events, read off values, or do slight manipulations. For example, here's what a view might look like:

var ReporterView = Backbone.View.extend({
  render: function() {
    this.$el.html(ReporterTemplate());
  },
  events: {
     'change .coursera-reporter-input': 'onInputChange'
     'click .coursera-reporter-submit': 'onSubmitClick'
  },
  onInputChange: function() {
    this.$('.coursera-reporter-submit').attr('disabled', null);
  },
  onSubmitClick: function() {
    this.model.set('title', this.$('.coursera-reporter-input').val());
    this.model.save();
  }
});

There are a few non-optimal aspects of the way that we reference DOM there:

• We are repeating those class names in multiple places. That means that changing the class name means changing it in many places - not so DRY!
• We are using CSS class names for events and manipulation. That means our designers can't refactor CSS safely without affecting functionality, and it also means that we must come up with very long overly explicit class names to avoid clashing with other CSS names, since we bundle our CSS together..

To avoid repeating the class names, we can store them in a constant that is accessible anywhere in the view, and only access them via that constant. For example:

var ReporterView = Backbone.View.extend({
  dom: {
     SUBMIT_BUTTON: '.coursera-reporter-submit',
     INPUT_FIELD:   '.coursera-reporter-input'
  },
  render: function() {
    this.$el.html(ReporterTemplate());
  },
  events: function() {
    var events = {};
    events['change ' + this.dom.INPUT_FIELD]    = 'onInputChange';
    events['click ' +  this.dom.SUBMIT_BUTTON]  = 'onSubmitClick';
    return events;
  },
  onInputChange: function() {
    this.$(this.dom.SUBMIT_BUTTON).attr('disabled', null);
  },
  onSubmitClick: function() {
    this.model.set('title', this.$(this.dom.INPUT_FIELD).val());
    this.model.save();
  }
});

As a bonus, this technique gives us easier-to-maintain testing code:

it('enables the submit button on change', function() {
  chai.expect(view.$(view.dom.SUBMIT_BUTTON).attr('disabled'))
 .to.be.equal('disabled');
  view.$(view.dom.INPUT_FIELD).trigger('change');
 chai.expect(view.$(view.dom.SUBMIT_BUTTON).attr('disabled'))
 .to.be.equal(undefined);
});

As for the use of class names entirely, we can avoid them by using data attributes instead, perhaps prefixing with js-* to indicate their use in JS. We would still have CSS class names in the HTML templates, but only for styling reasons.

So then our DOM would look something like:

var ReporterView = Backbone.View.extend({
  dom: {
     SUBMIT_BUTTON: '[data-js-submit-button]',
     INPUT_FIELD:   '[data-js-input-field]'
  },
...
});

Note that selecting via data attributes shown to be less performant but for the vast majority of our views, that performance difference is insignificant.

Backbone Views: Data Binding

Backbone makes it easy for you to find out when attributes on your Model have changed, via the "changed" event, and to query for all changed attributes since the last save via the changedAttributes method, but it does not officially offer any data ⟺ dom binding. If you are building an app where the user can change the data after it's been rendered, then you will find yourself wanting some sort of data binding to re-render that data when appropriate. We have many parts of Coursera where we need very little data-binding, like our course dashboard and course description pages, but we have other parts which are all data-binding, all-the-time, like our discussion forums and all of our admin editing interfaces.

Backbone.stickit is a lightweight data-binding library that we've started to use for a few of our admin interfaces. Here's a simple example from their docs:

Backbone.View.extend({bindings: {
    '#title': 'title',
    '#author': 'authorName'
  },render: function() {
    this.$el.html('<div id="title"/><input id="author">');
    this.stickit();
  }
});

We still do custom data-binding for many of our views (using the "changed" event, changedAttributes(), and partial re-rendering), and I like that because it gives me the most control to decide exactly how a view should change, and I don't have to fight against a binding library's assumptions.

We could also consider using: KnockBack

Maintaining State: Single-Page-Apps vs. Widgets

After we've created a view for our frontend, we still have big decisions to make:

• How will users get to that view?
• What state of the view will be kept in the URL, i.e., what can the user press back on and what can they bookmark?
• Will our view be used in multiple parts of the site or just one?

In our codebase, we have two main approaches to those questions: "single page apps" and "widgets".

Single-Page-Apps

Besides being the buzz word du jour, a single-page-app ("SPA") is what Backbone was originally designed for, via its Backbone.Router object. A SPA defines a set of routes, and each route is mapped to a function that renders a particular view into a part of the page. Backbone.History then takes care of figuring out which route is referred to by the current URL, and calling that function. It also takes care of changing the URL using the HTML5 History API (which makes it appear like a normal URL change) or window.location.hash in older browsers.

For example, we could have this routes file:

define([
  "jquery",
  "backbone",
  "pages/triage/app"
],
function($, Backbone, Coursera) {

  var routes = {};
  var triageurl   = Coursera.config.dir.home.replace(/^\//, "triage");

  routes[triageurl + '/items'] = function() {
    new MainView({el: $('.coursera-body')});
  };

  Coursera.router.addRoutes(routes);
 
  $(document).ready(function() {
      Backbone.history.start({pushState: true});
  });
});

After declaring its dependencies, it defines a mapping of routes, adds those to our global Coursera.router (an extension of Backbone.Router) and then kicks off Backbone.history.start() on page load.

SPAs: Syncing Users

More typically, for our logged in-apps, we will attempt to login the user before calling the routes, and our document.ready callback will look like this:

    (new User())
      .sync(function(err) {
      Coursera.user = this;
      if (!Backbone.history.start({
        pushState: true
      })) {
        Coursera.router.trigger("error", 404);
      }
    });

SPAs: Regions

Backbone lets you create views and render views into arbitrary parts of your DOM, but many developers soon run into the desire for standard "regions" or "layouts". We want to specify different parts of their page, and only swap out the view in those parts across routes - like the header, footer, and main area. That's a better user experience, since there's no unnecessary refreshing of unchanging DOM.

For that, we use origami.js, a custom library that lets us create regions associated with views, and then in a route, we'll specify which region we want to replace with a particular view file, plus additional options to pass to that view. In the view, we can bind to region events like "view:merged" or "view:appended" and take appropriate actions.

In our SPAs, we always render into the regions instead, so our routes code looks more like this. It is a bit of an unwieldy syntax, but it gets the job done:

routes[triageurl + '/items/:id'] = function(id) {
    Coursera.region.open({
      "pages/home/template/page": {
        regions: {
          body: {
            "pages/triage/views/MainView": {
              id: "MainView",
              initialize: {
                openItemId: id
              }
            }
          }
        }
      }
    });
  };

We could also consider using: Marionette.js or Chaplin.

SPAs: Dirty Models

In traditional web apps, it's common practice to warn a user before leaving a page that they have unsaved data, using the window.onunload event. However, we no longer have that event in Backbone SPAs, since what looks like a window unload is actually just a region swap in JS. So, we built a mechanism into origami.js that inspects a view for a "dirty model" before swapping a view, and it throws up a modal alert if it detects that.

To utilize this, a view needs to specify a hasUnsavedModel function and return true or false from that:

var view = Backbone.View.extend({
    // ...
    hasUnsavedModel: function() {
       return !this.$saveButton.is(':disabled');
    }
});

SPAs: Internal Links

In traditional web apps, it is easy to link to a part of a page using an internal anchor, like /terms#privacy. However, in a SPA, the hash cannot be used for internal anchors, since it is used as the fallback technology for the main URL in some browsers, and the URL would actually be /#terms#privacy. We have experimented with various alternative approaches to internal links, and the current favorite approach is to use a URL like /terms/privacy, define a route that understands that URL, pass the "section" into the view, and use JS to jump to that part of the view, post-rendering. For example:

In the routes file:

  routes[home + "about/terms/:section"] = function(section) {
    Coursera.region.open({
      "pages/home/template/page": {
        regions: {
          body: {
            "pages/home/about/tosBody": {
              initialize: {section: section}
            }
          }
        }
      }
    });
  };

In the view file:

var tosBody = body.extend({
    initialize: function() {
      var that = this;
      document.title = "Terms of Service | Coursera";

      that.bind("view:merged", function(options) {
        if(options && options.section)
          util.scrollToInternalLink(that.$el, options.section);
        else
          window.scrollTo(0,0);
      });
    },
    // ...
});

In the Jade template:

h2(data-section="privacy") Privacy Policy

Widgets

In some cases, we do not necessarily want our Backbone view to take full control over the URL, like if we want to easily have arbitrary, multiple Backbone views on the same page. We take that approach in our class platform, because that will ultimately make it easier for professors who want to compose together views to their own liking (i.e. if they'd like to mix a forum thread and a wiki view on the same page, that should be easy for them.)

To create a widget, we use a declarative HTML syntax, specifying data attributes that define the widget type and additional attributes to customize that instance of the widget:

<div data-coursera-reporter-widget
    data-coursera-reporter-title=""
    data-coursera-reporter-url="">
Just one moment while we load up our reporter wizard...
</div>

Then, we create a widgets.js file that will be included on that page, and knows how to turn DOM elements into Backbone views. Typically that file would know about multiple widgets, but we show one here to save space:

define([
  "jquery",
  "underscore",
  "backbone",
  'pages/forum/app',
  'pages/forum/views/ReporterView'
],
function($, _, Backbone, Coursera, ReporterView) {
  
  $(document).ready(function() {

    $('[data-coursera-reporter-widget]').each(function() {
      var title = $(this).attr('data-coursera-reporter-title');
      var url = $(this).attr('data-coursera-reporter-url');
      new ReporterView({el: $(this)[0],itemTitle: title, itemUrl: url}).render();
    });

  });

});

Widgets: Maintaining State

We still want to maintain state within those views and support the back button, however, without changing the main URL of the page.

jQuery BBQ is an external non-Backbone specific library for maintaining history in the hash, and as it turns out, it works pretty well with Backbone. You can read my blog post on it for a detailed explanation.

We could also considering using: Backbone.Widget.

Testing Architecture

First, let it be said: testing is important. We are building a complex product for many users that will pass through many engineer's hands, and the only way we can have a reasonable level of confidence in making changes to old code is if there are tests for it. We will still encounter bugs and users will still use the product in ways that we did not expect, but we can hope to avoid some of the more obvious bugs via our tests, and we can have a mechanism in place to test regressions. Traditionally, the frontend has been the least tested part of a webapp, since it was traditionally the "dumb" part of the stack, but now that we are putting so much logic and interactivity into our frontend, it needs to be just as thoroughly tested as the backend.

There are various levels of testing that we could do on our frontends: Unit testing, integration testing, visual regression testing, and QA (manual) testing. Of those, we currently only do unit testing and QA testing, but it's useful to keep the others in mind.

Unit Testing

When we call a function with particular parameters, does it do what we expect? When we instantiate a class with given options, do its methods do what we think they will? There are many popular JS unit testing frameworks now, like Jasmine, QUnit, and Mocha.

We do a form of unit testing on our Backbone models and views, using a suite of testing technologies:

• Mocha: An open-source test runner library that gives you a way to define suites of tests with setup and teardown functions, and then run them via the command-line or browser. It also gives you a way to asynchronously signal a test completion. For example:
  

  
  describe('tests for the reporter library', function() {
    beforeEach(function() {
      // do some setup code
    }
    afterEach(function() {
     // do some cleanup code
    }
    it('renders the reporter template properly', function() {
      // test stuff
    }
    it('responds to the ajax request correctly', function(done) {
      // in some callback, call:
      done();
    }
  });

• Chai: An open-source test assertion library that provides convenient functions for checking the state of a variable, using a surprisingly readable syntax. For example:
  

  
    chai.expect(2+2).to.be.equal(4);
    chai.expect(2+2).to.be.greaterThan(3);
  

• JSDom: An open-source library that creates a fake DOM, including fake events. This enables us to test our views without actually opening a browser, which means that we can run quite a few tests in a small amount of time. For example, we can check that clicking changes some DOM:
  

  
       var view = new ReporterView().render();
       view.$el.find('input[value=quiz-wronggrade]').click();
  
        var $tips = view.$el.find('[data-problem=quiz-wronggrade]');
        chai.expect($tips.is(':visible'))
          .to.be.equal(true);
        chai.expect($tips.find('h5').eq(0).text())
          .to.be.equal('Tips');
  

• SinonJS: An open-source library for creating stubs, spies, and mocks. We use it the most often for mocking out our server calls with sample data that we store with the tests, like so:
  

  
      var forumThreadsJSON  = JSON.parse(fs.readFileSync(path.join(__filename, '../../data/forum.threads.firstposted.json')));
     server    = sinon.fakeServer.create();
     server.respondWith("GET", getPath('/api/forum/forums/0/threads?sort=firstposted&page=1'), 
          [200, {"Content-Type":"application/json"}, JSON.stringify(forumThreadsJSON)]);
     // We call this after we expect the AJAX request to have started
     server.respond();
  

  We can also use it for stubbing out functionality that does not work in JSDom, like functions involving window properties, or functionality that comes from 3rd party APIs:

  
        var util = browser.require('js/lib/util');
        sinon.stub(util, 'changeUrlParam', function(url, name, value) { return url + value;});
        var BadgevilleUtil = browser.require('js/lib/badgeville');
        sinon.stub(BadgevilleUtil, 'isEnabled', function() { return true;});
  

  Or we can use it to spy on methods, if we just want to check how often they're called. Sometimes this means making an anonymous function into a view method, for easier spy-ability:

  
      sinon.spy(view, 'redirectToThread');
      // do some stuff to call function to be called
      chai.expect(view.redirectToThread.calledOnce)
          .to.be.equal(true);
       view.redirectToThread.restore();
  

Besides those testing-specific libraries, we also use NodeJS to execute the tests, along with various Node modules:

• require: Similar to how we use this in our Backbone models and views to declare dependencies, we use require in the tests to bring in whatever libraries we're testing.
• path: A library that helps construct paths on the file system.
• fs: A library that helps us read our test files.

Let's see what all of that looks like together in one test suite. These are a subset of the tests for our various about pages. The first test is a very simple one, of a basically interaction-less, AJAX-less posts. The second test is for a page that does an AJAX call:

describe('about pages', function() {
  var chai = require('chai');
  var path = require('path');
  var env  = require(path.join(testDir, 'lib', 'environment'));
  var fs   = require('fs');

  var Coursera;
  var browser;
  var sinon;
  var server;
  var _;

  beforeEach(function() {
    browser = env.browser(staticDir);
    Coursera  = browser.require('pages/home/app');
    sinon = browser.require('js/lib/sinon');
    _ = browser.require('underscore');
  });

  describe('aboutBody', function() {

    it('about page content', function() {
      var aboutBody = browser.require('pages/home/about/aboutBody');
      var body      = new aboutBody();
      var view      = body.render();

      chai.expect(document.title).to.be.equal('About Us | Coursera');
      chai.expect(view.$el.find('p').size()).to.be.equal(6);
      chai.expect(view.$el.find('h2').size()).to.be.equal(3);
    });
  });


  describe('jobsBody and jobBody', function(){

    var jobs     = fs.readFileSync(path.join(__filename, '../../data/about/jobs.json'), 'utf-8');
    var jobsJSON = JSON.parse(jobs);

    beforeEach(function() {
      server = sinon.fakeServer.create();
      server.respondWith("GET", Coursera.config.url.api + "common/jobvite.xml", 
        [200, {"Content-Type":"application/json"}, jobs]);
    });

    it('job page content', function(done) {
      var jobBody = browser.require('pages/home/about/jobBody');
      var view      = new jobBody({jobId: jobsJSON[0].id});

      var renderJob = sinon.stub(view, 'renderJob', function() {
        renderJob.restore();
        view.renderJob.apply(view, arguments);
        chai.expect(view.$('.coursera-about-body h2').text())
          .to.be.equal(jobsJSON[0].title);
        done();
      });

      view.render();
      chai.expect(document.title).to.be.equal('Jobs | Coursera');
      server.respond();
    });

  });

Integration testing

Can a user go through the entire flow of sign up, enroll, watch a lecture, and take a quiz? This type of testing can be done via Selenium WebDriver, which opens up a remote controlled browser on a virtual machine, executes commands, and checks expected DOM state. The same test can be run on multiple browsers, to make sure no regressions are introduced cross-browser. They can be slow to run, since they do start up an entire browser, so it is common to use cloud services like SauceLabs to distribute tests across many servers and run them in parallel on multiple browsers.

There are client libraries for the Selenium WebDriver written in several languages, the most supported being Java and Python. For example, here is a test for our login flow that enters the user credentials and checks the expected DOM:

from selenium.webdriver.common.by import By
import BaseSitePage

class SigninPage(BaseSitePage.BaseSitePage):
    def __init__(self, driver, waiter):
        super(SigninPage, self).__init__(driver, waiter)
        self._verify_page()

    def valid_login(self, email, password):
        self.enter_text('#signin-email', email)
        self.enter_text('#signin-password', password)
        self.click('.coursera-signin-button')
        self.wait_for(lambda: \
                self.is_title_equal('Your Courses | Coursera') or \
                self.is_title_equal('Coursera'))

We do not currently run our Selenium tests, as they are slow and fragile, and we have not had the engineering resources to put time into making them more stable and easier to develop locally. We may out source the writing and maintenance of these tests to our QA team one day, or hire a Testing engineer that will improve them, or both.

Visual regression testing

If we took a screenshot of every part of the site before and after a change, do they line up? If there's a difference, is it on purpose, or should we be concerned? This would be most useful to check affects of CSS changes, which can range from subtle to fatal.

There are few apps doing this sort of testing, but there's a growing recognition of its utility and thus, we're seeing more libraries come out of the woodwork for it. Here's an example using Needle with Selenium:

from needle.cases import NeedleTestCase

class BBCNewsTest(NeedleTestCase):
    def test_masthead(self):
        self.driver.get('http://www.bbc.co.uk/news/')
        self.assertScreenshot('#blq-mast', 'bbc-masthead')

There's also Perceptual Diffs, PhantomCSS, CasperJS, and SlimerJS. For a more manual approach, there's the Firefox screenshot command with Kaleidoscope. Finally, there's dpxdt (pronounced depicted).

We do not do visual regression testing at this time, due to lack of resources, but I do think it would be a good addition in our testing toolbelt, and would catch issues that no other testing layers would find.

QA (manual) testing

If we ask a QA team to try a series of steps in multiple browsers, will they see what we expect? This testing is the slowest and least automate-able, but it can be great for finding subtle usability bugs, accessibility issues, and cross-browser weirdness.

Typically, when we have a new feature and we've completed the frontend per whatever we've imagined, we'll create a worksheet in our QA testing spreadsheet that gives an overall description of the feature, a staging server to test it on, and then a series of pages or sequences of interactions to try. We'll also specify what browsers to test in (or "our usual" - Chrome, FF, IE, Safari, iPad), and anything in particular to look out for. QA takes about a night to complete most feature tests, and depending on the feedback, we can put a feature through multiple QA rounds.

Additional Reading

The following slides and talks may be useful as a supplement to this material (and some of it served as a basis for it):

• Blog post: Frontend architectures or Video
• Slides: Backbone, 3 Ways or Video
• Blog post: Rewriting our forums with Backbone
• Slides: When Bootstrap Attacks
• Just-in-time Renderer