Saturday, April 25, 2009

What is an API?

A game designer asked me the other day to explain the meaning of "API" - and I thought, wow, that is a good question, considering our massive (over?)use of the term on the web. Of course, the obvious meaning is "Application Programming Interface", but that's pretty damn non-helpful, particularly for a non-programmer. So here's what I told him:

There's the traditional meaning of "API" -

Some programmer writes a library with lots of functionality in it, and exposes the functionality as different functions, so that other programmers can just call the functions.

For instance, I could write a function that goes through all the complex steps of making out:

function makeOut(passionLevel, partsOfBody) {
 for (each partOfBody in partsOfBody) {
   partOfBody.kiss(passionLevel);
   lookIntoEyes();
   sighDeeply();
 }
 moanDaintily();
 complainAboutPeopleWatching();
}

So if you want to have your code make out, then you could just call:

makeOut(10, ["neck", "ear", "mouth"]);

You don't have to worry about the "implementation details" - all that code I wrote - you just know that it works.

The "library" is the actual code, the "API" is what programmers need to know in order to use it.

So my API documentation would provide the minimal info needed to use the library, and it could look like this:

makeOut(passionLevel: Number, partsOfBody:Array)

For an example of actual API docs, Java has the online documentation for all of its common libraries here: http://java.sun.com/j2se/1.5.0/docs/api/.

And then there are "Web APIs" -

A Web API hosts all of its code on one server, and documents how other servers can call that code.

There are JavaScript Web APIs, like the Google Maps API, where the Javascript is publicly viewable but quite "obfuscated" (compressed, hard to read).

This is our actual code: http://maps.google.com/intl/en_us/mapfiles/148e/maps2.api/main.js

This is our API documentation: http://code.google.com/apis/maps/documentation/reference.html

Developers only need to know what the API looks like, they don't have to figure out what all that crazy code means.

Then there are HTTP APIs, which means that all of the code is hidden on the server, and the code is executed when a programmer hits a particular URL. Often times, these APIs will create new information on the server for a user. All of our consumer products with user data have HTTP APIs - like Google Spreadsheets, Google Calendar, Blogger, etc. A programmer can hit a particular URL with an authentication header, and either get or create more user data.

One of the most popular HTTP APIs is the Flickr API, which can be used to retrieve information, upload photos, create albums, etc. Their API is documented here: http://www.flickr.com/services/api/

You can use the API without authenticating if you just want to do a simple search:

http://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=4d12a2ccfa7400584c162bfc104bf682&tags=fox

Notice the query parameters after the "?" like method, api_key, and tags. These are like the parameters to a function (like passionLevel and partsOfBody from above). You could imagine that flickr has code on their server that looks like:

function searchPhotos(api_key, tags) {
 ....
}

But once again, a developer never has to see the Flickr server code, they can just use the API and get the results they want.

Above all, an API is a promise.

It's a promise that no matter how the author changes the code (the implementation), the interface will remain the same. You can always call that makeOut function, and expect it to make out, even if it stops complaining about people watching.

Monday, April 20, 2009

Web09: Design, Code, Community.. and Maps!

Earlier in the year, I was invited to speak at Webstock, an annual conference in Wellington, New Zealand. It was a fantastic conference filled with inspirational speakers from around the world who discussed the future of the web, ways in which it sucks now, what we could do to change it.

This week, I had the opportunity to speak at Web09, a similarly named conference in Auckland, New Zealand - just an hour flight north. I was a bit surprised that New Zealanders felt the need to hold 2 web conferences in a year, and wondered if this conference would be different from Webstock. Well, it was completely different - in a good way. Web09 was a cornocupia of real-world advice from both local developers - like Karl Von Randow's talk about gaming the iPhone App store, and global evangelists - like Ryan Stewart's talk about new Flex features. Some of my favorite talks were:

  • Dan Rubin gave us a variety of ideas for creating better designs, including adding noise to a background, using your own photographs as the focal piece of a website, and scanning in every day objects to get web textures. In trying to define good design, he referenced Don Norman as the cognitive scientist who originally articulated the problem of doors that are so un-intuitively designed that they require signs - a problem that we have in our new Google building. Norman authored The Design of Everyday Things, which I've now put on my reading list.
  • Paul Burnett, Adobe Flash Evangelist, showed off various new features in Flash CS4 and its much better integration with AIR (now better than Flex Builder!). I was most impressed by the fact that Flash now supports adding bones (inverse kinematics) to shapes and graphics, and being able to set weights and properties and tweak the RTS graphs for every animation. It takes me way back (2 years!) to futzing around with character animation in Maya and Max, and makes me want to get CS4 and try animating a few simple characters - zombie, whale, maybe even a streetwalker.
  • David Karp, creator of Tumblr, described 5 areas of community building: engagement, use, negativity, change, feedback. Karp suggested that the reason that people participate in communities is for the "promise" of something happening - like the people who upload to Youtube thinking they have a small hope that one day their video will shoot to fame and they'll be a Youtube rockstar. Karp also described how Tumblr went from 1 support email a day to hundreds, and how they dealt with it. They categorized every support email as both "type of user" plus the actual problem - with user type ranging from paying customers and new customers to evangelist users. Visualizing the requests that way enabled them to decide what features were important to growth.

I myself gave a talk called "Avoiding Red Dot Fever: Tips for Improving Usability of Maps Mashups". It was a series of tips for ways web developers could improve the maps on their sites, based on my experiences seeing many badly-designed maps over the past 2 years. The talk went over really well (so well I was asked to give it twice), and gave me an excuse to meet the web developers of many New Zealand mapping sites - Zoodle, Yellow Pages NZ, vodafoneNZ, EventFinder, etc. The slides are embedded below, and there should be a video up in a few weeks.

Overall, it was a great conference, and I hope to see both Webstock and Web09 happen next year.

Thursday, April 9, 2009

Google Image Search + Color Restrict: Just look what you can find!

This week, Google officially announced support for filtering by colors in Google Image search results.I don't actually know if this is actually a useful tool yet, but even if it's not - it's freaking fun. Here are some cool results from myself and the twitterverse.

You can find beautiful landscapes: sky, red

You can find your school logos: usc, red
(via @oGLOWo)

You can find a world covered in fail stickers: fail, red
(via @niick)

You can find a phrase that's lived up to its colors: state, blue
(via @ManoMarks)

And one more of those: journalism, yellow
(via @KevinMarks)

You can find two colors at once: red, black

And finally, you can find creatures you never thought existed: dolphins, pink
(via @niick)

Wednesday, April 1, 2009

API Design: Beautiful Interfaces

This is part of a series on Web API design. The intro post is here.

When some developers think about the Google Maps API, they may envision a sea of GMarkers, GPolygons, GScaleControls, GOverviewMapControls, and the like. But really, it boils down to 2 things: GOverlay and GControl.

These two classes are interfaces in the API, and every class that is a visual item on the map extends from one of those two. Conceptually, a GOverlay is something that moves when the map moves and is somehow related to latitude/longitude coordinates, and a GControl is something that always stays in the same place, regardless of map movement. The screenshot below points to various examples in a standard maps mashup:

To add one of these to the map, a developer just calls addOverlay or addControl:

map.addControl(new GScaleControl());
map.addOverlay(new GMarker(new LatLng(37, -122));

To remove them, they call removeOverlay or removeControl.

There are some mapping APIs out there with calls like addMarker, addPolyline. This is is a bad idea for several reasons. It means you need an additional method each time you add a new visual class (bloating your API), and it likely also makes it hard for developers to extend the API. With generic methods like addOverlay or addControl, you can add a custom extension the *same way* you add a native class - so newbie developers don't have to learn a new syntax to add one to their map.

Along with using generic methods for adding stuff to the map, we can also take advantage of the interface to make our events more generic. A developer can listen to "addoverlay" or "removeoverlay" to find out when any overlay has been added to the map, and when they listen to a "click" event on the map, their callback gets passed in whatever GOverlay was clicked on. This means that whenever we add a new type of overlay to our API, the developer can continue using the same events they were before. We do often add specific events for each overlay later, but it's nice that the generic ones come for free.

And something for the advanced developers...

These interfaces make our API infinitely extensible in a standard way, and they're the reason I used the API for my personal and research projects even before working for Google. If I have a fundamental aesthetic issue with our GLargeMapControl3D, then I can simply create my own control (ExtLargeMapControl). If I want to visualize 5,000 markers, and GMarker is simply too full-featured and heavyweight for the job, then I can create my own lightweight marker (MarkerLight). If I want to visualize my points as <div>-based bar graphs instead of <img>-based markers, then I can create my own overlay (Bar).

To extend the interfaces requires defining only a few minimal methods:

GOverlay: initialize, redraw, remove

The initialize method, called when addOverlay is invoked, should create DOM objects for the overlay and append them to the desired map pane (there are several). The redraw method is called whenever the map moves, and it should re-position the overlay depending on the new latitude/longitude -> pixel mapping. The remove method, called when removeOverlay is invoked, should clean up any DOM objects created by the overlay. More info and an example is in the docs.

GControl: initialize, getDefaultPosition

The initialize method, called when addControl is invoked, should create and return the div for the control. The getDefaultPosition method should returns the recommended position for the control. There is no remove method, since the initialize method returned a <div> that the map can remove whenever removeControl is invoked. More info and an example is in the docs.

Using just these methods and some JavaScript/CSS skills, a developer can create anything their hacker heart desires. Custom controls and overlays account for 15/20 of the libraries in our official utility library, another 15 of the libraries listed in Mike William's list of 3rd party extensions, and based on my experience, many more are likely floating around in developer's code on the web.

Don't forget the actual API engineers...

Along the same lines, these interfaces also make API development easier for internal developers. If they want to add a visual class to the API, they just need to figure out whether it's a control or an overlay, define the methods, document that the new class exists, and external developers will use their existing familiarity with addControl or addOverlay to add it to their map.

It can't all be fluffy rainbows and unicorns...

We've certainly made a few mistakes along the way with the interfaces - but this post is getting long, so I'll save those for later.

To sum it all up...

The interfaces are easy for newbie developers to understand, easy for advanced developers to extend, and easy for internal engineers to implement. Beautiful.

Monday, March 30, 2009

Web API Design: The Good, The Bad, and the Ugly

Two recent events have prompted me to spend a lot more time thinking about API design: 1) A series of "Good, Bad, and Ugly" talks at the Google Sydney office which examine internal architectures and coding decisions, 2) An internal reading group starting with "Beautiful Architecture", a book of essays. Though the design of internal software (largely the focus of the talks and book) is important and should be done well, it's my opinion that the design of external-facing Web APIs is even more important, and yet, a much less discussed topic.

Why? Web APIs have some unique challenges:

Flexibility: Web APIs are intended to by whatever random developer finds them useful, not by a restricted set of internal developers. And, I can tell you from experience, random developers do things that you never even realize was possible with your API - and then they beg for feature requests to make it easier for them - and you'll want to fulfill their requests because you'll think "hot damn, that was cool!" So, the easier it is to implement new and different features, the better.

Backwards-Compatibility: Web APIs have to stay around for a long, long time. Even if you threaten that you'll change your API every 2 years and you warn users 1 year ahead of time, you'll still have some developers use the old version of the API and write angry posts in your forum about their site breaking. Or, developers that have written code for client sites, moved on to bigger and better things, and left it with webmasters that have no idea how to upgrade it. Or, developers using wrapper libraries that abstract on top of the older version of the API, and have no notion of the underlying API. So, it's safe to assume that you can never really deprecate an API without breaking old sites, and you should design the API with that in mind (future-proof it).

.. and other challenges, like needing to be equally usable by both newbie and advanced developers, and needing to be a reasonable number of kilobytes (for client-side APIs).

In the land of Web APIs, the Google Maps API makes for an interesting case study. It is one of the most used Web APIs in the worlds (150,000 sites), the first interactive web mapping API, likely the first hosted JavaScript API, and it has grown from a simple API with just markers and zooming to a complex API with 90+ classes. It's also gone from version 1 to version 2, but the changes were primarily in terms of latency; the actual interface has remained largely the same. All these facts make it interesting to look at the Maps API from an architecture and design point of view, and point out the places where it went right - and where it went wrong. Of course, as an advocate of the Maps API and someone who uses it daily, I should point out that I still think we have the best Maps API in the world. But even gods can have their flaws (I hear Zeus had a weakness for mortal women).

So, in an upcoming series of blog posts, I'll single out design decisions in our API that I think were good, bad, or ugly. To start it off on a good note, I'll start with something beautiful... stay tuned!

APUGS Round-Up: Bzoo, AIR, & Eclipse

Tonight, despite half of Sydney CBD being in a blackout (and zombies attacking the city), I attended the monthly APUGS (Adobe Platform Users Group Sydney) meeting. 3 developers gave talks, and I learnt a great deal of random information. I realize that all of my Flex experience currently comes from supporting the Maps API for Flash - so, if a developer hasn't asked it, I've never learnt it - and apparently, developers haven't asked me all the questions in the world yet. Here's a roundup of the topics covered:

  1. Developer Bachir El Khoury spoke about his AS3 library for creating a client-side database with full CRUD operations, called Bzoo. It can create the database from JSON, YAML, or ArrayCollections, and optionally, it can store the database persistently using Flash Shared Objects. Shared Objects are basically super cookies for Flash that are persistent across all browser sessions on a desktop, and can take up to 100 KB per domain. Cookies can be useful in Flash apps for remembering game configurations, backing up data before a save, or user sign-in state.
  2. Developer Andrew Muller showed off Bowernest, an AIR app he's creating for a client (BookTagger). It's a desktop app with a slick interface for managing books and other media. Andrew used the Mate (Mah-tay) RIA framework for creating the app, and the award-winning skin from ScaleNine's contest. Cleverly, the app can import files from Shelfari and LibraryThing, the two big competitors to BookTagger, so its easier for users to switch over. It then stores all the data locally in an AIR SQLite DB, and offers the user the option to export back out to the Booktagger file format. As a heavy user of web-apps, I find it a bit shocking that the data isn't constantly backed up to a BookTagger user-account, but hey, apparently some users are into the whole desktop-only app thing. When I asked Andrew about other apps out there that were desktop-only, he showed us Balsamiq Mockups, an awesome app for mocking up web prototypes in a sketchy way.
  3. APUGS organizer Chris Velevitch showed some tips for working in Eclipse. I only work in FlexBuilder, but it was useful to see the similarities and differences to Eclipse development. Eclipse has a cool feature called local history, which lets you see every version of the files in your project - for up to "9,999 days"! Also, in both Eclipse and FlexBuilder, you can right-click on a file tab and select "New Editor" to get the same file opened in two tabs. In FlexBuilder, you can then put one tab in Design mode and the other in Source mode, and drag one so it's below the other. Highly useful.

Friday, March 27, 2009

BarCamp Canberra 2: Mapping the Fires

I just spoke a few minutes ago at BarCamp Canberra 2 about our experience making a map of the Victoria BushFires. Here's a quick run-through of the presentation, with links:

  • We woke up, saw there were crazy fires going on, and looked at how we could help. The websites with the information were DSE (Public Lands) and CFA (Private Lands). The problem with the DSE site was the lack of an interactive map and the lack of an RSS feed. The lack of RSS feed meant that people were constantly reloading the page to see what had changed, and their server was crumbling under the load. The problem with the CFA site was the lack of a map -- and given all the number of small towns in Australia, it's very hard to visualize where the fires are happening in respect to where you or your loved. The problems with the RSS feed was that it lacked latitude/longitude coordinates, and the CFA website was struggling and producing corrupt feeds some of the time.
  • First we tried using Yahoo! Pipes to turn the RSS into a GeoRSS feed, but it didn't quite work correctly, and we were worried about relying on the service during an emergency.
  • So then I set up a database on my server, ran a script geocoding any new locations every 2 hours, output the database as an ActionScript object, copied and pasted it into my Flex app, and re-uploaded the SWF. It was ghetto and painful, but it worked.. provided I woke up every 2 hours. Sometimes, we had to manually find locations - like "DSE - 6KM ESE OF WARBURTON" - by using the distance measurer mapplet in Google Maps and eyeballing it. It sure would have been easier if CFA had done the geocoding for us!
  • So then we created the map. We sent the RSS feed through Google App Engine, caching it every 5 minutes to avoid hitting the CFA web server (and to ignore corrupt feeds), parsed that feed in the Flash map, and displayed the fires as markers. We later added a legend, search box, and colored/sized markers, but we basically had our map up 4 hours after we started coding. Not bad!
  • One important thing we did was make a gadget version of the map. This meant news agencies, blogs (like Mashable, and random concerned folks could embed the map in their sites. A lot of people discovered the map through the gadget, and the news agencies benefited since they didn't have to create their own visual to complement their articles.
  • The map got around 60 QPS at its peak - and App Engine handled it with no problems. It probably helped that we used our inside influence to raise all of our quotas to the absolute max (14 petabytes of bandwidth!). :)
  • After we had the map up, we wanted to make the geocoding less ghetto. We changed from doing the geocoding on my server to doing the geocoding on App Engine, and outputting the feed as a GeoRSS feed instead. The great thing about GeoRSS is that it's a pseudo-standard, and there are multiple websites that can parse and display them (including Google Maps). The cool thing about outputting the data in this portable format was that one concerned guy was able to import the GeoRSS into a Google My Map, plot markers where his family members were located, and figure out if they were in trouble.
  • At this point, we were doing good things with the CFA data, but we still had no data for the public lands. DSE was unwilling to give us structured XML data, as they had various concerns about copyrights, attribution, and data distortion. We still wanted to represent public lands data though, and that's when we discovered that NASA takes satellite imagery of South Australia every 12 hours and makes them available in various standard formats.
  • Using MapTiler, an open-source GUI for the command line GDAL utilities, we converted the NASA GeoTIFF into map tiles. Those map tiles can be used in most APIs - Google, Microsoft, OpenLayers, etc. We, of course, overlaid them on our Google Flash Map, and let people toggle the imagery on. NASA does an automatic analysis of the imagery to look for fires, so you could actually see red outlines around the center of big lines of smoke. Since we're still serving the imagery from 25 days ago, you can still see that on the website.
  • The aftermath of all of this is that we learned lessons about the issues with getting emergency data in a format that can be re-used, cached, syndicated, and visualized in useful ways. We wrote a white paper of sorts with recommendations for Emergency Data portability and we're consulting with various agencies to help them get their data up to snuff.