Web API Design: The Good, The Bad, and the Ugly

Two recent events have prompted me to spend a lot more time thinking about API design: 1) A series of "Good, Bad, and Ugly" talks at the Google Sydney office which examine internal architectures and coding decisions, 2) An internal reading group starting with "Beautiful Architecture", a book of essays. Though the design of internal software (largely the focus of the talks and book) is important and should be done well, it's my opinion that the design of external-facing Web APIs is even more important, and yet, a much less discussed topic.

Why? Web APIs have some unique challenges:

Flexibility: Web APIs are intended to by whatever random developer finds them useful, not by a restricted set of internal developers. And, I can tell you from experience, random developers do things that you never even realize was possible with your API - and then they beg for feature requests to make it easier for them - and you'll want to fulfill their requests because you'll think "hot damn, that was cool!" So, the easier it is to implement new and different features, the better.

Backwards-Compatibility: Web APIs have to stay around for a long, long time. Even if you threaten that you'll change your API every 2 years and you warn users 1 year ahead of time, you'll still have some developers use the old version of the API and write angry posts in your forum about their site breaking. Or, developers that have written code for client sites, moved on to bigger and better things, and left it with webmasters that have no idea how to upgrade it. Or, developers using wrapper libraries that abstract on top of the older version of the API, and have no notion of the underlying API. So, it's safe to assume that you can never really deprecate an API without breaking old sites, and you should design the API with that in mind (future-proof it).

.. and other challenges, like needing to be equally usable by both newbie and advanced developers, and needing to be a reasonable number of kilobytes (for client-side APIs).

In the land of Web APIs, the Google Maps API makes for an interesting case study. It is one of the most used Web APIs in the worlds (150,000 sites), the first interactive web mapping API, likely the first hosted JavaScript API, and it has grown from a simple API with just markers and zooming to a complex API with 90+ classes. It's also gone from version 1 to version 2, but the changes were primarily in terms of latency; the actual interface has remained largely the same. All these facts make it interesting to look at the Maps API from an architecture and design point of view, and point out the places where it went right - and where it went wrong. Of course, as an advocate of the Maps API and someone who uses it daily, I should point out that I still think we have the best Maps API in the world. But even gods can have their flaws (I hear Zeus had a weakness for mortal women).

So, in an upcoming series of blog posts, I'll single out design decisions in our API that I think were good, bad, or ugly. To start it off on a good note, I'll start with something beautiful... stay tuned!

APUGS Round-Up: Bzoo, AIR, & Eclipse

Tonight, despite half of Sydney CBD being in a blackout (and zombies attacking the city), I attended the monthly APUGS (Adobe Platform Users Group Sydney) meeting. 3 developers gave talks, and I learnt a great deal of random information. I realize that all of my Flex experience currently comes from supporting the Maps API for Flash - so, if a developer hasn't asked it, I've never learnt it - and apparently, developers haven't asked me all the questions in the world yet. Here's a roundup of the topics covered:

1. Developer Bachir El Khoury spoke about his AS3 library for creating a client-side database with full CRUD operations, called Bzoo. It can create the database from JSON, YAML, or ArrayCollections, and optionally, it can store the database persistently using Flash Shared Objects. Shared Objects are basically super cookies for Flash that are persistent across all browser sessions on a desktop, and can take up to 100 KB per domain. Cookies can be useful in Flash apps for remembering game configurations, backing up data before a save, or user sign-in state.
2. Developer Andrew Muller showed off Bowernest, an AIR app he's creating for a client (BookTagger). It's a desktop app with a slick interface for managing books and other media. Andrew used the Mate (Mah-tay) RIA framework for creating the app, and the award-winning skin from ScaleNine's contest. Cleverly, the app can import files from Shelfari and LibraryThing, the two big competitors to BookTagger, so its easier for users to switch over. It then stores all the data locally in an AIR SQLite DB, and offers the user the option to export back out to the Booktagger file format. As a heavy user of web-apps, I find it a bit shocking that the data isn't constantly backed up to a BookTagger user-account, but hey, apparently some users are into the whole desktop-only app thing. When I asked Andrew about other apps out there that were desktop-only, he showed us Balsamiq Mockups, an awesome app for mocking up web prototypes in a sketchy way.
3. APUGS organizer Chris Velevitch showed some tips for working in Eclipse. I only work in FlexBuilder, but it was useful to see the similarities and differences to Eclipse development. Eclipse has a cool feature called local history, which lets you see every version of the files in your project - for up to "9,999 days"! Also, in both Eclipse and FlexBuilder, you can right-click on a file tab and select "New Editor" to get the same file opened in two tabs. In FlexBuilder, you can then put one tab in Design mode and the other in Source mode, and drag one so it's below the other. Highly useful.

BarCamp Canberra 2: Mapping the Fires

I just spoke a few minutes ago at BarCamp Canberra 2 about our experience making a map of the Victoria BushFires. Here's a quick run-through of the presentation, with links:

• We woke up, saw there were crazy fires going on, and looked at how we could help. The websites with the information were DSE (Public Lands) and CFA (Private Lands). The problem with the DSE site was the lack of an interactive map and the lack of an RSS feed. The lack of RSS feed meant that people were constantly reloading the page to see what had changed, and their server was crumbling under the load. The problem with the CFA site was the lack of a map -- and given all the number of small towns in Australia, it's very hard to visualize where the fires are happening in respect to where you or your loved. The problems with the RSS feed was that it lacked latitude/longitude coordinates, and the CFA website was struggling and producing corrupt feeds some of the time.
• First we tried using Yahoo! Pipes to turn the RSS into a GeoRSS feed, but it didn't quite work correctly, and we were worried about relying on the service during an emergency.
• So then I set up a database on my server, ran a script geocoding any new locations every 2 hours, output the database as an ActionScript object, copied and pasted it into my Flex app, and re-uploaded the SWF. It was ghetto and painful, but it worked.. provided I woke up every 2 hours. Sometimes, we had to manually find locations - like "DSE - 6KM ESE OF WARBURTON" - by using the distance measurer mapplet in Google Maps and eyeballing it. It sure would have been easier if CFA had done the geocoding for us!
• So then we created the map. We sent the RSS feed through Google App Engine, caching it every 5 minutes to avoid hitting the CFA web server (and to ignore corrupt feeds), parsed that feed in the Flash map, and displayed the fires as markers. We later added a legend, search box, and colored/sized markers, but we basically had our map up 4 hours after we started coding. Not bad!
• One important thing we did was make a gadget version of the map. This meant news agencies, blogs (like Mashable, and random concerned folks could embed the map in their sites. A lot of people discovered the map through the gadget, and the news agencies benefited since they didn't have to create their own visual to complement their articles.
• The map got around 60 QPS at its peak - and App Engine handled it with no problems. It probably helped that we used our inside influence to raise all of our quotas to the absolute max (14 petabytes of bandwidth!). :)
• After we had the map up, we wanted to make the geocoding less ghetto. We changed from doing the geocoding on my server to doing the geocoding on App Engine, and outputting the feed as a GeoRSS feed instead. The great thing about GeoRSS is that it's a pseudo-standard, and there are multiple websites that can parse and display them (including Google Maps). The cool thing about outputting the data in this portable format was that one concerned guy was able to import the GeoRSS into a Google My Map, plot markers where his family members were located, and figure out if they were in trouble.
• At this point, we were doing good things with the CFA data, but we still had no data for the public lands. DSE was unwilling to give us structured XML data, as they had various concerns about copyrights, attribution, and data distortion. We still wanted to represent public lands data though, and that's when we discovered that NASA takes satellite imagery of South Australia every 12 hours and makes them available in various standard formats.
• Using MapTiler, an open-source GUI for the command line GDAL utilities, we converted the NASA GeoTIFF into map tiles. Those map tiles can be used in most APIs - Google, Microsoft, OpenLayers, etc. We, of course, overlaid them on our Google Flash Map, and let people toggle the imagery on. NASA does an automatic analysis of the imagery to look for fires, so you could actually see red outlines around the center of big lines of smoke. Since we're still serving the imagery from 25 days ago, you can still see that on the website.
• The aftermath of all of this is that we learned lessons about the issues with getting emergency data in a format that can be re-used, cached, syndicated, and visualized in useful ways. We wrote a white paper of sorts with recommendations for Emergency Data portability and we're consulting with various agencies to help them get their data up to snuff.

Ada Lovelace Tribute: A Q&A with My Mum

A few months ago, I pledged to write a blog post for Ada Lovelace day. The hope is that atleast 1,000 bloggers will write about a woman in technology on today, March 24th, and that this will draw (good) attention to woman in technology. I don't necessarily know if we'll succeed in the objective, but hey, it's different and could work, so why not?

Since I don't actually know very many women in technology (I'm too distracted by all the tech boys?), I've done a Q&A with my mum, Rosemary Kennett. Mum studied Physics at Caltech, met and married my dad there (a professor! scandal!), had 3 kids, worked for NASA JPL, did physics research at SU and Cornell, taught Astronomy for a few years, and then moved on to her current job as a scientific programmer in Spectral Sciences. Here goes:

Why did you choose to study Physics?

I actually became fascinated in high school where I was taught by a teacher who had just returned from a sabbatical at Oxford and was intrigued by the new physics e.g. nuclear physics. I remember that I used to borrow her collection of popular scientific paperbooks.

What was the first line of code that you wrote?

I remember writing code to model an oscillating violin string when I took an early math course involving programming as an undergraduate (we did not have CS courses back then). Most of the early coding was in Algol 60 (which most people agree was a pretty good language but somehow it died out)

What language have you programmed the most in?

Fortran, then C/C++, then Java

(Hmm. Is Fortran kind of like 4chan? - PF)

What app are you proudest of creating?

For the general public - there's a simple Java applet that often gets referred to: Epicycle and Deferent Demo.

You taught astronomy and ran the Syracuse Astronomical Club for a few years. What's the most fascinating aspect of astronomy for you?

How much we are learning about the distant past (likewise through high energy physics experiments, which now are trying to mimic conditions soon after the start of the universe!). Also the number, variation and beauty of objects in space, especially as observed using the Great Observatories (Hubble, Chandra for X-Rays, Spitzer for IR etc)

I always thought it was pretty cool to have a mum that worked at JPL. What were you doing there (besides rocket science, of course)?

Actually the technical work of which I'm most proud was that done at JPL. One project was to check the performance of a new design of a radar instrument called a scatterometer which measures oceanic winds (strength and direction) by bouncing radar off the waves - the Bragg effect means that the radar return depends on the roughness of the waves, which is in turn influenced by the wind speed. You do this from different directions and you can tell the direction of the wind also. Anyway that style of scatterometer is now the standard, although launches of these instruments are few and far between

(That's right, my mum's invention is orbiting space above your heads right now. Boo-yah! - PF)

You have 3 kids, and two of them ended up in Computer Science. Do you credit/blame nature or nurture?

Genes!

(Hmm. I'm pretty sure it was the fact that we had a T1 line before any other kids had the internet, and that we moved to cold, miserable upstate NY and I was forced to hibernate inside our well-supplied computer room. - PF

Bonus Factoid: My mum knows all the cool kids in CS: Steven Wolfram, because he was a student working with my dad on the Mathematica forerunner, Richard Feynman, because he was a Physics professor at Caltech, and Larry Page, because he attended a Hertz event and asked her a question about her research... I'm jealous!

Adelaide: More than OK, Actually

My mum's visiting Australia for the first time in her life, spending the majority of her three weeks wandering around Sydney (it's a big place!). She traveled with me last weekend to only one city outside of Sydney: Adelaide, in South Australia. When I told Aussies where we were going, the question was always "why?" - Adelaide doesn't exactly have the greatest reputation. The Melbourne mayor went so far as to effectively recommend that Adelaide be shut down, due to its uselessness. So I went to Adelaide prepared for a city of churches and dull-dom, but I actually encountered amazing natural sights and a vast array of entertainment options. Check out my description of the highlights below, or if you're a visual sort, just go through my Adelaide Flickr collection.

Faulty Towers Interactive Dinner

My mum is British, so when I saw the advert for a dinner in the style of this classic British comedy, I knew it'd be perfect for our first night in Adelaide. Our hosts were Basil, Sybil, and Manuel, and the dinner was a series of well orchestrated haphazards and improvised arguments with us, the guests. And as much as they tried to make the dinner service disruptive, the food was still delicious - my first stone grilled steak, but definitely not my last.

Glenelg Town Hall Museum

Yes, yes, a town's museum. But, this museum contained more than the usual "bla bla discovered X and settled Y and then war happened and stuff". It had *2* special exhibits dedicated to retro arcades - machines that you'd either wind or put a coin in, and you'd see some puppet animation - like a monkey's eyes moving, or 2 sharks boxing. Though we've moved way beyond such simple mechanics these days, it's still very satisfying to think "hmm, what's going to happen?", do some winding, and then, "oo! the arms moved!". One of the coin-operated arcades was a skeleton fortune teller - I asked it if I should ever get a PhD, and it responded back with exactly my thoughts (below). Oh, well...

Kangaroo Island

Most people know this as the "only good reason" for visiting Adelaide, and it is indeed a good one. During the guided bus tour of the island, you visit Seal beach, Remarkable Rocks (aptly named, trust me), Birds of Prey park, and Admiral's Arch. The most memorable moment was at the seal beach, when the guide was telling us we had to keep atleast 10 meters away from any seals at all times, and then suddenly realized that a seal was walking in front of us. We patiently followed 10 meters behind the seal as he led us to the beach, and at that point, I think we were all sufficiently in love with seals. Seeing 2 baby seals scampering on the beach "sealed" the deal. :)

Coorong River

In this tour, you spend much of the day on a boat with about 30 other people, and stop to take walks among the sand dunes. It's a lesser known tour, and that's a good thing - it's still got an intimacy/charm that the commercialized Kangaroo Island tour is lacking. Everything was a highlight: hunting for "cockles" (clams) in the ocean and swallowing them raw, eating deliciously fried cockles, rolling down a sand dune, accidentally stumbling upon an aboriginal skeleton, digging for (and drinking) freshwater from the ground, and spotting a koala on the ride back.

Botanical Gardens

Adelaide has an impressive set of gardens - cacti, roses, giant freaking Amazon water lilys, and more. It was a great place to wander around with a camera and a nose, and I was thrilled that a butterfly finally rested on a flower long enough for me to take a close-up.

World-Wide Developer Events & Groups: Let's Map This Shit!

After my travels around Asia and Australia for the past few months, I've seen that there are local developer groups and events *everywhere* - but it's not always easy to find a full list of all of them in one place. Wouldn't it be nifty if you could just consult a map of the world to see what was happening in your neck of the woods? Well, I think so, cuz I think maps solve everything. :)

So, let's see how big of a list (or how crowded of a map) we can make. If you know of a user group, local event (e.g. BarCamp Sydney), or local events aggregation site/calendar, then please submit it here. Then, check out all the contributions either in map form or spreadsheet form.

As a teaser, I've embedded a screenshot of the map with just some Australia data that I added. Keep reading below if you're curious how I've whipped up this mashup...

The whole system is built using Google Spreadsheets and the Google Maps API. There are two spreadsheets used: the first has rows of places with latitude/longitude coordinates, the second has rows of events/groups with their website and place. Both of them are contributed to using (modified) Google Spreadsheets Forms, which is one of the damn niftiest consumer features ever. The greatest thing about forms is that after you create a form using the spreadsheets UI, you can then copy the HTML, add whatever JavaScript/CSS/HTML you want, and host a new snazzy form on your server - as long as you retain the form input name attributes the same as the original and the form action set to the spreadsheet POST URL, then the form will continue working.

For the places form, I created it in spreadsheets and then added a map + geocoder to it so that I could get the latitude/longitude to store in the spreadsheet. More details on that hack are in this blog post. For the events/groups form, I created it in spreadsheets and then created the place dropdown by pulling in the JSON feed of the other spreadsheet and creating option elements for each row in the feed. More details on the JSON Spreadsheets feed are in this sample.

For the map of everything, I pull in the places spreadsheets feed, create a hash based on the location names, then pull in the events/groups spreadsheets feed, and push each row into an array in the matching location. Then I iterate through all the locations and plot them on the map with the associated stuff in the sidebar. To get the sized/labeled markers that represent the amount of stuff for each location, I use the Google Charts API (yep, it has a marker output). Oh, and finally, I set notification rules for each spreadsheet so that I know when anyone's contributed to them - it excites me! :)

((PLUG)): If you want to learn more about the Google Spreadsheets APIs, attend next week's talk at Google Sydney.

Webstock 2009: Client Killed the Server Star

After Anthony Baxter told me that he never repeats the same talk twice, I've challenged myself to do the same thing. It's disrespectful to your audience to give them a talk that wasn't custom tailored for them, and it's doing a disservice to your own personal learning - atleast for me, part of the fun of giving a talk is doing the research for the slides and content.

So, when I was asked to present on behalf of Google at Webstock 2009, I proposed a new talk called "Client Killed the Server Star: The New Client/Server Model". The talk was inspired by Brett Morgan's spiel about App Engine + Gears at the OSDC Hackathon, particularly his beginning slides about how the architecture for the web was changing from server-centric to client-centric, and it was motivated by my desire to research more about the traditional client/server model, and various HTML5/Gears APIs. Here's the talk abstract:

"Most modern websites still place a large burden on the server, constantly sending it requests and asking it to do heavy computations. In the brave new world, the client is king and the server is its faithful shadow. In this talk, we'll look at how cutting-edge technology like Gears, HTML5, and Google App Engine can be used to create websites where the caching, storage, and computing is done primarily in the browser/desktop and the server is used merely as a backup store."

You can check out the slides below, read a great write-up of the talk from one of the attendees, and watch the talk. Oh, and right, the talk title is clearly a parody of an 80s song, and if you're sharp, you'll notice other 80s references in the slides... I couldn't resist.

Here are some interesting links that I read while researching for this talk . Most of these are about the server aspect, as I was trying to imagine the ideal setup of the server/database for my proposed smart client:

• Is the Relational Database Doomed?
• Disk is the New Tape - Flash is the New Disk
• Disk trends will drive the need for column stores
• Memcache: When You Absolutely Positive Have to Get It To Scale the Next Day
• Client/Server Model on the Web

The talk went well, and it got me thinking about web development as it realistically stands now. Right now, we have places where we have really strong, smart clients - iPhone/Chrome/Safari/Google Earth (Webkit), Firefox, etc. But then we still have "dumb" clients lying around - IE6, Blackberry, etc. If we only had to cater to smart clients, we could design our applications to have dumb servers and most of the complexity in the client (as the talk shows). But, given the current state of things, we have to design our applications to replicate complexity in both the client and the server, and just figure out when it can safely pass off responsibility for complexity to the client and save our server effort. And that kinda sucks, because we all know redundant code in multiple languages is error-prone and messy. The solution right now is perhaps: 1) use frameworks that smartly export functionality to either the client or server as it sees fit (not sure if those exist), 2) decide that we don't care about dumb clients, or decide that we're content to send a very pared down version of the application to them, 3) get cryogenically frozen and wake up in 10 years when the world is awesome.

Read through the slides, and take this challenge: when writing your next web app, put all of the complexity in the client. See how far you can go. I dare you. ;)