API Usability Testing

I often joke that API hackathons are basically API usability testing days, the best opportunity for API teams to see first-hand how developers use their APIs and what problems they ran into - so when I was invited to an AT&T API day, I assumed it was another hackathon. But in fact, it was a formal "API Usability Day", structured and designed to find out what AT&T needs to do to improve their developer experience. It was the first I'd heard of any developer-facing company putting together something like that, so I was intrigued to check it out. (Okay, plus they rewarded us monetarily for our time spent).

The Subject Selection

Before being admitted a seat at the event, we answered a survey about our development experience, provided resumes, and gave links to our LinkedIn/Github profiles. The survey asked questions like what languages we usually develop in and what mobile platforms we've used. I imagine they used the survey information to ensure that developers had some degree of prerequisite knowledge and perhaps also to get a range of developers at the testing day. It was a small group, about 12 of us in all.

The API Tests

When we arrived at the AT&T lab, we were each given a seat, WiFi, and power outlets. We used our own laptops, as they wanted each of us to use our development environment of choice - to mimic what it'd be like if we were at home. We were given four packets describing tasks, and instructed to try to finish them without consulting eachother , and if we did, we were to invite them over so they could witness what we discussed. If we really ran into a stumbling block that we couldn't go over (like an API error that just wouldn't quit) we could invite the API engineer over, and they would take notes and videotape as we worked through it. The tasks increased in difficulty, so much so that we had 2 hours alotted for the last one and as far as I know, none of us get through it all.

Here's what we were tasked with:

• Download the SDK
• Use command-line to send and receive SMS
• Built an oauth flow for authorizing a user and reporting their device location to them
• Build a webapp for users to buy access to the app, then upload pictures via MMS and browse them in a gallery.

We used new APIs in each test, so that by the end, we were familiar with almost all of their API offerings and had seen most of their documentation. The final two tasks also involved us setting up a frontend (which we each did in our language of choice, like Python for me and Go for Anton) but we weren't meant to spend much time on that - just enough so that we could test how the API flow worked for a webapp.

The Evaluation

At the end of each task, we filled out a survey about how we did and what we thought. Some of the questions asked were:

• Did you complete the task? How hard was it?
• Would you recommend this API?
• What would have made the experience of using the API better?
• Whose API does it better than we do?

Then we had an AT&T engineer come and chat with us more, asking us to describe how we set about doing the task (like what documentation we tried to use and where we got misled), and just getting our general feedback on the experience. We also emailed a zip of our code to them, though I don't know what they'll be doing with that. Oh, and because I'm a documentation-aholic like that, I also took extensive notes in a Google Doc that I've shared with them over email.

We were quite solitary throughout this whole process, trying to solve the problems using our own approaches and encountering our own sets of problems. At the end of the day, we all came together for a 30 minute debriefing where we talked about our general impressions of the AT&T APIs and developer experience, comparing and contrasting them with other APIs, and explaining what would make us want to use or not use AT&T's offering. One of my own suggestions to them was for them to hold an internal hackathon using their competitor's APIs - so they could see for themselves how their experiences differed and learn from them (or steal, whatever works :).

The First of Many?

Product usability testing is now common place, and as APIs are increasingly becoming the product these days, I think that API usability testing like the AT&T day could become more than just a rarity, and I think they can come in many forms. The AT&T day was really about how easy it was to use their documentation — you could also have tests centering around how intuitive an API syntax is, how easy it is to find answers to common errors.

An API usability testing day should not be the only way for API companies to understand their developer experience, of course. API companies should still be learning as much as they can from looking at how developers use their site, what developers complain about in the forums, what features they request in the issue tracker, etc. And as Anton pointed out during the de-briefing, API companies should be using the APIs internally as much as possible and learning from their own experiences. But it's definitely an interesting way to put new developers under a microscope and zoom in on their experience.

If you're still reading... have you ever participated in or organized a day like this? If you have an API, is it something you'd do?