A Yelp for Developer Platforms?

For my talk on mobile app strategies, I really wanted to give the audience an idea of the options out there, and help them understand the pros and cons of each option. I'd only used a handful of mobile app platforms myself, so for most of them, I had to pore through the website checking out what they offered and where they differed from other platforms. But there's something very important that a website can't tell me: what is it like to actually *be* a developer trying to use that platform? As usual, I took to Twitter to see if anyone among my followers had tried the platforms themselves and whether they liked them. I got a handful of replies — I think most developers are either doing native mobile app development or no mobile app development at all — and it wasn't enough of a sample size for me to reach conclusions about the developer experience for any of the platforms.

That's when I realized we need something like Yelp, but for developer platforms. When you're thinking of going to a restaurant, you can determine that they're close to you and that they offer meals that sound good to you, but you can't be sure you'll actually enjoy the experience of going to the restaurant. What if the service sucks? What if the meals take too long to come out? What if they just don't taste good? It's the same thing with a developer tool. You can check out the documentation and see that their feature list jives with what you need, but you can't know that you'll enjoy working with it. What if the support sucks? What if it takes too long for new versions to come out? What if the API is buggy? When you have a bad restaurant experience, you've only lost a few hours of your life, but when you have a bad developer experience, you've potentially lost a few months. Many platforms seem fine and dandy when you're doing the hello world, but then the problems come out later, after you've committed to the platform and dove deep into it.

Sure, we can continue what we do now — asking our colleagues or Twitter or HackerNews everytime we're thinking of trying a new platform — but it's a shitty state of affairs for every developer to have to do that every time. I propose instead that we have a review site, where we can aggregate opinions on developer platforms in one place. And I don't just want a 1-star to 5-star rating — we've all seen how that turns into 3.5 stars for everything. I'd want actually useful ratings and reviews that would hone in on the stuff that matters in a developer experience. For example, here are some of the questions you'd be asked when reviewing:

• How many hours have you spent using the platform?
• How comprehensive is the documentation?
• How up-to-date is the documentation?
• How buggy is it?
• When something goes wrong, how easily/quickly do you get an answer from the support team?
• How active is the community around it?
• Would you recommend it to others?
• What other similar platforms have you used? (And which do you prefer?)

We basically want to figure out how experienced a reviewer is (the Hello World experience is very different from the 6 months experience) and then see what their experience was like. We could weight the ratings based on experience for an aggregate score, or simply present all the reviews and let people figure out whose reviews matter to them. That could all be tweaked, the important thing is to gather the opinions, and for those opinions to be more fine-grained than simply "yay" or "nay."

A developer-focused review site could also include a few more features, like:

• categories and/or curated lists, so you could see all the mobile app SDK reviews at a glance.
• automatic mining of the social web with sentiment analysis for gathering tweeted opinions and blog posts
• automatic analysis of forums and code repositories to show level of activity for a platform

Like any reviews site, there's a big problem: how do you get enough people to use it so that there's sufficient data to see a general trend, and not just a few anecdotes here and there? Well, one way is to do what Yelp does: they get restaurants themselves to display Yelp badges, usually the better restaurants. The better platforms (or the platforms that think they are better) could link to their reviews, and developers will learn that a review site is out there. Most people on the web are content consumers and not content creators, but I unscientifically think that developers tend to be creators more than consumers, and that would help in populating the reviews.

So, there you have it: my wish for a developer-focused review site. I know it's not simple — it would require a lot more than my airplane musing time to really figure out how to make it work well — but I also think it would be a good thing for the developer world at large and I hope someone takes it on someday. What do you think? What do you think?