RageTube: Using APIs to Bring Music Video Playlists Online

As I've mentioned in other posts, I have a thing for music videos. Whenever I visit a new foreign country, I try to find the local music videos channel and learn about their culture entirely through hours glued to the channel (it kind of works!). In Australia, I eventually discovered Rage, a 6-hour long stretch of music videos that plays every Friday night, Saturday morning, and Saturday night (midnight to 6am!). I did my best to reorganize my life so that I could catch atleast one of the 6-hour stretches each weekend, but alas, I eventually acquired hobbies and the semblance of a social life. At the same time, I've been increasingly disappointed with my worktime music options here — I can't use Pandora (US only), can't use Spotify (Europe only), and have had mixed success with Grooveshark. So, I decided to kill two birds with one stone, and make a mashup that'd let me watch Rage during the day in the form of Youtube music videos: RageTube!

To use RageTube, you provide a URL to any playlist from the Rage archives and it will take care of finding music videos on Youtube for all of the songs and playing through the list one at-a-time. You can skip through songs you don't like with the "Next" button, or you can click ahead or back to videos that you want to watch immediately. Plus, if you want to remember what songs you liked (or hated!), you can click the "Yay", "Meh", or "Nay" ratings widget and see your rating displayed in the playlist. That makes it even easier to find favorite videos to skip to later.

I wrote RageTube on a Monday morning and after sharing it on Twitter, I've discovered that there are quite a few Rage fans that are happy to have a way to enjoy Rage at work, including a few of my Australian colleagues abroad that never get the chance to watch Rage on TV anymore. I've added a few features since the initial version, like the ratings widgets, share-by-URL, and support for older playlists, but I still have much room to improve it, like by adding a playlist picker and playlist ratings.

Now, I have to admit I didn't write RageTube just because I needed a better music option at work — I also plan to use it as a demo in my upcoming GDD Tokyo Talk to showcase how much is possible with just the Google JavaScript APIs. The entire mashup is client-side - one HTML and 2 JS files - and it relies on 3 different JS APIs as well as some HTML5 functionality.

For those interested, here's how it works behind the scenes:

• When you enter a URL, I send that URL through open.dapper.net, service that screen-scrapes websites and gives you the desired nodes in the format of your choice. I fetch the JSON format using a lightweight JSONP library. I store the song and title in an internal JS array and render them out as a scrollable list.
  

  var dappUrl = 'http://open.dapper.net/transform.php';
  var params = {'dappName': dappName, 'transformer': 'JSON', 'applyToUrl': playlistUrl}
  JSONP.get(dappUrl, params, function(json) {
    ....
  });
  

• I then try to play the first song, and when I discover that there's no youtube ID stored in in the internal array (as will be the case for the first song), I use the Youtube JSON-C API to find the first matching video.
  

  var query = song.artist + ' ' + song.title;
  var searchUrl = 'http://gdata.youtube.com/feeds/api/videos';
  var params = {'v': '2',  'alt': 'jsonc',  'q': query}
  JSONP.get(searchUrl, params, function(json) {
    song.results = json.data.items;
    song.youtubeId = json.data.items[0].id;
    ...
  });
  

• The first time that I play a video, I embed the Youtube player SWF in the page using SWFObject, specifying a few parameters that will make it possible for me to use JavaScript to interact with that player later using the Youtube Player API.
  

  var params = {allowScriptAccess: 'always', allowFullScreen: 'true'};
  var atts = {id: 'youtubeplayer'};
  swfobject.embedSWF('http://www.youtube.com/v/' + song.youtubeId +
   '?autoplay=1&fs=1&enablejsapi=1&playerapiid=ytplayer',
   'videoBlock', '425', '356', '8', null, null, params, atts);
  

• Once I get programmatic access to the player, I register a callback so that I know whenever the player changes status (started/paused/ended).
  

  function onYouTubePlayerReady(playerId) {
    youtubePlayer = document.getElementById(playerId);
    youtubePlayer.addEventListener('onStateChange', 'onYouTubePlayerStateChange');
  }
  

• When the video ends, I play the next song. (I've already got the Youtube ID for the next song because I always search for the next video whenever I start playing a video.)
  

  function onYouTubePlayerStateChange(newState) {
    if (newState == 0) {
      currentSong++;
      playNext();
    }
  }
  

• To let users rate each video, I use a localStorage-based doodad that I wrote for another music video mashup.
  

  likerCol.appendChild(LIKER.createLikerMini(song.id));
  ...
  likerBlock.appendChild(LIKER.createLiker(song.id));
  

Happy RageTube'ing!

Embedding Feed Gadgets in Google Sites

Today, I spent a few hours re-organizing waveprotocol.org to be easier to navigate. As part of that re-org, I wanted to also make it clear to people visiting the site that the protocol is in active development by showing them the activity from our discussion group and code repository. Since both the group and project offer ATOM feeds, I figured I could just embed a gadget to show the latest posts from the feeds.
After spending a good half hour trying to find a gadget that would do just that, I gave up and wrote one myself. And now you can use the gadget yourself if you're in a similar situation. :)

Using the Gadget

Here's how you can actually embed the gadget on your site:

1. Put the target page in editing mode. Open the "Insert" menu and select the final "More gadgets" option.
2. Select "Add gadget by URL" in the sidebar of the dialog.
3. Enter this URL in the input box:
   http://pamelafox-samplecode.googlecode.com/svn/trunk/feedgadget/feedgadget.xml
4. Enter the URL to your feed in the "Feed" input box in "Setup your gadget". The feed must be either an ATOM or RSS feed.
5. Customize the width, height, and title as desired.

Tip: If you want to embed multiple gadgets next to eachother, change your page layout to a multi-column view and stick a gadget in each column.

Finding Feeds

Here are some tips for finding feed URLs for various Google properties:

• Blogger: Scroll to the bottom of the blog and click "Posts (ATOM)" link. Here's an example blogger feed URL:
  http://otherfancystuff.blogspot.com/feeds/posts/default
• Google Groups: Click the orange "XML" image at the bottom of the group home. You can pick from various feeds - the "15 new messages" ATOM feed works well in my experience. Here's an example group feed URL:
  http://groups.google.com/group/wave-protocol/feed/atom_v1_0_msgs.xml
• Google Code Projects: Click the "Feeds" link on the sidebar of the project front page. You can pick from various feeds, depending if you want to show all activity or limit to just issues/downloads/etc. Here's an example project feed activity URL:
  http://code.google.com/feeds/p/wave-protocol/updates/basic
• Google Sites: Each site has an associated feed for recent updates. There's not a good way to find the link to that feed, but you can construct it easily by adding "activity.xml" to the site URL. Here's an example site updates feed URL:
  http://www.waveprotocol.org/activity.xml

About the Gadget

For developers, here's some information about how the gadget works.

The gadget uses the AJAX Feeds API and the google.feeds.FeedControl class, and of course, it uses the gadgets API. It's actually a nice example of how to write a simple gadget that uses a Google API and user preferences:

<Module>
    <ModulePrefs title="Feed Control" height="400"/>
    <UserPref name="feedurl" display_name="Feed" default_value="https://groups.google.com/group/wave-protocol/feed/atom_v1_0_msgs.xml"/>
     <Content type="html"><![CDATA[ 
     <div id="feed" style="font-size: small;"></div>
     <script type="text/javascript" src="http://www.google.com/jsapi"></script>
     <script type="text/javascript">
     google.load("feeds", "1");
   

     function initialize() {
       var prefs = new gadgets.Prefs();
       var feedControl = new google.feeds.FeedControl();
       feedControl.addFeed(prefs.getString("feedurl"), "");
       feedControl.draw(document.getElementById("feed"), {});
     }
     google.setOnLoadCallback(initialize);
     </script>
     ]]>
    </Content>
</Module>

Using OAuth with Spreadsheets API on Django/AppEngine

In a previous blog post, I showed how to import a published spreadsheet feed into an App Engine datastore by just grabbing the JSON. For another project I'm working on, I need to be able to import a *private* spreadsheet into an App Engine datastore. Because of the need to authenticate the user (via the multiple steps of the ever-so-elegant OAuth dance), this importing requires much more finagling.

With the help of my trusty colleague Vic Fryzel, I've put together a set of Django views that use the Python GData Client Library and should run on both App Engine Django, and with some modification for token storage, other Django stacks. I'll walkthrough the views here.

There are four URL handlers required, two for token requests, one for actually importing the spreadsheet, and one to manage the flow:

urlpatterns = patterns('',  
  (r'^get_oauth_token', 'importer.views.get_oauth_token'),
  (r'^get_access_token', 'importer.views.get_access_token'),
  (r'^import_spreadsheet', 'importer.views.import_spreadsheet'),
  (r'^$', 'importer.views.main_page'),
) 

When the user visits the main page, they are asked to login so that the app can remember their auth tokens, and if they are already logged in, they are redirected to the first token handler:

def main_page(request):  
  if not users.get_current_user():
    return HttpResponseRedirect(users.create_login_url(request.build_absolute_uri()))

  access_token = gdata.gauth.AeLoad(ACCESS_TOKEN)
  if not isinstance(access_token, gdata.gauth.OAuthHmacToken):
    return HttpResponseRedirect('/importer/get_oauth_token') 

In this first step of the OAuth dance, the app requests an oauth token for the specified scope (Spreadsheets), key/secret (anonymous, as I haven't registered my app), and a callback URL to my app. It saves that token to the App Engine datastore using a convenience function in the client library. Then it redirects the user to the authorization URL for that token, and the user is presented with the "Grant access" screen.

def get_oauth_token(request):
    oauth_callback_url = 'http://%s:%s/importer/get_access_token' %
        (request.META.get('SERVER_NAME'), request.META.get('SERVER_PORT'))
    request_token = client.GetOAuthToken(SCOPES, oauth_callback_url,
        CONSUMER_KEY, consumer_secret=CONSUMER_SECRET)
    gdata.gauth.AeSave(request_token, REQUEST_TOKEN)

    authorization_url = request_token.generate_authorization_url()
   return HttpResponseRedirect(authorization_url) 

When the user returns from the authorization screen to the callback handler, the app retrieves the original token, asks Google to upgrade that to an access token, and saves the access token to the App Engine datastore again.

def get_access_token(request):  
  saved_request_token = gdata.gauth.AeLoad(REQUEST_TOKEN)
  request_token = gdata.gauth.AuthorizeRequestToken(saved_request_token,
      request.build_absolute_uri())
  access_token = client.GetAccessToken(request_token)
  gdata.gauth.AeSave(access_token, ACCESS_TOKEN)
  return HttpResponseRedirect('/importer/') 

The user is then redirected to the main page, and since it sees that there is now an access token for the user, it shows the user an input box for providing a spreadsheets URL. When it knows the spreadsheets URL, it retrieves the list feed for that spreadsheet and saves each row as an entity in the datastore.

def import_spreadsheet(request): 
  import re
  import models

  client.auth_token = gdata.gauth.AeLoad(ACCESS_TOKEN)

  spreadsheet = request.GET.get('spreadsheet')
  if spreadsheet.find('google.com') > -1:
    spreadsheet_key = re.search('key=([^(?|&)]*)', spreadsheet).group(1)
  else:
    spreadsheet_key = spreadsheet
  worksheet_id = 'od6'
  list_feed = 'https://spreadsheets.google.com/feeds/list/%s/%s/private/values' %
      (spreadsheet_key, worksheet_id)
  feed = client.get_feed(list_feed,
                         desired_class=gdata.spreadsheets.data.ListsFeed)
  for row in feed.entry:
    firstname = row.get_value('firstname')
    lastname = row.get_value('lastname')
    email = row.get_value('email')
    person = models.Person(firstname=firstname, lastname=lastname, email=email)
    person.save()

  return HttpResponse('Saved %s rows' % str(len(feed.entry))) 

Using that code, the end result is going from this spreadsheet...

... to these datastore entities:

To see the full code (with inline comments), check it out from my repository or download the zip.

For simplicity's sake, this sample shows the simplest possible import. In my actual project, I am also creating an entity that represents the entire spreadsheet, and the entity for each row refer to that entity. In addition, I have code to convert from the spreadsheets strings to other model types like dates.

Hopefully this project can serve as a basis for other developers using spreadsheets as an import source for their apps. Enjoy!

Porting from an App Engine RequestHandler to a Django View

For whatever reason, I've found myself porting Python App Engine apps over from App Engine's "django-esque" webapp framework to true django 1.0 with views.py, urls.py, and the like.

Besides learning about how urls.py and views.py work, I had to do some research to figure out how some of the webapp-isms translated to django-isms, so I thought I'd post my findings here in a table comparing the two:

Webapp	Django
author = self.request.get('author')	What you use depends on how specific you want to be about where parameter was passed: author = request.GET.get('author') author = request.POST.get('author') author = request.REQUEST.get('author')
class MyRequestHandler(RequestHandler): def get(self): # Do stuff def post(self): # Do other stuff	def handle_request(request): if request.method == 'GET': # do stuff elif request.method == 'POST': # do other stuff
host = self.request.host	host = request.get_host()
url = self.request.uri	url = request.build_absolute_uri(request.path)
query_string = request.query_string	query_string = request.META['QUERY_STRING']
self.error(500)	return HttpResponse(status=500) or from django.http import HttpResponseServerError return HttpResponseServerError()
self.redirect('/gallery')	from django.http import HttpResponseRedirect return HttpResponseRedirect('/gallery')
path = os.path.join(os.path.dirname(__file__), 'index.html') self.response.out.write(template.render(path, template_values))	from django.shortcuts import render_to_response render_to_response('index.html', path) Or, if for some reason you need the in-between products of that shortcut (like the generated string), you can use the longer version: from django.template.loader import get_template from django.template import Context t = get_template('index.html') html = t.render(Context(template_values)) return HttpResponse(html)
self.response.headers['Content-Type'] = 'application/atom+xml' self.response.out.write(xml_string)	return HttpResponse(xml, mimetype='application/atom+xml')

If you have suggestions for better "transformations", please let me know. I'm fairly new to Django and am happy to learn more about the right way to do things.

Teaching & Using Google Data APIs @ USYD

As I've posted on other blogs, I always love the idea of teaching Web APIs to university students and finding ways to use them in class assignments. Today, I visited the University of Sydney (USYD) and saw multiple ways that they're using APIs in education, and I'd like to share them here.

First, I gave a guest talk to an Object-Oriented Frameworks class on "Google Data APIs & the Google Docs API". For the next two months, the students in that class will be working on group projects with an education theme and combining the technologies they've studied, and I hope to show up on their demo day and see some cool examples of API usage.

After the talk, I went to lunch with the professor's research team, and watched videos about some really neat education & API related projects they're working on: iWrite, a system for submitting assignments as a doc & getting instructor feedback, and Glosser, a system for automatically creating questions about papers to make students think more about them; and for analyzing contributions of group members to a paper. Both of these use Google Data APIs in conjunction with the students' USYD Google Apps accounts, and are great examples of how research, APIs, and Google products can interact.

Though the Google Data APIs are not as "sexy" as our other APIs, they are incredibly useful and great teaching tools since they span across many Google products and build on existing web technologies like XML, ATOM, OAuth, and the HTTP protocol. They're also particularly useful for students at Google Apps enabled universities, since they can be used to create applications for accessing and modifying data in the Google Apps suite that they use daily.

When I was a TA at my old university (USC), we used Google docs and spreadsheets in our classes for keeping tabs on group work, and I used the APIs to automate processes for the professor. That was right before USC actually became a Google Apps domain -- if I was there as a TA or student now, I'd probably spend all day hacking on Google data APIs and App Engine to make cool apps for classes and clubs, and trying to get my classmates to join the fun.

Anyway, it was great to see how USYD is using our APIs across both their classes and research. Let me know if your university is up to anything similar! :)

Putting Europopped on the Map

After spending 3 years of my life making Maps API mashups, I now have a bit of an addiction. Whenever I see geographic information, I have this uncontrollable urge to visualize that information on a map. So when I started reading Europopped.com, a blog that chronicles awesome and awful music videos from European countries, I also started imagining how I could show those blog posts on a map. Last night, between episodes of Arrested development and True Blood (a balanced TV diet), I realized my fantasies: Europopped: On the Map.

Here's a step-by-step of how the mashup works:

• It uses the JavaScript Maps API v3 to create a map centered on Europe.
• It queries the Posterous API to retrieve all the tags and tag counts for the blog.
• It creates markers for each of the tags, using latitude/longitude coordinates stored in the JS, and showing the tag count on top of each marker.
• When you click on a particular country marker, it queries the Posterous API for all the posts for that tag.
• It creates a sidebar of links for each post, and setting a click listener that embeds the video for each post in the infowindow.

Here are some tips for how I made it quickly:

• I used my Spreadsheets Geocoder wizard to get the coordinates for all of the countries, and used a spreadsheet formula to generate JSON from my geocoded spreadsheet.
• I used getlatlon.com to find the ideal map center.
• I used MapIconMaker Wizard to generate a template marker image URL for my markers, and passesd the tag count into that image URL to change the number for each country.
• I used my CenterBox control class from another project to create a centered info box instead of the typical infowindow.
• I used regexpal.com to test the regular expression that extracted the Youtube URLs.
• I used my Posterous JSON API proxy for all of the API calls.

That's why I love APIs and the web -- once you're aware of them, they make it possible to quickly create new ways to use and explore the data and sites that you love.

Happy EuroPopping! :)