Several social networking sites including Twitter and Digg have implemented some form of a streaming API. This is an emerging pattern with many high-volume content providers. At first glance, a streaming API may seem to be more resource intensive than traditional polling API, but as discussed here, it is actually much more streamlined.
Most of these server implementations use some sort of queuing system and a long-lived HTTP connection which clients use to have data delivered in near-realtime. In this post, I’m going to demonstrate just how easy it is to hook into the Twitter Streaming API. Specifically, we’ll be using the
streaming/filter method to consume any tweets which match a filter condition in realtime. I’m sure you can think of some neat ideas that can leverage this concept!
To demonstrate, let’s just create a console application in Visual Studio. We’re going to create a simple
WebRequest to connect and just start reading the stream. Then, we’ll just print out the json result to the console window. I’ll leave it up to you to write some parsing logic to actually do something with the data.
That’s all there is to it! Your output on the console window will look like this:
What you do with the data is where the real magic happens. The last I heard Twitter was producing somewhere in the range of 1000 tweets per second and I’m sure its much higher than that now. You will probably want to implement message queuing system where you hand of the response as soon as possible for some other process to handle without blocking the incoming stream.
In a future post I’ll describe some neat ideas for what you can do with the data from the stream.
Hope this helps.