I’d like to introduce an interesting side project that’s been keeping me busy for a couple of months. It uses the Twitter Streaming API to see which dance music artists are popular right now and at any given time since tracking began – a couple of weeks ago. The site also enables drilling down to see which content caused spikes in popularity.
Update: This project is currently offline but will be resurfacing in the future.
The tracker currently handles about 120,000 (re)tweets a day. While I expect to see this figure increase during festivals, and other live events, experiments with higher volume Twitter users has demonstrated support for rates in excess of five million per day.
The app uses Rails as a backend REST API for an AngularJS client side application. Tweets and retweets mentioning or posted by a set of artists I define are pushed onto a redis queue and then aggregated in MySQL. With the data that I’m now accumulating, in realtime, I’ll able to add further visualizations and perform more analysis in the future.