I built a Twitter based search for #Screenr on Mozealous.com

If you have ever tried to use the Twitter search you have probably noticed it doesn’t feel much like a search.  It is more of a live filter or live pulse for a specific word, showing you a stream of live tweets containing that word.  Try using the Twitter search at http://search.twitter.com and search for something like “#CS5″ (Adobe Creative Suite 5) and you will see what I mean.  It doesn’t return the most interesting results, it just returns the most current results (imagine if Google did this).  Don’t get me wrong, current is cool, like in the event of an earthquake, or zombie apocalypse it is nice to “See what’s happening — right now.”  But maybe that isn’t what you are looking for, maybe you want to find the most interesting Tweet on #CS5, or in the case of Screenr the most interesting screencast on PowerPoint, not just the most recent.  So several months ago I was bored and set out to build a better search system for Screenr, and I did, and here is how I did it.

How to make a better search

This morning the New York Times posted an article on TweetUp, a company that is building a smarter search for Twitter (read the article here).  Essentially, what they are trying to do is turn the search into more than just a pulse, they want an actual search, like Google’s that ranks the search based on factors to determine if it will be the most relevant.

TweetUp measures a Tweets popularity by:

• Measuring how often readers repost the tweets (number of ReTweets)
• How many clicks each links get

They also just raised $3.5 million in funding for this concept (along with an interesting monetization idea).

Measure interestingness by the number of ReTweets

I think this is a pretty cool concept, if something is ReTweeted it usually means it is interesting, and the more ReTweets a tweet gets, the more interesting it is.  What I was happy about was this was the same conclusion I came to when building my search.  So tweets that are more heavily ReTweeted should appear higher in the search results, so this is what I did.

Other ways to measure interestingness

So I kept thinking about this and wondered if there were other ways that you could measure “interestingness.”

Here were some alternative ideas I had for measuring interestingness:

• Reach – “Reach” I defined as the total number of followers of everyone who tweeted the screencast.
• Views – Simply measuring the number of views of a particular screencast.
• Views per Reach - The number of views of a particular screencast divided by the “Reach” of the tweet.  The idea behind this though would be that the most interesting screencast would be the one that compelled people that saw the Tweet to actually view the video.

The difficulty of these approaches is that at Screenr we don’t give you access to the number of views through any type of API so that I can analyze the data, and when trying to determine the “Reach” of a Tweet it can use several hundred API calls to twitter that quickly push you over the API call limit.  So I abandoned any of these other approaches.

How I built my search

Here is a quick rundown of what I did:

• Dump all Screenr tweets from Twitter in a database every hour using the Twitter API. Twitter only gives you access to a couple of weeks worth of data using their search.  To deal with this I run an automated script every hour that dumps all tweets that contain the text “http://screenr.com” into a database.  This database is what I use for the search queries.  Obviously on screencasts that have been tweeted will show up in my search results because I am using the Twitter API.
• Search the database when someone searches, then rank results based on RTs. When you search using my search tool, I will query the database and find every screencast that uses your search term.  I then look for which screencast has been tweeted the most, then display that Tweet at the top of the results.

Pretty simple huh?

Use my Twitter based search for #Screenr

If you want to try it out you can use my Screenr search here:

Or by visiting it directly here:

http://www.mozealous.com/screenr/

This isn’t an official Screenr search

Just so you know, this is something I was just playing around with, at some point in the future Screenr will release an official Screenr search that is better than this.

How this search can be improved

There are a couple of things I don’t like about my search, and I think that it could be improved.  Here is what I think should be done:

• Only display the original tweet in the search results (no need to show all RTs)
• Create a better search results page, like maybe something like what YouTube does, or Bing video search, that shows the video, or video thumb, not just video description.  After all, you are looking for the video, not the description.
• Handle apostrophes better.  I don’t handle them well.

You should improve it

If you are interested in improving this search, and are comfortable programming in PHP I would be more than happy to give you access to source code I used to build this search.  If you are, let me know in the comments and I’ll give you the code and the current database I am using that stores the tweets.