When you have to perform a real big amount of operations you have two options Increase your computational power (like using a cloud solution or scale on more servers) Move your computations to the most available cloud platform of the world: your users To improve tweelter performances and avoid overloading twitter API we are studying a new computational architecture for tweelter which can return results to the user faster and give less overhead to our servers. The key to achieve the result is to move most of the search overhead to the computers of the other users currently viewing tweelter, like SETI@home does, by using tweelter you would speed up other users searches and also your own searches.
While speaking with the top-ix people during a meeting we started to talk about the need of a way to filter out “noise” from twitter searches. Probably everyone found that searching something on twitter returns a big list of retweets and duplicated tweets. As those reduce the ability to follow a discussion or an event on twitter they are usually more a problem than a useful result. At the end of that meeting Tweelter was born. Tweelter is a twitter search engine which filters out duplicated entries, retweets and permits to search results older than one month on most followed topics. More interesting thing is that tweelter performs those search in a parallel manner and on a distributed mongodb. While […]