When you have to perform a real big amount of operations you have two options
- Increase your computational power (like using a cloud solution or scale on more servers)
- Move your computations to the most available cloud platform of the world: your users
To improve tweelter
performances and avoid overloading twitter API we are studying a new computational architecture for tweelter which can return results to the user faster and give less overhead to our servers.
The key to achieve the result is to move most of the search overhead to the computers of the other users currently viewing tweelter, like SETI@home does, by using tweelter you would speed up other users searches and also your own searches.
While speaking with the top-ix people during a meeting we started to talk about the need of a way to filter out “noise” from twitter searches.
Probably everyone found that searching something on twitter returns a big list of retweets and duplicated tweets. As those reduce the ability to follow a discussion or an event on twitter they are usually more a problem than a useful result.
At the end of that meeting Tweelter was born.
Tweelter is a twitter search engine which filters out duplicated entries, retweets and permits to search results older than one month on most followed topics. More interesting thing is that tweelter performs those search in a parallel manner and on a distributed mongodb. While retrieving all the results of the same search using the twitter api would require more then 10-20 seconds by using tweelter you will get the same results in 2-3 seconds and the more a search is performed the faster it gets.
So give tweelter a try if you need to follow a discussion on twitter, it might help you to follow the discussion in an easier manner.