mongodb

Redis and MongoDB insertion performance analysis

Tuesday, March 16th, 2010 | Computer Science | Comments

Recently we had to study a software where reads can be slow, but writes need to be as fast as possible. Starting from this requirement we thought about which one between redis and mongodb would better fit the problem. Redis should be the obvious choice as its simpler data structure should make it light-speed fast, and actually that is true, but we found a we interesting things that we would like to share.

This first graph is about MongoDB Insertion vs Redis RPUSH.
Up to 2000 entries the two are quite equivalent, then redis starts to get faster, usually twice as fast as mongodb. I expected this, and I have to say that antirez did a good job in thinking the redis paradigm, in some situations it is the perfect match solution.
Anyway I would expect mongodb to be even slower relating to the features that a mongodb collection has over a simple list.

This second graph is about Redis RPUSH vs Mongo $PUSH vs Mongo insert, and I find this graph to be really interesting.
Up to 5000 entries mongodb $push is faster even when compared to Redis RPUSH, then it becames incredibly slow, probably the mongodb array type has linear insertion time and so it becomes slower and slower. mongodb might gain a bit of performances by exposing a constant time insertion list type, but even with the linear time array type (which can guarantee constant time look-up) it has its applications for small sets of data.

I would like to say that this benchmarks have no real value, as usual, and have been performed just for curiosity

You can find here the three benchmarks snippets

import redis, time
MAX_NUMS = 1000
 
r = redis.Redis(host='localhost', port=6379, db=0)
del r['list']
 
nums = range(0, MAX_NUMS)
clock_start = time.clock()
time_start = time.time()
for i in nums:
    r.rpush('list', i)
time_end = time.time()
clock_end = time.clock()
 
print 'TOTAL CLOCK', clock_end-clock_start
print 'TOTAL TIME', time_end-time_start
import pymongo, time
MAX_NUMS = 1000
 
con = pymongo.Connection()
db = con.test_db
db.testcol.remove({})
db.testlist.remove({})
 
nums = range(0, MAX_NUMS)
clock_start = time.clock()
time_start = time.time()
for i in nums:
    db.testlist.insert({'v':i})
time_end = time.time()
clock_end = time.clock()
 
print 'TOTAL CLOCK', clock_end-clock_start
print 'TOTAL TIME', time_end-time_start
import pymongo, time
MAX_NUMS = 1000
 
con = pymongo.Connection()
db = con.test_db
db.testcol.remove({})
db.testlist.remove({})
oid = db.testcol.insert({'name':'list'})
 
nums = range(0, MAX_NUMS)
clock_start = time.clock()
time_start = time.time()
for i in nums:
    db.testcol.update({'_id':oid}, {'$push':{'values':i}})
time_end = time.time()
clock_end = time.clock()
 
print 'TOTAL CLOCK', clock_end-clock_start
print 'TOTAL TIME', time_end-time_start

Tags: ,

Turbogears authentication over mongodb users database

Wednesday, March 3rd, 2010 | Software Development | Comments

As we saw that there isn’t a lot of documentation around about how to perform authentication in turbogears over mongodb we decided to create a simple code snippet and public it here to help people trying to obtain the same thing.

This is mainly a proof of concept and is quick and dirty way to obtain it. You will probably have something like ming as your model, instead of directly accessing mongo.

This code also validates password over the clear text one, you will probably have hashed passwords in your database, so remember to change validate_password method as required

To make it work you will have to place this code inside your config.app_cfg, it also expects you to have you database exposed as db inside your model

from my_app.model import db
from zope.interface import implements
from repoze.who.interfaces import IAuthenticator, IMetadataProvider
from repoze.who.plugins.friendlyform import FriendlyFormPlugin
from repoze.who.plugins.auth_tkt import AuthTktCookiePlugin
from repoze.who.middleware import PluggableAuthenticationMiddleware
 
def validate_password(user, password):
   return user['password'] == password
 
class MongoAuthenticatorPlugin(object):
    implements(IAuthenticator)
 
    # IAuthenticator
    def authenticate(self, environ, identity):
        if not ('login' in identity and 'password' in identity):
            return None
 
        login = identity.get('login')
        user = db.users.find_one({'user_name':login})
        if user and validate_password(user, identity.get('password')):
            return identity['login']
 
class MongoUserMDPlugin(object):
    implements(IMetadataProvider)
 
    def add_metadata(self, environ, identity):
        user_data = {'user_name':identity['repoze.who.userid']}
        identity['user'] = db.users.find_one(user_data)
 
class MyAppConfig(AppConfig):
    auth_backend = 'sqlalchemy' #this is a fake, but it's needed to enable
                                #auth middleware at least on TG2.0
 
    login_url = '/login'
    login_handler = '/login_handler'
    post_login_url = None
    logout_handler = '/logout_handler'
    post_logout_url = None
    login_counter_name = None
 
    def add_auth_middleware(self, app, skip_authentication):
        cookie_secret = pylons_config.get('auth_cookie_secret', 
                                          'myapp_adsfsdfh3423')
        cookie_name = pylons_config.get('auth_cookie_name', 
                                        'myapp_auth')
 
        who_args = {}
 
        form_plugin = FriendlyFormPlugin(self.login_url,
                              self.login_handler,
                              self.post_login_url,
                              self.logout_handler,
                              self.post_logout_url,
                              login_counter_name=self.login_counter_name,
                              rememberer_name='cookie')
        challengers = [('form', form_plugin)]
 
        auth = MongoAuthenticatorPlugin()
        authenticators = [('mongoauth', auth)]
 
        cookie = AuthTktCookiePlugin(cookie_secret, cookie_name)
 
        identifiers = [('cookie', cookie), ('form', form_plugin)]
 
        provider = MongoUserMDPlugin()
        mdproviders = [('mongoprovider', provider)]
 
        from repoze.who.classifiers import default_request_classifier
        from repoze.who.classifiers import default_challenge_decider
        log_stream = None
 
        app = PluggableAuthenticationMiddleware(app,
                                          identifiers,
                                          authenticators,
                                          challengers,
                                          mdproviders,
                                          default_request_classifier,
                                          default_challenge_decider)
 
        return app
 
base_config = MyAppConfig()
base_config.renderers = []
 
base_config.package = my_app
 
#Set the default renderer
base_config.default_renderer = 'genshi'
base_config.renderers.append('genshi')
base_config.renderers.append('json')
 
#Configure the base SQLALchemy Setup
base_config.use_sqlalchemy = False
base_config.model = my_app.model

Tags: , ,

Tweelter, the twitter filter

Saturday, December 5th, 2009 | Web | Comments

While speaking with the top-ix people during a meeting we started to talk about the need of a way to filter out “noise” from twitter searches.

Probably everyone found that searching something on twitter returns a big list of retweets and duplicated tweets. As those reduce the ability to follow a discussion or an event on twitter they are usually more a problem than a useful result.

At the end of that meeting Tweelter was born.

Tweelter is a twitter search engine which filters out duplicated entries, retweets and permits to search results older than one month on most followed topics. More interesting thing is that tweelter performs those search in a parallel manner and on a distributed mongodb. While retrieving all the results of the same search using the twitter api would require more then 10-20 seconds by using tweelter you will get the same results in 2-3 seconds and the more a search is performed the faster it gets.

So give tweelter a try if you need to follow a discussion on twitter, it might help you to follow the discussion in an easier manner.

Tags: ,

Search