Axant Tech Blog

TurboGears future performances comparison

Wednesday, March 21st, 2012 | Uncategorized | Comments

Recently I decided to give a quick benchmark for curiosity to the going to be branches of TurboGears2.

I quickstarted a simple genshi based application (plain turbogears2 quckstart) and then I created a plain controller method without template, to avoid counting the template generation overhead.

The application has been installed in three virtual environments: one with TG2.1.4, one with the development branch which is going to be TG2.2 and one with the development branch which is going to be TG2.3

The following graph reports the resulting requests/second that my pc has been able to serve on each turbogears version.

I have to admit that I’m quite happy with the results, the grow is steady and TG2.3 seems to be three times faster than the current turbogears while still being backward compatible (The benchmark application has been quickstarted with TG2.1.4 and ran without issues on all the three environments)

Tags: turbogears

Mastering the TurboGears EasyCrudRestController

Tuesday, January 31st, 2012 | Uncategorized | Comments

One of the key features of TurboGears2 is the great CRUD extension. Mastering the CRUD extension can really make the difference between spending hours or just a few minutes on writing a web app prototype or even a full application.

The CRUD extension provides two main features, the CrudRestController which is meant to help creating totally custom CRUDs and the EasyCrudRestController which provides a quick and easy way to create CRUD interfaces.

I’ll focus on the EasyCrudRestController as it is the easiest and more productive one, moving forward to the CrudRestController is quite straightforward after you feel confident with the Easy one.

The target will be to create, in no more than 40 lines of controller code, a full featured photo gallery application with:

• Multiple Albums
• Uploads with Thumbnails Generation
• Authenticated Access, only users in group “photos” will be able to manage photos
• Contextual Management, manage photos of one album at time instead of having all photos mixed together in a generic management section

If you don’t already know how to create a new TurboGears project, start by giving a look at TurboGears Installation for The Impatient guide. Just remember to add tgext.datahelpers to dependencies inside your project setup.py before running the setup.py develop command.

I’ll start by providing a Gallery and Photo model. To store the images I’ll use tgext.datahelpers to avoid having to manage the attachments. Using datahelpers also provides the advantage of having thumbnails support for free.

from tgext.datahelpers.fields import Attachment, AttachedImage
 
class Gallery(DeclarativeBase):
    __tablename__ = 'galleries'
 
   uid = Column(Integer, autoincrement=True, primary_key=True)
   name = Column(Unicode(100), nullable=False)
 
class Photo(DeclarativeBase):
    __tablename__ = 'photos'
 
    uid = Column(Integer, autoincrement=True, primary_key=True)
    name = Column(Unicode(100), nullable=False)
    description = Column(Unicode(2048), nullable=False)
    image = Column(Attachment(AttachedImage))
 
    author_id = Column(Integer, ForeignKey(model.User.user_id)))
    author = relation(app_model.User, backref=backref('photos'))
 
    gallery_id = Column(Integer, ForeignKey(Gallery.uid))
    gallery = relation(Gallery, backref=backref('photos', cascade='all, delete-orphan'))

Now to be able to start using our galleries we will have to provide a place where to view them and a gallery management controller to create and manage them. Viewing them should be quite straightforward, I’ll just retrieve the galleries from the database inside my index method and render them. To access a single gallery I’ll rely on the datahelpers SQLAEntityConverter which will retrieve the gallery for us ensuring it exists and is valid. For the management part I’ll create an EasyCrudRestController mounted as /manage_galleries

from tgext.crud import EasyCrudRestController
 
class GalleriesController(EasyCrudRestController):
    allow_only = predicates.in_group('photos')
    title = "Manage Galleries"
    model = model.Gallery
 
    __form_options__ = {
        '__hide_fields__' : ['uid'],
        '__omit_fields__' : ['photos']
    }
 
class RootController(BaseController):
    manage_galleries = GalleriesController(DBSession)
 
    @expose('photos.templates.index')
    def index(self, *args, **kw):
        galleries = DBSession.query(Gallery).order_by(Gallery.uid.desc()).all()
        return dict(galleries=galleries)
 
    @expose('photos.templates.gallery')
    @validate(dict(gallery=SQLAEntityConverter(Gallery)), error_handler=index)
    def gallery(self, gallery):
        return dict(gallery=gallery)

Logging in with an user inside the photos group and accessing the /manage_galleries url we will be able to create a new gallery and manage the existing ones.

To configure how the crud controller forms should appear and behave the __form_options__ property of the EasyCrudRestController can be used. This property relies on the same options as Sprox FormBase and customizes both the Edit and Add forms.
The next part is probably to be able to upload some photos inside our newly created galleries. To perform this we will create a new EasyCrudRestController for gallery photos management.

from tgext.crud import EasyCrudRestController
from tw.forms import FileField
from tw.forms.validators import FieldStorageUploadConverter
from webhelpers import html
 
class PhotosController(EasyCrudRestController):
    allow_only = predicates.in_group('photos')
    title = "Manage Photos"
    model = model.Photo
    keep_params = ['gallery']
 
    __form_options__ = {
        '__hide_fields__' : ['uid', 'author', 'gallery'],
        '__field_widget_types__' : {'image':FileField},
        '__field_validator_types__' : {'image':FieldStorageUploadConverter},
        '__field_widget_args__' : {'author':{'default':lambda:request.identity['user'].user_id}}
    }
 
    __table_options__ = {
        '__omit_fields__' : ['uid', 'author_id', 'gallery_id', 'gallery'],
        '__xml_fields__' : ['image'],
        'image': lambda filler,row: html.literal('‹img src="%s"/›' % row.image.thumb_url)
    }

Mounting this inside the RootController as manage_photos = PhotosController(DBSession) it will be possible to upload new photos inside any gallery. To manage the photos inside the first gallery for example we will have to access /manage_photos?gallery=1url.

Each parameter passed to the EasyCrudRestController is used to filter the entries to show inside the management table and the keep_params option provides a way to keep the filter around. This makes possible to edit the photos of only one gallery at the time instead of having all the photos mixed together. Also when a new photo is created it will be created in the current gallery.

The PhotosController got more customization than the GalleriesController, through the __field_widget_types__ and __field_validator_types__ options we force the image field to be a file field and using the __field_widget_args__ we ensure that the newly uploaded photos have the current user as the author.

__table_options__ provide a way to customize the management table. The available options are the same as the Sprox TableBase and Sprox TableFiller objects. in this case we hide the indexes of the rows on the database and the gallery itself, as we are managing the photos of a specific gallery we probably don’t need to know which galleries the photos belong to. Using the __xml_fields__ we also specify that the image field provides HTML and so doesn’t have to be escaped. The image entry forces the table to show the image thumbnail for the image column of the table instead of printing the AttachedImage.__repr__ as it would by default.

At first sight it might sound a bit complex, but once you start feeling confident, the CRUD extension makes possible to create entire applications in just a bunch of code lines. With just a few lines of code we created a photo gallery with multiple albums support and we can now focus on the index and gallery templates to make the gallery as pleasant as possible for our visitors.

The complete implementation of the photo gallery is available as a pluggable application on bitbucket, feel free to use it in your TurboGears projects.

Tags: turbogears

TurboGears2 DebugBar

Saturday, November 12th, 2011 | Uncategorized | Comments

Recently some work has been done to extend the hooks support in TurboGears, to play a little with the new hooks I decided to try creating the famous and envied Django Debug Toolbar. I’m quite happy of the result and most of the features are there. In a few days I’ll be able to place it on a public repository and I’ll release it concurrently with the 2.1.4 release of TurboGears.

Debug Toolbar

Timings

Request and Headers

SQLAlchemy Queries

Mounted Controllers

The code has been heavily inspired by the Pyramid Debug Toolbar and have to thank the Pyramid team for the good job they did at making the Toolbar code clean and simple.

Tags: turbogears

TurboGears2 Performance Improvements

Tuesday, October 4th, 2011 | Uncategorized | Comments

As recently some effort has been involved in improving the performances of TurboGears2, I was curious to see how much things improved. As usually, the test isn’t really reliable in any way and was just for fun.

All the graphs report the request/sec the application has been able to perform on my computer with only 1 concurrent client. So higher is better.

Here is the comparison between TG2.0 and TG2dev (will be 2.1.4)

I also compared various setups with different template engines on TG2dev

The comparison happened on an application similar to the quickstarted one.
Actually as there is no database involved in this application the template engine impacts a lot and so was a good benchmark for the template engines themselves.

Tags: genshi, kajiki, mako, turbogears

TurboGears 2.1.1 released!

Sunday, June 19th, 2011 | Uncategorized | Comments

After a reflection moment caused by the need to think what will follow after the pylons and repoze.bfg merge in pyramid the TurboGears2 team has decide to gather up all its forces and give to TurboGears2 its own independent life.

The first steps have been oriented to improve the framework reliability and brought to life the TurboGears continuous integrations system and a standard project release process.

After a few months of work 2.1.1 has been released and it brings many fixes and improvements, 2.1.2 is under its way and a 2.2 release with major improvements is already planned!

TG2 Core:
* Fixed dependencies for Python 2.4. Now any packages that are
needed are automatically installed.
* Updated package requirements as high as possible.
* Verified nested RestControllers work as expected
* Added/fixed Kajiki support
* Ignore repoze.who_testutil when running nosetests
* Fixed import order for pylons.middleware
* Fixed crash when PYTHONOPTIMIZE is enabled
* Report a warning about ErrorMiddleware is disabled
* Fixed concurrency issues with use_custom_format
* Fixed 404 errors if a controller uses only custom formats
* Verified that user object is available inside of the error controller/template
* Fixed expansion of arguments on before/after calls
* Fixed wrong header response for 405 error
* Fixed WebOb version requirment. Newer version required
* Added test case to check for replace_header when called from WSGIApp
* Fixed issues with Content-Type header appearing multiples times on 204/205 responses
* Removed redundant hasattr checks on override_template
* Improved support for pylons 1.0 strict_c
* Fixed post traceback, now reports to Pylons correctly
* Added test case to check for spurious content type removal on empty content
* Fixed crash when content type header is missing
* Fixed crash when response Content-Type is set to None
* Fixed support for etags. Pylons 1.0 changes slightly, we support the correct version now
* Added dependency_links and setup.cfg allow_hosts: easy_install TurboGears2 now works
* Fixed DecoratedController. should not call super(), 2.6 revealed a problem
* Fixed Genshi output method. Use XHTML if none specified, instead of XML

TG2 Devtools:
* Fixed Python 2.4 compatibility issues. Dependencies are now automatically specified
* Updated package version requirements as high as possible
* Fixed about.html instructions about where the logo is found
* Set “zip_safe=False” by default in the templates now
* Tests fixed, now pass
* Added support for sqlalchemy-migrate
* Added option to choose config file
* Added archive_tw_resources command for projects
* Fixed deprecated redirect calls
* Set Genshi templating method by default to XHTML
* Adding dependency_links: easy_install tg.devtools now works

Tags: turbogears

Mobile devices detection with TurboGears2

Wednesday, April 20th, 2011 | Uncategorized | Comments

We just released tgext.mobilemiddleware for turbogears2 to make easier to handle templates for mobile devices and detect mobile devices requests.

Indeed it is quite simple to use as it makes possible just to register a different template by using @expoe_mobile decorator which will be used for mobile devices, making possible to create mobile version of web page by using for example jquery mobile

Tags: turbogears, turbogears2

Fixing broken inline genshi tags on TG2.1

Friday, January 14th, 2011 | Uncategorized | Comments

Turbogears 2.1 got a little issue with genshi templates being rendered as xml instead of xhtml. This causes the most various strange problems with tags nesting being closed in random points as the browser likes.

This is caused by an improvement to the mechanism used to decide with rendering format to use that it is now based on the response content type. If your applications returns Content-Type text/xml or text/plain your templates will be rendered as xml or as plain text.

But what happens with your application returns text/html which is the default case?

Indeed in this case the application asks to your configuration file what to do, and as in your configuration file by default there isn’t any templating.genshi.method entry you get the default genshi behaviour, which is xml. Your browser doesn’t really like receiving some xml when he is expecting html and so handles the tags as not being closed.

In the recent future it will probably be fixed, in the mean time you can work around this problem by declaring in your development.ini

[DEFAULT]
templating.genshi.method = xhtml

This will reassemble the default behaviour that was available in TG2.0 rendering your templates as xhtml and correctly showing them inside the browser. If you have specified in your template the html doctype instead of the default xhtml one, then you might also want to change that value to html.

Tags: turbogears

Workaround for empty helpers in Turbogears 2.1

Tuesday, January 4th, 2011 | Uncategorized | Comments

Last release of TG2.1 has a subtle bug due to the changes helpers management to support in pylons 1.0, this makes the “h” object appear as an empty dict instead of your application helpers module.

The bug is already tracked on http://trac.turbogears.org/ticket/2488 and as the fix is quite easy it will be probably fixed soon. In the mean time you can work around it by monkey patching the pylons.templating.pylons_globals (I do it in lib/app_globals.py but any place is fine).

import tg.render, pylons
def patched_pylons_globals():
    x = tg.render.my_pylons_globals()
    if x['h'] == {}:
        conf = pylons.config._current_obj()
        x['h'] = conf['package'].lib.helpers
    return x
pylons.templating.pylons_globals = patched_pylons_globals

This will make your helpers work again in TG2.1.
Please pay attention that this code is specific for TG2.1, TG2.0 didn’t perform pylons_globals monkey patching and so the code won’t work.

Tags: turbogears

Lowering Tg2 memory usage by running multiple instances of an app inside same WSGI daemon process

Friday, August 6th, 2010 | Software Development | Comments

I was recently trying to deploy one app multiple times inside the same
WSGIProcessGroup and WSGIApplicationGroup %{GLOBAL} to reduce memory
usage.
This works quite well except for all the SQLAlchemy sessions which end being
attached to the engine of the last wsgi script started.

The best solution that I have been able to get so far is to create a proxy interface to the application engine. This way each wsgi script gets binded to the same engine, but the engine itself keeps track of all the available real engines and responds to the script requests sending them to the right real engine.

class MultiSiteEngine(object):
   def __init__(self):
       self.engines = {}
 
   def __getattr__(self, name):
       if name == 'engines':
           return object.__getattribute__(self, name)
 
       if not self.engines.has_key(config['sqlalchemy.url']):
           self.engines[config['sqlalchemy.url']] = engine_from_config(config, 'sqlalchemy.')
 
       return getattr(self.engines[config['sqlalchemy.url']], name)

To make this work just allocate a multi_engine = MultiSiteEngine() inside your tg2 app model __init__ and change init_model method to pass multi_engine instead of the engine itself.

Tags: turbogears

Patched TG2 bootstrap script for python 2.6

Wednesday, July 28th, 2010 | Uncategorized | Comments

As actually I end up installing Tg2 on various systems having python2.6 quite often I have patched the bootstrap script to work on it.

If you had problems installing TG2 on python2.6 failing on Extremes or Zope dependencies you can try to use this script, it should work for you:

https://bitbucket.org/_amol_/tg-bootstrap-py2.6/src/tip/tg2-bootstrap.py

Tags: turbogears