TurboGears 2.3 Hidden Gems #2 - Application Wrappers

amol | November 10th, 2013 | Web | Comments

One of the less known features introduced in TurboGears 2.3 are application wrappers.
Application wrappers are much like controller wrappers (available since 2.2), but instead of wrapping controllers they actually wrap the whole application providing an easier way to implement what in plain WSGI is done through middlewares.

The advantage of application wrappers over middlewares is that they have full access to TurboGears stack, they can access the current request, the database, session and caches as the application itself would do.

The great part is that, as they run between TGApp and TGController, they can also replace the TurboGears Context and the TurboGears Response providing a great way to hijack requests, responses or even replace entire features of the framework like the cache layer. A very similar concept is available in other frameworks like Pyramid Tweens.

A very simple application wrapper that intercepts exceptions and logs them without messing with the standard TurboGears error handling might look like:

class ErrorLoggingWrapper(object):
    def __init__(self, handler, config):
        self.handler = handler
        self.config = config
 
    def __call__(self, controller, environ, context):
        path = context.request.path
        try:
            return self.handler(controller, environ, context)
        except:
            log.exception('Error while handling %s', path)
            raise

The wrapper can then be enabled calling

base_config.register_wrapper(ErrorLoggingWrapper)

inside config/app_cfg.py

Now that we have an application wrapper able to log exceptions we can decide for example to add another one that suppresses exceptions and prints “Something went wrong!”, as it is possible to specify the order of execution for application wrappers we can register a SuppressErrorsWrapper that should execute after the ErrorLoggingWrapper:

from webob import Response
 
class SuppressErrorsWrapper(object):
    def __init__(self, handler, config):
        self.handler = handler
        self.config = config
 
    def __call__(self, controller, environ, context):
        try:
            return self.handler(controller, environ, context)
        except:
            return Response('Oh! Oh! Something went wrong!', status=500, content_type='text/plain')

Then it can be registered after the ErrorLoggingWrapper using:

base_config.register_wrapper(SuppressErrorsWrapper, after=ErrorLoggingWrapper)

While applications wrappers are a powerful feature, most of their power comes from the new response management refactoring that makes possible to access the current context and replace the outgoing response while working with high level objects instead of having to manually cope with WSGI.

Tags:

TurboGears 2.3 Hidden Gems #1 - New Response Management

amol | November 2nd, 2013 | Opensource | Comments

TurboGears2.3 has been a major improvement for the framework, most of its code got rewritten to achieve less dependencies, cleaner codebase a cleaner API and a faster framework. This resulted in reduction to only 7 dependencies in minimal mode and a 3x faster codebase.

While those are the core changes for the release, there are a lot of side effects that users can exploit at their benefit. This is the reason why I decided to start this set of posts to describe some of those hidden gems and explain users how to achieve the best from the new release.

The first change I’m going to talk about is how the response management got refactored and simplified. While this has some direct benefits it also provided some interesting side effects it makes sense to explore.

How TurboGears on Pylons did it

TurboGears tried to abstract a lot of response complexity through tg.response object and as there were not many reasons to override TGController.__call__ it was common that the response object body was always set by TurboGears itself.

Due to the fact that Pylons controllers were somehow compliant to WSGI itself the TGController was then in charge of calling the start_response function by actually providing all the headers user set into tg.response

response = self._dispatch_call()
 
# Here the response body got set, removed for brevity
 
if hasattr(response, 'wsgi_response'):
  # Copy the response object into the testing vars if we're testing
  if 'paste.testing_variables' in environ:
      environ['paste.testing_variables']['response'] = response
  if log_debug:
      log.debug("Calling Response object to return WSGI data")
 
  return response(environ, self.start_response)

While this made sense for Pylons, where you are expected to subclass the controller to perform advanced customizations, it was actually something unexposed to TurboGears users.

TurboGears made possible to change application behaviour using hooks and controller_wrappers. So the use for subclassing the TGController was actually strictly related to custom dispatching methods, which was usually better solved by specializing the TGController._dispatch method (tgext.routes is a simple enough example of this).

Cleaning Up Things

This lead to a curious situation where the TGController needed to speak with TGApp through WSGI to make Pylons happy, so it needed to call start_response and return the response iterator itself. TGApp was supposed to be the WSGI application, but in fact most of the real work was happening into TGController, in the end we had two WSGI applications: both TGController and TGApp were callable that spoke WSGI.

The 2.3 rewrite has been a great occasion to solve this ambiguity by providing a clear communication channel between TGController and TGApp by assigning each one a specific responsibility.

Communication Channel

In TG2.3 only the TGApp is now in charge of exposing the WSGI application interface. The TGController is expected to get a TurboGears Request Context object and provide back a TurboGears Response object. The TGApp will then use the provided response object to submit headers and response body.

The TGController code got much more straightforward and the whole testing and call response part was moved to the TGApp itself:

try:
    response = self._perform_call(context)
except HTTPException as httpe:
    response = httpe
 
# Here the response body got set, removed for brevity
 
return response

This has been possible without breaking backward compatibility thanks to the fact that the only subclassing of TGController common in TurboGears world was the BaseController class implemented by most applications.

The BaseController usually acts just as a pass-through between TGApp and TGController to setup some shortcuts to authentication data and other helpers for each request. So the fact that the parameters received by BaseController.__call__ changed didn’t cause an huge issue as they were just forwarded to TGController.__call__

A little side effect

One of the interesting effects of this change is that your controllers are now enabled to return any instance of webob.Response.

In previous versions it was possible to return practically only webob. WSGIHTTPException subclasses (as they exposed a wsgi_response property which was consumed by Pylons), so it was possible to return an HTTPFound instance to force a redirect, but it was not possible to return a plain response.

A consequence of the new change is enabling your controller to call third party WSGI applications by using tg.request.get_reponse with a given application. The returned response can be directly provided as the return value of your controller.

This behaviour also makes easier to write reusable components that don’t need to rely on tg.response and change it. Your application can forward the request to them and proxy back the response they return.

Part #2 will cover Application Wrappers, which greatly benefit from the new response management.

Tags:

It’s a Pluggable World

amol | September 8th, 2012 | Uncategorized | Comments

One of the new additions in TG2.1.4 has been the support for the so called pluggable applications, this is a really powerful and convenient feature that probably not enough TurboGears users started embracing.

For people that never used them, pluggable applications provide a python package that can be installed and “plugged” inside any existing TurboGears application to add new features. Django has been probably the first framework to bring this feature to Python world and TurboGears implementation tries to be as convenient by making pluggable applications identical to plain TurboGears applications and providing a “quickstart-pluggable” command that creates the application skeleton for you. Pluggable applications can be installed using easy_install or pip and they can off course depend on any other pluggable application they need.

This year, at EuroPython 2012, I have been pleased to present a talk about using TurboGears for rapid prototyping (both in Italian and English, you should be able to find the videos on EuroPython youtube channel), so I decided to dedicate a part of it to pluggable applications as they are actually the fastest way to rapidly prototype a project. With my surprise most the questions I received were about the EasyCrudRestController and not about pluggable applications.

While the EasyCrudRestController is definitively a powerful tool, it’s far from being the answer to all the web developments needs. In most of the applications you are going to develop, users will probably prefer consulting content from something more engaging than an administration table of database entries.

This month, to create a set of utilities that can help people with their everyday needs, I decided to ask guys that work with me to make every part of the web sites that they were writing as pluggable applications. The result of this experiment has been that most of the pluggable apps that I did in my spare time (tgapp-smallpress, tgapp-photos, tgapp-tgcomments, tgext.tagging and so on) ended being used in real world projects and started to improve exposing hooks and ways to customize their behavior for the project they were going to be used.

After a few weeks, new pluggables like tgapp-fbauth, tgapp-userprofile, tgapp-calendarevents, tgapp-fbcontest, tgapp-youtubevideo has seen light and developing the target application started becoming blazing fast: Just plug what you need and customize it.

Embracing this philosophy the last project I’m working on has an app_cfg.py file that looks like:

plug(base_config, 'tgext.debugbar', inventing=True)
plug(base_config, 'tgext.scss')
plug(base_config, 'tgext.browserlimit')
plug(base_config, 'registration')
plug(base_config, 'photos')
plug(base_config, 'smallpress', 'press', form='XXX.lib.forms.ArticleForm')
plug(base_config, 'tgcomments', allow_anonymous=False)
from XXX.lib.matches import MyKindOfEvent
plug(base_config, 'calendarevents', 'eventi', event_types=[MyKindOfEvent()])
replace_template(base_config, 'smallpress.templates.article', 
                              'XXX.templates.press.article')
replace_template(base_config, 'smallpress.templates.excerpt', 
                              'XXX.templates.press.excerpt')

Thanks to this our development process has really improved: whenever a developer finds a bug he just has to propose a patch for the target pluggable, whenever someone notices a missing index on a query he has just to add it to the given pluggable. All the websites under development improved like people were working on the same project.

While existing pluggables might be limited, buggy or slow I’m getting confident that they will continue to improve, and some day they will surpass whatever custom implementation I can think of. I think I’m going to heavily rely on pluggable applications for any future project sticking to only one rule: “make it opensource”. This way, apart from probably helping other people, I’m also improving my own projects through other people feedbacks, bug reports and patches to the pluggables I used.

So, next time you have to start a new project give a look at the TurboGears CogBin and check if there is a pluggable application that looks like what you need. If you find any issue or find space for improvements just fork it and send a pull request, or send an email on the TurboGears Mailing List I’ll do my best to address any reported issue thanking you for your feedbacks as I’m aware that you are actually improving any past and future project that relies on that pluggable.

Tags:

What’s new about Sprox 0.8

amol | August 16th, 2012 | Uncategorized | Comments

Today Sprox 0.8 got released, it is the first release to add ToscaWidgets2 support. Depending on which version of ToscaWidgets is available inside your environment Sprox will either use TW1 or TW2 to generate its forms.

Being mostly a TW2 oriented release it might seem that not a lot changed since the previous version, but a little gem is hidden between all the TW2 changes as Sprox now supports setting default behavior for models themselves using the __sprox__ attribute inside model declaration.

class Parent(DeclarativeBase):
    __tablename__ = 'parents'
 
    uid = Column(Integer, primary_key=True)
    data = Column(String(100))
 
class Child(DeclarativeBase):
    __tablename__ = 'children'
 
    class __sprox__(object):
        dropdown_field_names = {'owner': ['data']}
 
    uid = Column(Integer, primary_key=True)
    data = Column(String(100))
 
    owner_id = Column(Integer, ForeignKey(Parent.uid))
    owner = relation('Parent')

The previous code example makes Sprox use the Parent data field for selection fields when choosing the parent of Child entities.

Apart from making easier to share options between your AddRecordForm and EditableForm __sprox__ attribute opens a great way to customize the TurboGears admin.

By adding a __sprox__ attribute inside your models you will be able to change the TurboGears admin behavior without having to create a custom admin configuration. Setting __sprox__ attribute makes possible to change most sprox properties changing CrudRestController behavior, the same properties that are documented on sprox.org can be specified inside the __sprox__ attribute by simply removing the underscores.

Tags:

TurboGears future performances comparison

amol | March 21st, 2012 | Uncategorized | Comments

Recently I decided to give a quick benchmark for curiosity to the going to be branches of TurboGears2.

I quickstarted a simple genshi based application (plain turbogears2 quckstart) and then I created a plain controller method without template, to avoid counting the template generation overhead.

The application has been installed in three virtual environments: one with TG2.1.4, one with the development branch which is going to be TG2.2 and one with the development branch which is going to be TG2.3

The following graph reports the resulting requests/second that my pc has been able to serve on each turbogears version.

I have to admit that I’m quite happy with the results, the grow is steady and TG2.3 seems to be three times faster than the current turbogears while still being backward compatible (The benchmark application has been quickstarted with TG2.1.4 and ran without issues on all the three environments)

Tags:

Mastering the TurboGears EasyCrudRestController

amol | January 31st, 2012 | Uncategorized | Comments

One of the key features of TurboGears2 is the great CRUD extension. Mastering the CRUD extension can really make the difference between spending hours or just a few minutes on writing a web app prototype or even a full application.

The CRUD extension provides two main features, the CrudRestController which is meant to help creating totally custom CRUDs and the EasyCrudRestController which provides a quick and easy way to create CRUD interfaces.

I’ll focus on the EasyCrudRestController as it is the easiest and more productive one, moving forward to the CrudRestController is quite straightforward after you feel confident with the Easy one.

The target will be to create, in no more than 40 lines of controller code, a full featured photo gallery application with:

  • Multiple Albums
  • Uploads with Thumbnails Generation
  • Authenticated Access, only users in group “photos” will be able to manage photos
  • Contextual Management, manage photos of one album at time instead of having all photos mixed together in a generic management section

If you don’t already know how to create a new TurboGears project, start by giving a look at TurboGears Installation for The Impatient guide. Just remember to add tgext.datahelpers to dependencies inside your project setup.py before running the setup.py develop command.

I’ll start by providing a Gallery and Photo model. To store the images I’ll use tgext.datahelpers to avoid having to manage the attachments. Using datahelpers also provides the advantage of having thumbnails support for free.

from tgext.datahelpers.fields import Attachment, AttachedImage
 
class Gallery(DeclarativeBase):
    __tablename__ = 'galleries'
 
   uid = Column(Integer, autoincrement=True, primary_key=True)
   name = Column(Unicode(100), nullable=False)
 
class Photo(DeclarativeBase):
    __tablename__ = 'photos'
 
    uid = Column(Integer, autoincrement=True, primary_key=True)
    name = Column(Unicode(100), nullable=False)
    description = Column(Unicode(2048), nullable=False)
    image = Column(Attachment(AttachedImage))
 
    author_id = Column(Integer, ForeignKey(model.User.user_id)))
    author = relation(app_model.User, backref=backref('photos'))
 
    gallery_id = Column(Integer, ForeignKey(Gallery.uid))
    gallery = relation(Gallery, backref=backref('photos', cascade='all, delete-orphan'))

Now to be able to start using our galleries we will have to provide a place where to view them and a gallery management controller to create and manage them. Viewing them should be quite straightforward, I’ll just retrieve the galleries from the database inside my index method and render them. To access a single gallery I’ll rely on the datahelpers SQLAEntityConverter which will retrieve the gallery for us ensuring it exists and is valid. For the management part I’ll create an EasyCrudRestController mounted as /manage_galleries

from tgext.crud import EasyCrudRestController
 
class GalleriesController(EasyCrudRestController):
    allow_only = predicates.in_group('photos')
    title = "Manage Galleries"
    model = model.Gallery
 
    __form_options__ = {
        '__hide_fields__' : ['uid'],
        '__omit_fields__' : ['photos']
    }
 
class RootController(BaseController):
    manage_galleries = GalleriesController(DBSession)
 
    @expose('photos.templates.index')
    def index(self, *args, **kw):
        galleries = DBSession.query(Gallery).order_by(Gallery.uid.desc()).all()
        return dict(galleries=galleries)
 
    @expose('photos.templates.gallery')
    @validate(dict(gallery=SQLAEntityConverter(Gallery)), error_handler=index)
    def gallery(self, gallery):
        return dict(gallery=gallery)

Logging in with an user inside the photos group and accessing the /manage_galleries url we will be able to create a new gallery and manage the existing ones.

To configure how the crud controller forms should appear and behave the __form_options__ property of the EasyCrudRestController can be used. This property relies on the same options as Sprox FormBase and customizes both the Edit and Add forms.
The next part is probably to be able to upload some photos inside our newly created galleries. To perform this we will create a new EasyCrudRestController for gallery photos management.

from tgext.crud import EasyCrudRestController
from tw.forms import FileField
from tw.forms.validators import FieldStorageUploadConverter
from webhelpers import html
 
class PhotosController(EasyCrudRestController):
    allow_only = predicates.in_group('photos')
    title = "Manage Photos"
    model = model.Photo
    keep_params = ['gallery']
 
    __form_options__ = {
        '__hide_fields__' : ['uid', 'author', 'gallery'],
        '__field_widget_types__' : {'image':FileField},
        '__field_validator_types__' : {'image':FieldStorageUploadConverter},
        '__field_widget_args__' : {'author':{'default':lambda:request.identity['user'].user_id}}
    }
 
    __table_options__ = {
        '__omit_fields__' : ['uid', 'author_id', 'gallery_id', 'gallery'],
        '__xml_fields__' : ['image'],
        'image': lambda filler,row: html.literal('‹img src="%s"/›' % row.image.thumb_url)
    }

Mounting this inside the RootController as manage_photos = PhotosController(DBSession) it will be possible to upload new photos inside any gallery. To manage the photos inside the first gallery for example we will have to access /manage_photos?gallery=1url.

Each parameter passed to the EasyCrudRestController is used to filter the entries to show inside the management table and the keep_params option provides a way to keep the filter around. This makes possible to edit the photos of only one gallery at the time instead of having all the photos mixed together. Also when a new photo is created it will be created in the current gallery.

The PhotosController got more customization than the GalleriesController, through the __field_widget_types__ and __field_validator_types__ options we force the image field to be a file field and using the __field_widget_args__ we ensure that the newly uploaded photos have the current user as the author.

__table_options__ provide a way to customize the management table. The available options are the same as the Sprox TableBase and Sprox TableFiller objects. in this case we hide the indexes of the rows on the database and the gallery itself, as we are managing the photos of a specific gallery we probably don’t need to know which galleries the photos belong to. Using the __xml_fields__ we also specify that the image field provides HTML and so doesn’t have to be escaped. The image entry forces the table to show the image thumbnail for the image column of the table instead of printing the AttachedImage.__repr__ as it would by default.

At first sight it might sound a bit complex, but once you start feeling confident, the CRUD extension makes possible to create entire applications in just a bunch of code lines. With just a few lines of code we created a photo gallery with multiple albums support and we can now focus on the index and gallery templates to make the gallery as pleasant as possible for our visitors.

The complete implementation of the photo gallery is available as a pluggable application on bitbucket, feel free to use it in your TurboGears projects.

Tags:

Our First Hackathon

puria | January 16th, 2012 | Opensource, Software Development | Comments

Axant codes for food

Axant codes for food

We are coders, and we love to code. Our job is coding, and we have fun to.

Hence we decided to have fun, just by having a 24h non-stop coding experiment in our preferred and beloved Italian restaurant (well actually a taverna)!

We will have a mini-hackathon tomorrow at 3pm our time, to celebrate together the past 2011 working year, that was great! Free Food, free alcohol and a couple of cots is our necessaire for brainstorm and actuate our plan to take over the world!

Actually this “coding marathon” it has a subject, all the projects and ideas that will be developed should be related to “Food / Dining”!

The products result of the day, if useful, will be developed on field and used by a restaurant and a couple of bars, free of charge. Hence if there are good result could be a first step for a new free-software project.

In fact probably tomorrow we’ll not have just software projects cause the team is heterogeneous, built of lawyers, designers, creative people, musicians… and obviously coders!

The plan includes a couple of hours to study the problematics of a restaurant and in particular of our host. The owner will illustrate to us his business processes and we will try to understand were we could help them with our notions.

The next step is to actuate the thoughts into reality in the next 20hrs (: that’s it, nothing easier huh?

In the next post I’ll update the outcomes and the people behind it!

Tags: , ,

TurboGears2 DebugBar

amol | November 12th, 2011 | Uncategorized | Comments

Recently some work has been done to extend the hooks support in TurboGears, to play a little with the new hooks I decided to try creating the famous and envied Django Debug Toolbar. I’m quite happy of the result and most of the features are there. In a few days I’ll be able to place it on a public repository and I’ll release it concurrently with the 2.1.4 release of TurboGears.

Debug Toolbar

Debug Toolbar

Timings

Timings

Request and Headers

Request and Headers

SQLAlchemy Queries

SQLAlchemy Queries

Mounted Controllers

Mounted Controllers

The code has been heavily inspired by the Pyramid Debug Toolbar and have to thank the Pyramid team for the good job they did at making the Toolbar code clean and simple.

Tags:

TurboGears2 Performance Improvements

amol | October 4th, 2011 | Uncategorized | Comments

As recently some effort has been involved in improving the performances of TurboGears2, I was curious to see how much things improved. As usually, the test isn’t really reliable in any way and was just for fun.

All the graphs report the request/sec the application has been able to perform on my computer with only 1 concurrent client. So higher is better.

Here is the comparison between TG2.0 and TG2dev (will be 2.1.4)

I also compared various setups with different template engines on TG2dev

The comparison happened on an application similar to the quickstarted one.
Actually as there is no database involved in this application the template engine impacts a lot and so was a good benchmark for the template engines themselves.

Tags: , , ,

ACRcms 0.3.5 released with libacr 0.7

amol | July 8th, 2011 | Uncategorized | Comments

New features introduced are:

  • Assets manager, Upload your videos and images using the Asset manager instead of the RDisk if you want to create slices from them. Assets manager will also create thumbnails for everything and will convert the videos to support HTML5
  • BlogPost view (an HTML view with a title)
  • Attributes for Pages and Slices
  • Hide Pages from menu with hidden=1 attribute
  • Reorder Pages in menu menu-weight attribute
  • Add metatag description to pages with description attribute
  • Automatic Database migrations, never care about changes to the schema when you upgrade acr again, it will be automatically upgraded by acr itself on first visit
  • Hide edit bars in edit mode to view the slice as a standard visitor
  • Abstraction layer for encoding data, now each view encodes using EncodedView, so that it will be possible to switch to any key/value encoding or store in the future

Tags:

Search