I spent some time today learning about the Build System. In addition to the built-in build systems for various languages, and the possibility of defining custom build systems at the user preference level, you can also specify project build systems.

I added the following stanza to one of my .sublime-project files:

"build_systems":
 [
     {
         "name": "unittest-discover",
         "shell_cmd": "~/.virtualenvs/randopony-tetra-2.7/bin/python
                       -m unittest discover ${project_path}"
     }
 ]

to enable me to run the test suite for the project within Sublime at the touch of a key.

Things to note:

• The name element defines how the build will appear in the Tools > Build System menu.
• In the shell_cmd element:
  • I use an explicit path to the python interpreter in the project's virtualenv so that the project's dependencies are found correctly.
  • The shell that the Sublime build command launches will have as its cwd the directory that the file you initiate the build command from is in. ${project_path} is a build system variable that points to the directory where the .sublime-project file is, and I use that to tell unittest discover where to start searching for tests.
• You can define as many build systems as you want in the .sublime-project file, just be sure to give them different names.

After reloading the .sublime-project file, choose unittest-discover from the Tools > Build System menu, and launch a build (⌘B on OS/X), and voilà, the test suite runs in a pane that pops up at the bottom of the Sublime window.

Georges Dubus has created pyramid_persona, a nice library to integrate Mozilla Persona authentication into Pyramid projects. He also wrote a tutorial blog post explaining the basics of how to use the library. In principle it should be easy to go from what Georges has written to getting Persona-based authentication fully integrated with group or object level authorization in a Pyramid project, but it took some head scratching for me to do that. What better reason to blog about what I came up with and build on Georges' excellent work...

By the way, pyramid_persona is also a really nice example of advanced configuration tactics in Pyramid that show off the framework's plug-in friendliness.

Before diving is, I should also mention that Michael Merickel's authentication and authorization tutorial is an indispensable reference.

The Goal: The example I'm going to use is a very simple group-level authorization one. We'll create a site with a public side that anyone can interact with, and a private side that only allows access by users who have authenticated via Persona, and are known to the app as members of the admin group. We'll use the alchemy scaffold, so the app will have URL mapping via URL dispatch and persistence via SQLAlchemy.

So, let's get started.

Create a virtualenv, install Pyramid, and create an alchemy scaffold app:

(ppga)$ pcreate -s alchemy pyramid_persona_group_auth_demo

Add pyramid_persona to the requires list in setup.py, and add some settings to development.ini:

persona.secret = some secret string
persona.audiences = http://localhost:6543
persona.siteName = Pyramid Persona Group Auth Demo

mako.directories = pyramid_persona_group_auth_demo:templates

The persona settings are described in the Configuration section of the pyramid_persona README. The mako.directories setting is needed because we're going to use Mako for our login and admin page templates.

Next up, we add an Administrator class to our models.py file:

class Administrator(Base):
    __tablename__ = 'admins'
    id = Column(Integer, primary_key=True)
    persona_email = Column(Text, index=True, unique=True)

    def __init__(self, persona_email):
        self.persona_email = persona_email

We also need to bootstrap our Persona email address into the database so that we can sign-in to our app when we first start it up. To do that, import Administrator into scripts/initializedb.py and add a couple of lines to the end of the main function in that file:

from ..models import Administrator
...
with transaction.manager:
    model = MyModel(name='one', value=1)
    DBSession.add(model)
    admin = Administrator(persona_email='me@example.com')
    DBSession.add(admin)

Obviously, you'll want to use an email address that you have registered with Persona in place of me@example.com.

Now it's time to create some templates and views for the admin site of our site. These pages are almost verbatim from Georges' blog post or the pyramid_persona README - he really has done most of the heavy lifting! We start with a stub for the admin home page in templates/admin_home.mako:

<!DOCTYPE html>
<html>
<head>
  <script type="text/javascript"
    src="//ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js">
  </script>
  <script src="https://login.persona.org/include.js"></script>
  <script >${request.persona_js}</script>
</head>
<body>
  <h1>Pyramid Persona Group Auth Demo Admin</h1>
  Welcome to the admin side, ${userid}
  ${request.persona_button}
</body>
</html>

All we've done here is hooked into Persona, and set up a welcome message and a sign-out button in lieu of any real admin interface content and funtionality.

The request.person_js and request.persona_button attributes are available thanks to config.set_request_property() calls in pyramid_persona.__init__.py. Recall that I mentioned what a nice example of advanced Pyramid configuration pyramid_persona is - it's well worth reading Georges' code to learn how he has made things so easy for the rest of us.

We'll also create a very similar looking template for our sign-in page in templates/admin_signin.mako:

<!DOCTYPE html>
<html>
<head>
  <script type="text/javascript"
    src="//ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js">
  </script>
  <script src="https://login.persona.org/include.js"></script>
  <script >${request.persona_js}</script>
</head>
<body>
<h1>Pyramid Persona Group Auth Demo Admin</h1>
  ${request.persona_button}
</body>
</html>

There are obviously some opportunities to use template inheritance here, to say nothing of the total absence of styling in those templates.

Moving along to the views that render those templates, we create admin_views.py containing:

from pyramid.renderers import render
from pyramid.response import Response
from pyramid.security import authenticated_userid
from pyramid.view import (
    forbidden_view_config,
    view_config,
    )

@forbidden_view_config()
def admin_signin(request):
    body = render('admin_signin.mako', {}, request=request)
    return Response(body, status='403 Forbidden')

@view_config(
    route_name='admin.home',
    renderer='admin_home.mako',
    permission='admin')
def admin_home(request):
    userid = authenticated_userid(request)
    return {'userid': userid}

The admin_signin function renders our admin_signin.mako template and packages it up as a 403 Forbidden response. That function is decorated with @forbidden_view_config() so that it will replace the default forbidden view that pyramid_persona supplies. That's how we get our sign-in page to appear when an unauthenticated user hits the admin URL.

The admin_home function renders our admin home page stub. The important thing to note here is the permission='admin' argument in the @view_config decorator.

Finally, we add our security policy to pyramid_persona_group_auth_demo/__init__.py and add our admin home page view to the route map. We need some new imports:

from pyramid.authentication import AuthTktAuthenticationPolicy
from pyramid.security import ALL_PERMISSIONS
from pyramid.security import Allow
from sqlalchemy.orm.exc import NoResultFound
from .models import Administrator

Then we add a dirt simple resource tree to provide the access control list that implements our security policy; i.e. any user in the admin group has full access to admin resources:

class Root(object):
    """Simplest possible resource tree to map groups to permissions.
    """
    __acl__ = [
        (Allow, 'g:admin', ALL_PERMISSIONS),
    ]

    def __init__(self, request):
        self.request = request

We also need a groupfinder function to provide to the authentication policy constructor so that it can identify whether or not an authenticated user is in the admin group:

def groupfinder(userid, request):
    query = DBSession.query(Administrator).\
                filter(Administrator.persona_email == userid)
    try:
        query.one()
        return ['g:admin']
    except NoResultFound:
        return None

Now let's hook everything together via the configuration in the main function. We add our resource tree to the Configurator constructor call:

config = Configurator(settings=settings, root_factory=Root)

as well as including the pyramid_persona configuration, and overriding its authentication policy setting with our own that hooks in the groupfinder function:

config.include('pyramid_persona')
authn_policy = AuthTktAuthenticationPolicy(
    settings['persona.secret'],
    callback=groupfinder)
config.set_authentication_policy(authn_policy)

Last but not least, we add a route to map /admin to our admin home page:

config.add_route('admin.home', '/admin')

It's time to test!

Install our app to get all of the dependencies installed, initialize the database with the handy initialize_pyramid_persona_group_auth_demo_db command that Pyramid creates for us during installation of an alchemy scaffold app, and start the server:

(ppga)$ python setup.py develop
(ppga)$ # lots of output
(ppga)$ initialize_pyramid_persona_group_auth_demo_db development.ini
(ppga)$ pserve --reload development.ini

Browsing to http://localhost:6543 should show you the default Pyramid app page - that's the public side of our site.

Going to http://localhost:6543/admin should show you the admin sign-in page:

and, if you inspect the requests and responses with your browser's web dev tools, you'll see that we got the expected 403 Forbidden response status.

Clicking the sign-in button pops the login.persona.org site in a new window with our site name and audience domain displayed:

And, upon successfully signing in at Persona, we are redirected to the admin home page of our site:

And that's all there is to it! Extending the site security to a more fine-grained group-level hierachy, or to object-level security should be relatively easy with the guidance in Michael Merickel's authentication and authorization tutorial.

The source code for the demo I've created here is available on Bitbucket at https://bitbucket.org/douglatornell/pyramid_persona_group_auth_demo

Thanks again to Georges Dubus for making this easy with the pyramid_persona library, and to Mozilla for creating Persona.

Updated 2012-11-02

• Fixed typo in import statement for Administrator model. Thanks ppaez!
• Added index on persona_email column of Administrator model. Thanks Georges Dubus!

I've done some work recently to get the blogofile plugins branch working under Python 2.6, 2.7 and 3.2 from single codebase. The blogofile serve command runs an HTTP server on localhost to let you check the results of your blogofile build before you deploy it to the waiting world. In Python 3 (in which the plugins branch was initially implemented) the relevant bits of the implementation go like:

import http.server
import threading

class Server(threading.Thread):
    def __init__(self, port, address):
        HandlerClass = BlogofileRequestHandler
        ServerClass = http.server.HTTPServer
        self.httpd = ServerClass(('127.0.0.1', 8080), HandlerClass)

    def run(self):
        self.httpd.serve_forever()

    def shutdown(self):
        self.httpd.shutdown()
        self.httpd.socket.close()

class BlogofileRequestHandler(http.server.SimpleHTTPRequestHandler):
    def __init__(self, *args, **kwargs):
        http.server.SimpleHTTPRequestHandler.__init__(
            self, *args, **kwargs)

    def translate_path(self, path):
        ...

Some import acrobatics are required to get that to also run under Python 2:

try:
    from http.server import HTTPServer as http_server
except ImportError:
    from SocketServer import TCPServer as http_server
try:
    from http.server import SimpleHTTPRequestHandler \
        as http_handler
except ImportError:
    from SimpleHTTPServer import SimpleHTTPRequestHandler \
        as http_handler
import threading

class Server(threading.Thread):
    def __init__(self, port, address):
        HandlerClass = BlogofileRequestHandler
        ServerClass = http_server
        self.httpd = ServerClass(('127.0.0.1', 8080), HandlerClass)

    def run(self):
        self.httpd.serve_forever()

    def shutdown(self):
        self.httpd.shutdown()
        self.httpd.socket.close()

class BlogofileRequestHandler(http_handler):
    def __init__(self, *args, **kwargs):
        http_handler.__init__(self, *args, **kwargs)

    def translate_path(self, path):
        ...

The six library provides a nice way to clean up the hard to read try:... except ImportError:... blocks:

import threading
from six.moves import SimpleHTTPServer
from six.moves import socketserver

http_server = socketserver.TCPServer
http_handler = SimpleHTTPServer.SimpleHTTPRequestHandler

class Server(threading.Thread):
    def __init__(self, port, address):
        HandlerClass = BlogofileRequestHandler
        ServerClass = http_server
        self.httpd = ServerClass(('127.0.0.1', 8080), HandlerClass)

    def run(self):
        self.httpd.serve_forever()

    def shutdown(self):
        self.httpd.shutdown()
        self.httpd.socket.close()

class BlogofileRequestHandler(http_handler):
    def __init__(self, *args, **kwargs):
        http_handler.__init__(self, *args, **kwargs)

    def translate_path(self, path):
        ...

Note that this implementation steps back from the Python 3 http.server.HTTPServer as the server class to the underlying socketserver.TCPServer (or SocketServer.TCPServer in Python 2).

I finally got around to fixing a couple of minor annoyances I have with Blogofile. These fixes apply to the plugins development branch of Blogofile and the Blogofile_blog plugin, but they should be easily backport-able to the Blogofile master branch. I opened pull requests for these changes on Github and I'm happy to report that @EnigmaCurry merged the Blogofile_blog ones within hours! But since Blogofile development and this blog have been languishing for a while, I figured I should write about the changes here.

Using Python 2.7 and 3.2 with the PYTHONWARNINGS environment variable set to default reveals that both Blogofile and the Blogofile_blog plugin raise ResourceWarning exceptions when the blogofile build command is run. Admittedly, this is a really minor issue, but seeing a screen full of tracebacks every time I build my site is annoying, and it can obscure more serious problems. Those warnings are easily silenced by changing the offending open statements to use with statement context managers. The fix for Blogofile is in pull request 119 and the one for Blogofile_blog is in pull request 7.

The blogofile blog create post command creates a file with the extension .markdown by default but Blogofile also supports RST and Textile markup. I use RST and really want my newly created post files to have the extension .rst so that emacs goes to rst-mode automatically when I open a post file for editing. Again, a minor annoyance, and my fix was easily implemented. I chose to add a blog.post.default_markup config option. With blog.post.default_markup = 'rst' in my site's _config.py file my new posts get the .rst extension I want. If blog.post.default_markup is not set the created post file extension defaults to .markdown as before. This feature is in pull request 8.

I really like Blogofile and using it's plugins branch was my spur to get serious about using Python 3 and blogging again. So, I'm really happy to see @EnigmaCurry accepting pull requests again on the project. Hooray!!

One of the users of an intranet app I maintain was using copy/paste to put data from the app into Excel. She asked if there was a better way. The pages she was copying from have tables of datastore objects, so adding a CSV download feature was an obvious solution. Providing that feature turned out to be pretty easy with the help of the StringIO and csv modules in the standard library.

While the code below is from a Pylons controller class, I can't see it being difficult to implement this in Django, Pyramid, or other web framework stacks.

The request handler for the CSV download looks like:

import cString
import csv

def csv_download(self, ...):
    csv_buffer = cStringIO.StringIO()
    csv_writer = csv.writer(csv_buffer)
    header = self._build csv_header(...)
    csv_writer.writerow(header)
    query_result = self.get_data_for_csv(...)
    for result in query_result:
        row = self._build_csv_row(result)
        csv_writer.writerow(row)
    content = csv_buffer.getvalue()
    csv_buffer.close()
    response.content_type = 'text/csv; charset=utf-8'
    response.content_disposition = (
        'attachment; filename="your_file.csv"')
    return content

This method uses StringIO to set up a file-like memory buffer, and instantiates a default CSV writer to write to the buffer. Next we build the header row and write it to the buffer. Then we get an iterator for the content that we want to write the the CSV file from the datastore, and format and write it to the buffer, one line at a time. Finally we get the CSV data from the buffer, set the response headers appropriately, and return the CSV data for download.

One thing I'm uncertain about: Is it necessary to explicitly call close method on a StringIO instance, or could I just do:

return csv_buffer.getvalue()

and let garbage collection take care of releasing the memory allocated for csv_buffer?

To get Excel to play nice with UTF-8 encoded data it's necessary to include 3 specific bytes as a Byte Order Mark (BOM) at the beginning of the file. I did that by prepending them to the heading string for the first column:

def _build_csv_header(self, ...)
    UTF_8_BOM = '\xef\xbb\xbf'
    header = [
        UTF_8_BOM + 'Column 1 Heading',
        ...
    ]
    return header

Building the content for each row of the CSV file is just a matter of formatting each query result into an array of strings. Fields containing non-ASCII characters stored as Unicode need to be encoded to UTF-8:

def _build_csv_row(self, result)
    row = [
        result.column_1_value,
        '{:%Y-%m-%d}'.format(result.some_date)
        ...
        result.unicode_value.encode('utf-8'),
        ...
    ]
    return row

I had an additional complication to deal with. The data for one of the CSV columns is stored in the database as HTML fragments that may contain non-ASCII characters. That data had to be rendered to Unicode before it could be added to the CSV row (encoded as UTF-8). It turns out that the Python standard library provides a fairly painless way of handling that complication too, but I'll leave that for another post.