I spent some time today learning about the Build System. In addition to the built-in build systems for various languages, and the possibility of defining custom build systems at the user preference level, you can also specify project build systems.

I added the following stanza to one of my .sublime-project files:

"build_systems":
 [
     {
         "name": "unittest-discover",
         "shell_cmd": "~/.virtualenvs/randopony-tetra-2.7/bin/python
                       -m unittest discover ${project_path}"
     }
 ]

to enable me to run the test suite for the project within Sublime at the touch of a key.

Things to note:

• The name element defines how the build will appear in the Tools > Build System menu.
• In the shell_cmd element:
  • I use an explicit path to the python interpreter in the project's virtualenv so that the project's dependencies are found correctly.
  • The shell that the Sublime build command launches will have as its cwd the directory that the file you initiate the build command from is in. ${project_path} is a build system variable that points to the directory where the .sublime-project file is, and I use that to tell unittest discover where to start searching for tests.
• You can define as many build systems as you want in the .sublime-project file, just be sure to give them different names.

After reloading the .sublime-project file, choose unittest-discover from the Tools > Build System menu, and launch a build (⌘B on OS/X), and voilà, the test suite runs in a pane that pops up at the bottom of the Sublime window.

Georges Dubus has created pyramid_persona, a nice library to integrate Mozilla Persona authentication into Pyramid projects. He also wrote a tutorial blog post explaining the basics of how to use the library. In principle it should be easy to go from what Georges has written to getting Persona-based authentication fully integrated with group or object level authorization in a Pyramid project, but it took some head scratching for me to do that. What better reason to blog about what I came up with and build on Georges' excellent work...

By the way, pyramid_persona is also a really nice example of advanced configuration tactics in Pyramid that show off the framework's plug-in friendliness.

Before diving is, I should also mention that Michael Merickel's authentication and authorization tutorial is an indispensable reference.

The Goal: The example I'm going to use is a very simple group-level authorization one. We'll create a site with a public side that anyone can interact with, and a private side that only allows access by users who have authenticated via Persona, and are known to the app as members of the admin group. We'll use the alchemy scaffold, so the app will have URL mapping via URL dispatch and persistence via SQLAlchemy.

So, let's get started.

Create a virtualenv, install Pyramid, and create an alchemy scaffold app:

(ppga)$ pcreate -s alchemy pyramid_persona_group_auth_demo

Add pyramid_persona to the requires list in setup.py, and add some settings to development.ini:

persona.secret = some secret string
persona.audiences = http://localhost:6543
persona.siteName = Pyramid Persona Group Auth Demo

mako.directories = pyramid_persona_group_auth_demo:templates

The persona settings are described in the Configuration section of the pyramid_persona README. The mako.directories setting is needed because we're going to use Mako for our login and admin page templates.

Next up, we add an Administrator class to our models.py file:

class Administrator(Base):
    __tablename__ = 'admins'
    id = Column(Integer, primary_key=True)
    persona_email = Column(Text, index=True, unique=True)

    def __init__(self, persona_email):
        self.persona_email = persona_email

We also need to bootstrap our Persona email address into the database so that we can sign-in to our app when we first start it up. To do that, import Administrator into scripts/initializedb.py and add a couple of lines to the end of the main function in that file:

from ..models import Administrator
...
with transaction.manager:
    model = MyModel(name='one', value=1)
    DBSession.add(model)
    admin = Administrator(persona_email='me@example.com')
    DBSession.add(admin)

Obviously, you'll want to use an email address that you have registered with Persona in place of me@example.com.

Now it's time to create some templates and views for the admin site of our site. These pages are almost verbatim from Georges' blog post or the pyramid_persona README - he really has done most of the heavy lifting! We start with a stub for the admin home page in templates/admin_home.mako:

<!DOCTYPE html>
<html>
<head>
  <script type="text/javascript"
    src="//ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js">
  </script>
  <script src="https://login.persona.org/include.js"></script>
  <script >${request.persona_js}</script>
</head>
<body>
  <h1>Pyramid Persona Group Auth Demo Admin</h1>
  Welcome to the admin side, ${userid}
  ${request.persona_button}
</body>
</html>

All we've done here is hooked into Persona, and set up a welcome message and a sign-out button in lieu of any real admin interface content and funtionality.

The request.person_js and request.persona_button attributes are available thanks to config.set_request_property() calls in pyramid_persona.__init__.py. Recall that I mentioned what a nice example of advanced Pyramid configuration pyramid_persona is - it's well worth reading Georges' code to learn how he has made things so easy for the rest of us.

We'll also create a very similar looking template for our sign-in page in templates/admin_signin.mako:

<!DOCTYPE html>
<html>
<head>
  <script type="text/javascript"
    src="//ajax.googleapis.com/ajax/libs/jquery/1.8.2/jquery.min.js">
  </script>
  <script src="https://login.persona.org/include.js"></script>
  <script >${request.persona_js}</script>
</head>
<body>
<h1>Pyramid Persona Group Auth Demo Admin</h1>
  ${request.persona_button}
</body>
</html>

There are obviously some opportunities to use template inheritance here, to say nothing of the total absence of styling in those templates.

Moving along to the views that render those templates, we create admin_views.py containing:

from pyramid.renderers import render
from pyramid.response import Response
from pyramid.security import authenticated_userid
from pyramid.view import (
    forbidden_view_config,
    view_config,
    )

@forbidden_view_config()
def admin_signin(request):
    body = render('admin_signin.mako', {}, request=request)
    return Response(body, status='403 Forbidden')

@view_config(
    route_name='admin.home',
    renderer='admin_home.mako',
    permission='admin')
def admin_home(request):
    userid = authenticated_userid(request)
    return {'userid': userid}

The admin_signin function renders our admin_signin.mako template and packages it up as a 403 Forbidden response. That function is decorated with @forbidden_view_config() so that it will replace the default forbidden view that pyramid_persona supplies. That's how we get our sign-in page to appear when an unauthenticated user hits the admin URL.

The admin_home function renders our admin home page stub. The important thing to note here is the permission='admin' argument in the @view_config decorator.

Finally, we add our security policy to pyramid_persona_group_auth_demo/__init__.py and add our admin home page view to the route map. We need some new imports:

from pyramid.authentication import AuthTktAuthenticationPolicy
from pyramid.security import ALL_PERMISSIONS
from pyramid.security import Allow
from sqlalchemy.orm.exc import NoResultFound
from .models import Administrator

Then we add a dirt simple resource tree to provide the access control list that implements our security policy; i.e. any user in the admin group has full access to admin resources:

class Root(object):
    """Simplest possible resource tree to map groups to permissions.
    """
    __acl__ = [
        (Allow, 'g:admin', ALL_PERMISSIONS),
    ]

    def __init__(self, request):
        self.request = request

We also need a groupfinder function to provide to the authentication policy constructor so that it can identify whether or not an authenticated user is in the admin group:

def groupfinder(userid, request):
    query = DBSession.query(Administrator).\
                filter(Administrator.persona_email == userid)
    try:
        query.one()
        return ['g:admin']
    except NoResultFound:
        return None

Now let's hook everything together via the configuration in the main function. We add our resource tree to the Configurator constructor call:

config = Configurator(settings=settings, root_factory=Root)

as well as including the pyramid_persona configuration, and overriding its authentication policy setting with our own that hooks in the groupfinder function:

config.include('pyramid_persona')
authn_policy = AuthTktAuthenticationPolicy(
    settings['persona.secret'],
    callback=groupfinder)
config.set_authentication_policy(authn_policy)

Last but not least, we add a route to map /admin to our admin home page:

config.add_route('admin.home', '/admin')

It's time to test!

Install our app to get all of the dependencies installed, initialize the database with the handy initialize_pyramid_persona_group_auth_demo_db command that Pyramid creates for us during installation of an alchemy scaffold app, and start the server:

(ppga)$ python setup.py develop
(ppga)$ # lots of output
(ppga)$ initialize_pyramid_persona_group_auth_demo_db development.ini
(ppga)$ pserve --reload development.ini

Browsing to http://localhost:6543 should show you the default Pyramid app page - that's the public side of our site.

Going to http://localhost:6543/admin should show you the admin sign-in page:

and, if you inspect the requests and responses with your browser's web dev tools, you'll see that we got the expected 403 Forbidden response status.

Clicking the sign-in button pops the login.persona.org site in a new window with our site name and audience domain displayed:

And, upon successfully signing in at Persona, we are redirected to the admin home page of our site:

And that's all there is to it! Extending the site security to a more fine-grained group-level hierachy, or to object-level security should be relatively easy with the guidance in Michael Merickel's authentication and authorization tutorial.

The source code for the demo I've created here is available on Bitbucket at https://bitbucket.org/douglatornell/pyramid_persona_group_auth_demo

Thanks again to Georges Dubus for making this easy with the pyramid_persona library, and to Mozilla for creating Persona.

Updated 2012-11-02

• Fixed typo in import statement for Administrator model. Thanks ppaez!
• Added index on persona_email column of Administrator model. Thanks Georges Dubus!

I've done some work recently to get the blogofile plugins branch working under Python 2.6, 2.7 and 3.2 from single codebase. The blogofile serve command runs an HTTP server on localhost to let you check the results of your blogofile build before you deploy it to the waiting world. In Python 3 (in which the plugins branch was initially implemented) the relevant bits of the implementation go like:

import http.server
import threading

class Server(threading.Thread):
    def __init__(self, port, address):
        HandlerClass = BlogofileRequestHandler
        ServerClass = http.server.HTTPServer
        self.httpd = ServerClass(('127.0.0.1', 8080), HandlerClass)

    def run(self):
        self.httpd.serve_forever()

    def shutdown(self):
        self.httpd.shutdown()
        self.httpd.socket.close()

class BlogofileRequestHandler(http.server.SimpleHTTPRequestHandler):
    def __init__(self, *args, **kwargs):
        http.server.SimpleHTTPRequestHandler.__init__(
            self, *args, **kwargs)

    def translate_path(self, path):
        ...

Some import acrobatics are required to get that to also run under Python 2:

try:
    from http.server import HTTPServer as http_server
except ImportError:
    from SocketServer import TCPServer as http_server
try:
    from http.server import SimpleHTTPRequestHandler \
        as http_handler
except ImportError:
    from SimpleHTTPServer import SimpleHTTPRequestHandler \
        as http_handler
import threading

class Server(threading.Thread):
    def __init__(self, port, address):
        HandlerClass = BlogofileRequestHandler
        ServerClass = http_server
        self.httpd = ServerClass(('127.0.0.1', 8080), HandlerClass)

    def run(self):
        self.httpd.serve_forever()

    def shutdown(self):
        self.httpd.shutdown()
        self.httpd.socket.close()

class BlogofileRequestHandler(http_handler):
    def __init__(self, *args, **kwargs):
        http_handler.__init__(self, *args, **kwargs)

    def translate_path(self, path):
        ...

The six library provides a nice way to clean up the hard to read try:... except ImportError:... blocks:

import threading
from six.moves import SimpleHTTPServer
from six.moves import socketserver

http_server = socketserver.TCPServer
http_handler = SimpleHTTPServer.SimpleHTTPRequestHandler

class Server(threading.Thread):
    def __init__(self, port, address):
        HandlerClass = BlogofileRequestHandler
        ServerClass = http_server
        self.httpd = ServerClass(('127.0.0.1', 8080), HandlerClass)

    def run(self):
        self.httpd.serve_forever()

    def shutdown(self):
        self.httpd.shutdown()
        self.httpd.socket.close()

class BlogofileRequestHandler(http_handler):
    def __init__(self, *args, **kwargs):
        http_handler.__init__(self, *args, **kwargs)

    def translate_path(self, path):
        ...

Note that this implementation steps back from the Python 3 http.server.HTTPServer as the server class to the underlying socketserver.TCPServer (or SocketServer.TCPServer in Python 2).

I finally got around to fixing a couple of minor annoyances I have with Blogofile. These fixes apply to the plugins development branch of Blogofile and the Blogofile_blog plugin, but they should be easily backport-able to the Blogofile master branch. I opened pull requests for these changes on Github and I'm happy to report that @EnigmaCurry merged the Blogofile_blog ones within hours! But since Blogofile development and this blog have been languishing for a while, I figured I should write about the changes here.

Using Python 2.7 and 3.2 with the PYTHONWARNINGS environment variable set to default reveals that both Blogofile and the Blogofile_blog plugin raise ResourceWarning exceptions when the blogofile build command is run. Admittedly, this is a really minor issue, but seeing a screen full of tracebacks every time I build my site is annoying, and it can obscure more serious problems. Those warnings are easily silenced by changing the offending open statements to use with statement context managers. The fix for Blogofile is in pull request 119 and the one for Blogofile_blog is in pull request 7.

The blogofile blog create post command creates a file with the extension .markdown by default but Blogofile also supports RST and Textile markup. I use RST and really want my newly created post files to have the extension .rst so that emacs goes to rst-mode automatically when I open a post file for editing. Again, a minor annoyance, and my fix was easily implemented. I chose to add a blog.post.default_markup config option. With blog.post.default_markup = 'rst' in my site's _config.py file my new posts get the .rst extension I want. If blog.post.default_markup is not set the created post file extension defaults to .markdown as before. This feature is in pull request 8.

I really like Blogofile and using it's plugins branch was my spur to get serious about using Python 3 and blogging again. So, I'm really happy to see @EnigmaCurry accepting pull requests again on the project. Hooray!!

I have a Django project called RandoPony that handles event registration for the BC Randonneurs Cycling Club. It's on an annual release cycle; i.e. I spend the few weeks that pass for winter in Vancouver updating the project. That's when I bump it to the latest version of Django, fixing minor bugs, and adding new features that I and other users have come up with during the preceding year. Once I release a new version for the new year, I usually don't have to worry about the code until the next winter. The pony just works, facilitating people doing hundreds of thousands of kilometres of crazy long cycling events, and we like it that way!

My workflow at the beginning of the annual update looks something like:

• Create a new virtualenv
• Install the latest version of Django and other project dependencies
• Read the release notes for the Django releases since the one I was working with last
• Run the RandoPony test suite to find deprecations and other obvious breakage
• Start hacking

I recently started working on the 2012 release of RandoPony and was blown away when I ran the test suite because there were over 60 failing tests! It took me way longer than it should have to figure out why things were so massively broken.

The problem was that the test fixtures weren't being installed. They weren't being installed because they are YAML files and I had forgotten to install PyYAML in the virtualenv. What's really annoying is that the fixtures files were being ignored silently.

It turns out that if you specify a YAML fixture for a Django TestCase:

class TestPopulairesListView(django.test.TestCase):
    """Functional tests for populaires-list view.
    """
    fixtures = ['populaires']

without giving the fixture file a .yaml extension, the fixture will be silently ignored if PyYAML isn't installed. Really, Django?!

So, the number 1 thing that I should have done to save myself from this thrash was to explicitly specify the serialization format of my fixtures:

class TestPopulairesListView(django.test.TestCase):
    """Functional tests for populaires-list view.
    """
    fixtures = ['populaires.yaml']

Then the Django test runner would have told me:

Problem installing fixture 'populaires': yaml is not a known
serialization format.

I'll take the hit for ignoring the PEP 20 aphorism "Explicit is better than implicit". But shouldn't Django get docked for "Errors should never pass silently"?

The other thing I should have done was use a pip requirements file for the project.

RandoPony has 2 requirements files now. requirements.txt for the packages required for the production deployment, and requirements-dev.txt for the additional packages, like PyYAML, required for development work. Now I just have to hope that I'm smart enough when I start work on the 2013 release to do:

(randopony)$ pip install -r requirements.txt
(randopony)$ pip install -r requirements-dev.txt