I have a Django project called RandoPony that handles event registration for the BC Randonneurs Cycling Club. It's on an annual release cycle; i.e. I spend the few weeks that pass for winter in Vancouver updating the project. That's when I bump it to the latest version of Django, fixing minor bugs, and adding new features that I and other users have come up with during the preceding year. Once I release a new version for the new year, I usually don't have to worry about the code until the next winter. The pony just works, facilitating people doing hundreds of thousands of kilometres of crazy long cycling events, and we like it that way!

My workflow at the beginning of the annual update looks something like:

• Create a new virtualenv
• Install the latest version of Django and other project dependencies
• Read the release notes for the Django releases since the one I was working with last
• Run the RandoPony test suite to find deprecations and other obvious breakage
• Start hacking

I recently started working on the 2012 release of RandoPony and was blown away when I ran the test suite because there were over 60 failing tests! It took me way longer than it should have to figure out why things were so massively broken.

The problem was that the test fixtures weren't being installed. They weren't being installed because they are YAML files and I had forgotten to install PyYAML in the virtualenv. What's really annoying is that the fixtures files were being ignored silently.

It turns out that if you specify a YAML fixture for a Django TestCase:

class TestPopulairesListView(django.test.TestCase):
    """Functional tests for populaires-list view.
    """
    fixtures = ['populaires']

without giving the fixture file a .yaml extension, the fixture will be silently ignored if PyYAML isn't installed. Really, Django?!

So, the number 1 thing that I should have done to save myself from this thrash was to explicitly specify the serialization format of my fixtures:

class TestPopulairesListView(django.test.TestCase):
    """Functional tests for populaires-list view.
    """
    fixtures = ['populaires.yaml']

Then the Django test runner would have told me:

Problem installing fixture 'populaires': yaml is not a known
serialization format.

I'll take the hit for ignoring the PEP 20 aphorism "Explicit is better than implicit". But shouldn't Django get docked for "Errors should never pass silently"?

The other thing I should have done was use a pip requirements file for the project.

RandoPony has 2 requirements files now. requirements.txt for the packages required for the production deployment, and requirements-dev.txt for the additional packages, like PyYAML, required for development work. Now I just have to hope that I'm smart enough when I start work on the 2013 release to do:

(randopony)$ pip install -r requirements.txt
(randopony)$ pip install -r requirements-dev.txt

Stayin' Alive

MySQL Connection Timeout and SQLAlchemy Connection Pool Recycling

Recently a Pyramid web app surprised me with some odd behaviour. The app uses SQLAlchemy to interface with a MySQL database. The first request of the day would always fail with a traceback that ended with some sort of database connection failure, though not always the same error. This was happening in the development environment where I was the only one sending requests to the server. Hitting refresh, or sending another request resulted in the expected response, and things would continue to work that way - until the next morning...

The subtitle gives lots of hints about what was going on, so if you're still reading at this point I'll assume that you're still as puzzled as I was.

It took a surprising number of tries on Google before I found this section of the SQLAlchemy docs which explains all. To summarize, the default configuration of MySQL drops connections on which there has been no activity for 8 hours. SQLAlchemy provides the pool_recycle parameter for its engine creation functions as a way of working around that behaviour. (Although, as noted in the SQLAlchemy Engine creation API docs full description of pool_recycle, the behaviour is also configurable at the MySQLDB connection, and database configuration levels too.)

Since my Pyramid app uses the sqlalchemy.engine_from_config function, all I had to do was add:

sqlalchemy.pool_recycle = 3600

to my development.ini and production.ini config files, and ... problem solved.

P.S. Sorry - I couldn't resist the Bee Gees reference in the title. It's an age thing...

Following Tarek Ziade's lead:

1. What’s the coolest Python application, framework or library you have discovered in 2011?

   It's a toss-up between Pyramid and requests.

   For the admittedly small, specialized application that I've used Pyramid for, I like how the framework interface is largely confined to the __init__.py module. Outside of that I find myself just writing Python code to get stuff done and throwing in a few decorators here and there to link into the framework. I find the Pyramid docs to be really well organized, informative, and complete. They worked for me at the introductory level (though I did arrive with quite a bit of experience of other Python web frameworks) and continue to work as I dig deeper and do more advanced things.

   Requests has made my life so much better in several projects, whether its collecting data from well structured web services or scraping hydrometric data from a particularly annoying Government of Canada site.

2. What new programming technique did you learn in 2011?

   I got a lot better at writing simple, clean, uncoupled unit tests, inspired in large part by the Pyramid unit testing guidelines. I shifted from being a skeptic to a proponent of mocking thanks to the mock library. I was spurred in that by the need to refactor a test suite that had become way too slow to be useful.

   I also got a lot more proficient in JavaScript, using it for client-side stuff in web apps, database views for CouchDB, and a Firefox add-on.

3. What’s the name of the open source project you contributed the most in 2011? What did you do?

   CouchDBkit. I spent most of my time at the PyCon 2011 sprints adding the SetProperty and LazySet class to store Python sets as lists of unique elements in CouchDB. I also added some missing Python list methods to the LazyList class. It was a good learning experience in the realm of subclassing Python builtins. It was also cool to work sitting beside @benoitc (the CouchDBkit lead developer) in contrast to communicating electronically across 9 time zones.

4. What was the Python blog or website you read the most in 2011?

   The Planet Python feed, by far.

5. What are the three top things you want to learn in 2012?

   • Message passing systems. I've got a Django project that needs to get some asynchronicity and I'm planning to explore Celery for that.
   • More message passing systems. I've got some devops issues around server synchronization and failover that I think I might be able to address with ZeroMQ.
   • Python 3. Because it's time! Most of the libraries that I use have already been ported, so the transition shouldn't be overwhelmingly difficult, and perhaps I can lend a hand porting some of the libraries that I need that haven't made the jump yet.

6. What are the top software, app or lib you wish someone would write in 2012?

   I wish there were more open source libraries and applications in the home automation realm. One of my first personal Python projects was a wrapper around heyu to make our house look lived in when we are away. It's still working okay after nearly 6 years, but the X10 hardware it interfaces with is getting rather long in the tooth. Sadly, newer home automation hardware and protocols seem to be mired in the muck of "vendor associations" that only pay lip service to openness.

Want to do your own list ? here’s how:

• copy-paste the questions and answer to them in your blog
• tweet it with the #2012pythonmeme hashtag

I decided to give myself a new blog for Christmas. All of my other blogs are based on b2evolution. It has served me well for years but I don't use much of its functionality, and hate dealing with its comment spam issues so much that I've turned off comments on all of those blogs. Most of all though, I want to use reStructuredText to write blog posts, just like I use it to write the vast majority of other content and docs that I create.

Blogofile caught my eye a while ago, not just because it easy supports reStructuredText markup, but also because:

• it simplifies the blogging workflow to:
  • write in your favourite editor
  • build
  • upload
• it gets rid of the overhead of a blogging app and database backend
• it fits nicely into a version control workflow
• it encourages delegation of comments, etc. (and the related spam issues) to Disqus