I Am Seb

At WooMe, we’ve been using Django and Postgresql in anger for coming up on three years, scaling up to tens of thousands of queries per second means we’ve had to look long and hard at Django’s database support to meet our needs.

Our performance tweaks have run the gamut from simple queryset changes, through changing Django’s transactional behaviour, right the way up to writing our own table partitioning and master/slave replication support long before multidb support hit Django trunk.

Of biggest impact so far was rewriting Django’s transactional layer to allow database transactions to commit automatically and run with PgBouncer as a connection pooler.

(more…)

The last few weeks I’ve been experimenting with different Django setups, which virtualenv makes trivial. (If you’re not using virtualenv and virtualenvwrapper for Python development, the secret is to bang the rocks together guys!) My goal has been to create a reusable project framework for quick social experiments that provides the following features:

• External authentication, to exploit as wide an audience as possible without the hurdle of requiring registration.
• Useful debugging information, both through logging and in-page debug.
• Database schema versioning and migration.
• Asynchronous task queuing and control.
• A lightweight WSGI HTTP server to sit between Django and Nginx.
• Straightforward deployment.
• Messaging, so that users can communicate with one another.
• Friends / social groups, so that users know who they want to communicate with.
• A rewards system, to reinforce and drive positive activity.

(more…)

PyPlants has come on in leaps and bounds over the past few days (well, evenings), and now from its new home as PyPlants on bitbucket sports a completely rewritten rendering backend which is more modular, should be really easy to plug into, and now supports POV-Ray out of the box.

What’s that you say? A 3-D ray-tracer? Yes indeed, as promised in the second part of this series of development diaries, I’ve now finished work on an update that turns this:

("A", "I+[A+O]-->>[--L]I[++L]-[AO]++AO")
("I", "FS[>>&&L][>>^^L]FS")
("S", "SFS")
("L", "['{+f-ff-f+|+f-ff-f}]")
("O", "[&&&C`>W>>>>W>>>>W>>>>W>>>>W]")
("C", "FF")
("W", "[`^F][{&&&&-f+f|-f+f}]")

Into this:

Rendering of an olive bush from pyplants povray renderer

Unfortunately it’s now 2am, so the write-up will have to wait for the weekend. Do feel free to grab the code and have a poke around. You’ll obviously need povray, pygame, and pycairo installed, but everything else should work with python’s included batteries.

In the previous part of this article we looked at the background to L-Systems, and how they could be used for describing self-similar biological systems. In this part we’ll look at a sample implementation of a very basic 2D L-System in Python, together with a basic PNG renderer using PyCairo.

This is an implementation of a deterministic, context-less L-System, which given the same initial conditions will always produce the same results. Whilst this won’t produce very naturalistic results for our plant system, it’s a useful first step to produce results like this:

(more…)

For a fledgling project idea, I’ve recently needed to work out how to draw plants procedurally, and of course Python is my language of choice for some rapid prototyping. Whilst some richly-featured professional applications exist for generating flora in a procedural fashion for high-end rendering, there are precious few systems available for the kind of bulk task that I require.

At first the task of generating plants might seem like a massive endeavour. The various permutations of stem widths, lengths, branch frequency, colour variation, leaf size, shape, frequency and mind-boggling. And that’s not even considering the problem space represented by flowers.

Fortunately, the problem of representing self-similar biological development is one that attracted Aristid Lindenmayer, who in 1968 introduced L-Systems:

… as a theoretical framework for studying the development of simple multicellular organisms, and subsequently applied to investigate higher plants and plant organs. After the incorporation of geometric features, plant models expressed using L-systems became detailed enough to allow the use of computer graphics for realistic visualization of plant structures and developmental processes.

(Lindenmayer’s book The Algorithmic Beauty of Plants is no longer in print, but is available for free in PDF.)

L-Systems are remarkably powerful, simple systems that express the development of a formal grammar through a parallel rewriting process. In effect, you start with a string that represents certain graphical operations, and a set of rules for rewriting the string. You iterate over the string a certain number of times, and are left with a final product.

For example in a 2D system, the symbol F represents drawing a straight line in a forward direction, the symbol + represents rotating left, and — represents rotating right. If you start with the string “F”, and a rule that says “(F → F+F−F−F+F)”, then you’d see the following iterations:

1. F+F-F-F+F
2. F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F
3. F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F+ F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F– F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F– F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F+ F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F

If you then plot those rules using a unit vector length and an angle of 90 degrees, you end up with the following iterations:

There are several more symbols that can be applied to the graphical operations, including pushing the current position and orientation onto a stack and popping it off again, rotations in 3D, and operations to define closed polygons and moving through colour and size lookup tables.

The most basic type of L-System is the deterministic context-less system, D0L, where each operation is independent and will always produce the same output. The Koch Square above is one of the most simplistic examples, but with a couple of variations much more complex patterns arise:

• Start: “X”
• Rule 1: “(X → F-[[X]+X]+F[+FX]-X)”
• Rule 2: “(F → FF)”
• Angle: 25 degrees
• Iterations: 5
• 

In Part 2 we’ll look at a first approach to implementing L-Systems in Python, and a rudimentary renderer for 2D systems using PyCairo.

This is a handy little snippet that allows you to set rate limits for views in Django.

Beautiful Soup is a great little Python module that will read just about any HTML page and give you back a structured parsed tree. It’s awesome because you can pass it just about any mangled markup — I’ve never known it to choke on anything. For some web service consumers I’ve had to write over the years Beautiful Soup has saved me many, many hours of slogging through crappy HTML parsing. Great software deserves appreciation.

Whilst browsing my good friend Rachel’s website I happened to notice that her brother Leonard wrote Beautiful Soup. He also wrote RESTful Web Services, which is part of my (recently pruned) dead tree collection, and which I’d heartily recommend to anyone who has to work with REST web services. The Django examples were especially useful!

Recently I’ve been putting some time into writing a database adapter for Django that uses Amazon’s S3 and SimpleDB services as a storage layer, whilst trying to retain as much of Django’s QuerySet functional layer as possible. The general goal is to provide a storage back-end for Django that isn’t dependent on the traditional vertically-scaling database server, but can scale horizontally in the same way as the EC2 computing cloud does. My eventual goal being the ability to deploy Django in the cloud with no external dependencies. Just throw out a Django machine image, deploy your app’s code and config, and you have a scaling solution that takes minutes rather than days or weeks.

It’s a non-trivial exercise that is both stimulating and frustrating in equal measure, and progress has been steady, if not exactly rapid. It’s worth it to me though, as the ability to roll out scaling infrastructure is dramatically hampered by the database layer.

Imagine my delight then to find that Google have launched AppEngine, their own cloud-based web application system. It’s Python without any messy machine-based libraries, uses WSGI so you can use pretty much any Python web app, and with GFS for a distributed file storage and BigTable as a data persistence layer. Google even throws in Django 0.96.1 with instructions on how to use their storage layers by doing away with Django’s own model  (more on this later).

There’s a lot of whining about how Google’s solution cripples Python (which is crazy when you look at how trivial it is to refactor code to use Google’s supplied alternatives), and locks you into their solution. I suspect that this is mostly from people who have never even contemplated building an application that needs to really scale, and are therefore still thinking in terms services provided by the underlying OS. That’s a big problem for scaling, because disk, IO, threads, sockets, etc are finite resources that are hardware-bound. Abstracting access to these things is tough. Most scaling solutions these days are about providing multiple hardware instances, but unfortunately that only solves the hardware problem. Building an app that scales transparently over multiple hardware instances is a huge challenge in comparison to procuring more servers.

Google’s approach is to do away with the concept of hardware entirely. That means a change of mindset towards every request being an atomic operation. Persistence occurs (correctly) in your persistence layer and not in transient storage available to an instance of your application. Google have provided extensive Python libraries and API calls to enable applications to take advantage of this, but it seems that a fairly vocal group aren’t interested unless their applications work on AppEngine without any additional effort. Considering the paradigm shift that AppEngine represents (from machine-centric programming to distributed programming) it’s not unreasonable to expect some small effort to be required. Especially when you take into account that AppEngine is currently in a very limited trial phase.

I’m extremely optimistic that Google’s approach will work well for a number of reasons. As an application programmer I spend huge amounts of time working around hardware and platform limitations that I should be spending on core functional areas. If Google can provide a solution that means I never have to worry about specific hardware problems ever again, I doubt I’ll look back.

I’m somewhat of a fan of Python, using it as I do for almost everything I do in programming, so it’s great to see other people appreciating the language.

Randall Munroe, however, might be getting a little… carried away: