What’s that you say? A 3-D ray-tracer? Yes indeed, as promised in the second part of this series of development diaries, I’ve now finished work on an update that turns this:

("A", "I+[A+O]-->>[--L]I[++L]-[AO]++AO")
("I", "FS[>>&&L][>>^^L]FS")
("S", "SFS")
("L", "['{+f-ff-f+|+f-ff-f}]")
("O", "[&&&C`>W>>>>W>>>>W>>>>W>>>>W]")
("C", "FF")
("W", "[`^F][{&&&&-f+f|-f+f}]")

Into this:

Rendering of an olive bush from pyplants povray renderer

Unfortunately it’s now 2am, so the write-up will have to wait for the weekend. Do feel free to grab the code and have a poke around. You’ll obviously need povray, pygame, and pycairo installed, but everything else should work with python’s included batteries.

This is an implementation of a deterministic, context-less L-System, which given the same initial conditions will always produce the same results. Whilst this won’t produce very naturalistic results for our plant system, it’s a useful first step to produce results like this:

First, we’ll define some 2D operations:

• F: Move forward LINE_LENGTH, drawing a line.
• +: Rotate clockwise THETA degrees.
• -: Rotate anti-clockwise THETA degrees.
• [: Push the current position and rotation onto a stack
• ]: Pop the current position and rotation off the stack

PyCairo is a great library for vector drawing operations and very well suited to the drawing operations we’ll be using. To use PyCairo, you simply declare a drawing surface (in this case a cairo.ImageSurface), and pass that to a context. All drawing operations will then happen on this context.

This sample implementation is also available on pastebin.

# this code is an improved version of the l-system
# sample code originally distributed with pycairo.
# originally Copyright 2003 Jesse Andrews (jdandr2 at uky.edu)
# this version Copyright 2009 Seb Potter (iamseb at iamseb.com)
# licensed under GPL

import logging
import cairo

# setup logging to just print to stdout.
# python's logging module is significantly
# better than using print for debugging
LOG_FILENAME = '/dev/stdout'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG)

class Lindenmayer:
    """
    A class to represent L-Systems and render them using cairo
    """
    def __init__( self ):
        self.width = self.height = 500 # use a 500 pixel square image
        self.prod = {'[':'[','F':'F',']':']','+':'+','-':'-'} # the identity products
        self.start_pos = (self.width*0.5, self.height) # we translate to the middle bottom of the image to start
        self.start_angle = 180 # we rotate through 180 degrees to draw upwards - cairo's origin is top-left
        self.theta = 90 # the rotation angle in degrees
        self.stack = [] # this will be the stack for storing translation and orientation tuples
        self.str = 'f' # the starting string, or 'axiom'

        self.line_length = 5 # how far we move forward on each step
        self.line_width = 2 # just controls the rendered width of the line
        self.logger = logging.getLogger("flower.draw.lindenmayer") # use logging to print debugging
        self.logger.setLevel(logging.DEBUG)

    def addProd(self, let, prod):
        """
        Add a production to the ordered list of productions to apply to the current string.
        """
        self.prod[let]=prod

    def iterate(self, iterations=1):
        """
        Iterate over the list of productions and apply them to the string, in a loop.
        The end result is the final transformed string.
        """
        for i in xrange(iterations):
            self.str = ''.join([ self.prod[l] for l in self.str])

        self.logger.info("String is: %s" % self.str)

    def line(self, ctx, len):
        ctx.rel_line_to( 0, len )

    def rotate(self, ctx, deg):
        ctx.rotate( 2*3.141592653589793*deg/360.0  )

    def draw(self, colour):
        """Render the string representation of the L-System"""
        self.logger.info(colour)

        # create a cairo drawing context from the provided surface.
        surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, self.width, self.height)
        ctx = cairo.Context(surface)

        # first we'll create a nice white rectangle as our background
        ctx.rectangle(0, 0, self.width, self.height)
        ctx.set_source_rgb(1, 1, 1)
        ctx.fill()

        # set the line colour, width, and make sure that cairo knows how close
        # lines should be to join them
        ctx.set_source_rgb(*colour)
        ctx.set_line_width(self.line_width)
        ctx.set_tolerance(0.1)
        ctx.set_line_join(cairo.LINE_JOIN_BEVEL)

        # start drawing a path, move to our start point, and rotate to
        # the starting angle
        ctx.new_path()
        ctx.move_to(*self.start_pos)
        self.logger.debug("Initial position: %s, %s" % ctx.get_current_point())
        self.rotate(ctx, self.start_angle)

        # this is the very simple way we iterate over the final string
        # and perform a drawing operation for each symbol in the string
        for c in self.str:
            if c == 'F':
                # move forward
                self.logger.debug("f: draw a line of %s" % self.line_length)
                self.line(ctx, self.line_length)
            if c == '+':
                # rotate clockwise
                self.logger.debug("+: rotate %s" % self.theta)
                self.rotate(ctx, self.theta)
            if c == '-':
                # rotate anti-clockwise
                self.logger.debug("-: rotate -%s" % self.theta)
                self.rotate(ctx, -self.theta)
            if c == '[':
                # push the transform and orientation onto the stack
                m = ctx.get_matrix()
                p = ctx.get_current_point()
                self.logger.debug("[: push the matrix %s onto the stack" % m)
                self.stack.append((p, m))
            if c == ']':
                # restore the transform and orientation from the stack
                p, m = self.stack.pop()
                self.logger.debug("]: pop the matrix %s off the stack" % m)
                ctx.set_matrix(m)
                ctx.move_to(*p)

        # now draw the path created as a stroke on the context
        ctx.stroke()

        # write this to a png
        surface.write_to_png("test.png")
        self.logger.info("Wrote test.png")

def main():

    colour = (0, 0.3, 0)

    # setup the initial parameters for this l-system
    lin = Lindenmayer()
    lin.start_angle = 205
    lin.start_pos = (lin.width*0.25, lin.height)
    lin.str = 'X'
    lin.addProd('X', 'F[+X]F[-X]+X')
    lin.addProd('F', 'FF')
    lin.theta = 20

    # generate the final string
    lin.iterate(iterations=5)

    # render the string using pycairo
    lin.draw(colour)

if __name__ == '__main__':
    main()

In the third part of this series we’ll look at more realistic approaches to describing plant systems, including introduction of randomness through Stochastic L-Systems, and ways to create more plant-like features including leaves and flowers.

At first the task of generating plants might seem like a massive endeavour. The various permutations of stem widths, lengths, branch frequency, colour variation, leaf size, shape, frequency and mind-boggling. And that’s not even considering the problem space represented by flowers.

Fortunately, the problem of representing self-similar biological development is one that attracted Aristid Lindenmayer, who in 1968 introduced L-Systems:

… as a theoretical framework for studying the development of simple multicellular organisms, and subsequently applied to investigate higher plants and plant organs. After the incorporation of geometric features, plant models expressed using L-systems became detailed enough to allow the use of computer graphics for realistic visualization of plant structures and developmental processes.

(Lindenmayer’s book The Algorithmic Beauty of Plants is no longer in print, but is available for free in PDF.)

L-Systems are remarkably powerful, simple systems that express the development of a formal grammar through a parallel rewriting process. In effect, you start with a string that represents certain graphical operations, and a set of rules for rewriting the string. You iterate over the string a certain number of times, and are left with a final product.

For example in a 2D system, the symbol F represents drawing a straight line in a forward direction, the symbol + represents rotating left, and — represents rotating right. If you start with the string “F”, and a rule that says “(F → F+F−F−F+F)”, then you’d see the following iterations:

1. F+F-F-F+F
2. F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F
3. F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F+ F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F– F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F– F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F+ F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F

If you then plot those rules using a unit vector length and an angle of 90 degrees, you end up with the following iterations:

There are several more symbols that can be applied to the graphical operations, including pushing the current position and orientation onto a stack and popping it off again, rotations in 3D, and operations to define closed polygons and moving through colour and size lookup tables.

The most basic type of L-System is the deterministic context-less system, D0L, where each operation is independent and will always produce the same output. The Koch Square above is one of the most simplistic examples, but with a couple of variations much more complex patterns arise:

• Start: “X”
• Rule 1: “(X → F-[[X]+X]+F[+FX]-X)”
• Rule 2: “(F → FF)”
• Angle: 25 degrees
• Iterations: 5
• 

In Part 2 we’ll look at a first approach to implementing L-Systems in Python, and a rudimentary renderer for 2D systems using PyCairo.

1. Be clear what your direction is. Identify your key proposition and focus on it. Don’t get distracted by unrelated features — somebody else is probably already doing those better than you.
2. Be merciless with features that don’t cut it. No matter how much you like it, if your audience doesn’t like it, either change it or kill it fast. Wasting time on your personal pet project when there’s no clear demand for it is time you should be spending on features users want.
3. Use what’s already there rather than inventing your own. Need a message queue? Video processing? Load balancer? Email distribution? Payment system? Use something that already exists rather than waste time you should be spending on feature development.
4. Quick and dirty is better than late to market. At the rate this industry moves, capturing your audience’s attention first is more important than having a well-engineered solution. For the most part, users don’t care about your code quality, as long as it works.
5. Don’t optimise until you need to. Sure, you may end up chasing your tail for a bit when you need to scale, but until you hit that point you’re wasting valuable time that should be spent on feature development.
6. Scaling is most often a data problem. When you do need to optimise, scaling won’t be about the performance of your application. Scaling web applications horizontally is easy. Scaling databases vertically is expensive. Scaling databases horizontally is very, very hard. Don’t worry about what web application framework you use, worry about what database you put behind it.
7. Abstraction is great, as long as you understand what you’re abstracting. This has been a constant pain with Django’s ORM. Yes, it’s a powerful and useful abstraction, but without a solid understanding of what the database is doing behind it, it’s easy to get into very deep trouble, very quickly.
8. Log everything. Nothing is harder than trying to find problems on a live environment without detailed logging.
9. Storage is cheap. It’s better to store everything than to throw away data. You never know when you might need something.
10. Communication is critical. Not just outside your organisation, but within it. Tell people what you’re doing, frequently. Ask them what they’re doing just as often. Not only do ideas percolate through this process much more effectively, but when things go wrong, clear and frequent communication will stop them from getting worse.

One such occasion is tomorrow night, when the US electorate takes to the polls and — with a little luck — elects Barack Obama their next President.

There are all sorts of reasons why this is such an important election, both in terms of America’s current standing within the world community, and a number of imminent domestic problems that the next President will find himself dealing with probably before he’s even inaugurated.

That’s perhaps a post or two for another time; once the results are in and the residents of Awesome Manor have either celebrated or commiserated until they drop.

For now, I simply offer my humble contribution to the election festivities. I call it “Bingo We Can Believe In”.

Whilst browsing my good friend Rachel’s website I happened to notice that her brother Leonard wrote Beautiful Soup. He also wrote RESTful Web Services, which is part of my (recently pruned) dead tree collection, and which I’d heartily recommend to anyone who has to work with REST web services. The Django examples were especially useful!

It’s a non-trivial exercise that is both stimulating and frustrating in equal measure, and progress has been steady, if not exactly rapid. It’s worth it to me though, as the ability to roll out scaling infrastructure is dramatically hampered by the database layer.

Imagine my delight then to find that Google have launched AppEngine, their own cloud-based web application system. It’s Python without any messy machine-based libraries, uses WSGI so you can use pretty much any Python web app, and with GFS for a distributed file storage and BigTable as a data persistence layer. Google even throws in Django 0.96.1 with instructions on how to use their storage layers by doing away with Django’s own model  (more on this later).

There’s a lot of whining about how Google’s solution cripples Python (which is crazy when you look at how trivial it is to refactor code to use Google’s supplied alternatives), and locks you into their solution. I suspect that this is mostly from people who have never even contemplated building an application that needs to really scale, and are therefore still thinking in terms services provided by the underlying OS. That’s a big problem for scaling, because disk, IO, threads, sockets, etc are finite resources that are hardware-bound. Abstracting access to these things is tough. Most scaling solutions these days are about providing multiple hardware instances, but unfortunately that only solves the hardware problem. Building an app that scales transparently over multiple hardware instances is a huge challenge in comparison to procuring more servers.

Google’s approach is to do away with the concept of hardware entirely. That means a change of mindset towards every request being an atomic operation. Persistence occurs (correctly) in your persistence layer and not in transient storage available to an instance of your application. Google have provided extensive Python libraries and API calls to enable applications to take advantage of this, but it seems that a fairly vocal group aren’t interested unless their applications work on AppEngine without any additional effort. Considering the paradigm shift that AppEngine represents (from machine-centric programming to distributed programming) it’s not unreasonable to expect some small effort to be required. Especially when you take into account that AppEngine is currently in a very limited trial phase.

I’m extremely optimistic that Google’s approach will work well for a number of reasons. As an application programmer I spend huge amounts of time working around hardware and platform limitations that I should be spending on core functional areas. If Google can provide a solution that means I never have to worry about specific hardware problems ever again, I doubt I’ll look back.

Unfortunately, the BBC only makes the streams available on their site. I checked the iPlayer terms and conditions, and there’s nothing stopping remote embedding, it’s just a technical hurdle.

Fortunately, the internet eats technical hurdles for breakfast.

So, in the half hour before I go out to the pub, I’ve knocked together a little WordPress plugin that will take an iPlayer URI, and embed the streaming video into your page for you.

Like so:

Streaming full-screen Doctor Who on my website. Excuse me whilst I geek-out a moment, this is the coolest thing I’ve played with all week!

I’ll tidy up the code and release it to the community as soon as I get back from a night of drinking and dancing!