Category: programming


PyPlants — Now with added dimensions

PyPlants has come on in leaps and bounds over the past few days (well, even­ings), and now from its new home as PyPlants on bit­bucket sports a com­pletely rewrit­ten ren­der­ing backend which is more mod­u­lar, should be really easy to plug into, and now sup­ports POV-Ray out of the box.

What’s that you say? A 3-D ray-tracer? Yes indeed, as prom­ised in the second part of this series of devel­op­ment diar­ies, I’ve now fin­ished work on an update that turns this:

("A", "I+[A+O]-->>[--L]I[++L]-[AO]++AO")
("I", "FS[>>&&L][>>^^L]FS")
("S", "SFS")
("L", "['{+f-ff-f+|+f-ff-f}]")
("O", "[&&&C`>W>>>>W>>>>W>>>>W>>>>W]")
("C", "FF")
("W", "[`^F][{&&&&-f+f|-f+f}]")

Into this:

olive_bush

Rendering of an olive bush from pyplants pov­ray renderer

Unfortunately it’s now 2am, so the write-up will have to wait for the week­end. Do feel free to grab the code and have a poke around. You’ll obvi­ously need pov­ray, pygame, and pycairo installed, but everything else should work with python’s included batteries.

Procedural Plants in Python — Part 2

In the pre­vi­ous part of this art­icle we looked at the back­ground to L-Systems, and how they could be used for describ­ing self-similar bio­lo­gical sys­tems. In this part we’ll look at a sample imple­ment­a­tion of a very basic 2D L-System in Python, together with a basic PNG ren­derer using PyCairo.

This is an imple­ment­a­tion of a determ­in­istic, context-less L-System, which given the same ini­tial con­di­tions will always pro­duce the same res­ults. Whilst this won’t pro­duce very nat­ur­al­istic res­ults for our plant sys­tem, it’s a use­ful first step to pro­duce res­ults like this:

Test render of a D0 L-System

First, we’ll define some 2D operations:

  • F: Move for­ward LINE_LENGTH, draw­ing a line.
  • +: Rotate clock­wise THETA degrees.
  • -: Rotate anti-clockwise THETA degrees.
  • [: Push the cur­rent pos­i­tion and rota­tion onto a stack
  • ]: Pop the cur­rent pos­i­tion and rota­tion off the stack

PyCairo is a great lib­rary for vec­tor draw­ing oper­a­tions and very well suited to the draw­ing oper­a­tions we’ll be using. To use PyCairo, you simply declare a draw­ing sur­face (in this case a cairo.ImageSurface), and pass that to a con­text. All draw­ing oper­a­tions will then hap­pen on this context.

This sample imple­ment­a­tion is also avail­able on pastebin.

# this code is an improved version of the l-system
# sample code originally distributed with pycairo.
# originally Copyright 2003 Jesse Andrews (jdandr2 at uky.edu)
# this version Copyright 2009 Seb Potter (iamseb at iamseb.com)
# licensed under GPL

import logging
import cairo

# setup logging to just print to stdout.
# python's logging module is significantly
# better than using print for debugging
LOG_FILENAME = '/dev/stdout'
logging.basicConfig(filename=LOG_FILENAME,level=logging.DEBUG)

class Lindenmayer:
    """
    A class to represent L-Systems and render them using cairo
    """
    def __init__( self ):
        self.width = self.height = 500 # use a 500 pixel square image
        self.prod = {'[':'[','F':'F',']':']','+':'+','-':'-'} # the identity products
        self.start_pos = (self.width*0.5, self.height) # we translate to the middle bottom of the image to start
        self.start_angle = 180 # we rotate through 180 degrees to draw upwards - cairo's origin is top-left
        self.theta = 90 # the rotation angle in degrees
        self.stack = [] # this will be the stack for storing translation and orientation tuples
        self.str = 'f' # the starting string, or 'axiom'

        self.line_length = 5 # how far we move forward on each step
        self.line_width = 2 # just controls the rendered width of the line
        self.logger = logging.getLogger("flower.draw.lindenmayer") # use logging to print debugging
        self.logger.setLevel(logging.DEBUG)

    def addProd(self, let, prod):
        """
        Add a production to the ordered list of productions to apply to the current string.
        """
        self.prod[let]=prod

    def iterate(self, iterations=1):
        """
        Iterate over the list of productions and apply them to the string, in a loop.
        The end result is the final transformed string.
        """
        for i in xrange(iterations):
            self.str = ''.join([ self.prod[l] for l in self.str])

        self.logger.info("String is: %s" % self.str)

    def line(self, ctx, len):
        ctx.rel_line_to( 0, len )

    def rotate(self, ctx, deg):
        ctx.rotate( 2*3.141592653589793*deg/360.0  )

    def draw(self, colour):
        """Render the string representation of the L-System"""
        self.logger.info(colour)

        # create a cairo drawing context from the provided surface.
        surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, self.width, self.height)
        ctx = cairo.Context(surface)

        # first we'll create a nice white rectangle as our background
        ctx.rectangle(0, 0, self.width, self.height)
        ctx.set_source_rgb(1, 1, 1)
        ctx.fill()

        # set the line colour, width, and make sure that cairo knows how close
        # lines should be to join them
        ctx.set_source_rgb(*colour)
        ctx.set_line_width(self.line_width)
        ctx.set_tolerance(0.1)
        ctx.set_line_join(cairo.LINE_JOIN_BEVEL)

        # start drawing a path, move to our start point, and rotate to
        # the starting angle
        ctx.new_path()
        ctx.move_to(*self.start_pos)
        self.logger.debug("Initial position: %s, %s" % ctx.get_current_point())
        self.rotate(ctx, self.start_angle)

        # this is the very simple way we iterate over the final string
        # and perform a drawing operation for each symbol in the string
        for c in self.str:
            if c == 'F':
                # move forward
                self.logger.debug("f: draw a line of %s" % self.line_length)
                self.line(ctx, self.line_length)
            if c == '+':
                # rotate clockwise
                self.logger.debug("+: rotate %s" % self.theta)
                self.rotate(ctx, self.theta)
            if c == '-':
                # rotate anti-clockwise
                self.logger.debug("-: rotate -%s" % self.theta)
                self.rotate(ctx, -self.theta)
            if c == '[':
                # push the transform and orientation onto the stack
                m = ctx.get_matrix()
                p = ctx.get_current_point()
                self.logger.debug("[: push the matrix %s onto the stack" % m)
                self.stack.append((p, m))
            if c == ']':
                # restore the transform and orientation from the stack
                p, m = self.stack.pop()
                self.logger.debug("]: pop the matrix %s off the stack" % m)
                ctx.set_matrix(m)
                ctx.move_to(*p)

        # now draw the path created as a stroke on the context
        ctx.stroke()

        # write this to a png
        surface.write_to_png("test.png")
        self.logger.info("Wrote test.png")

def main():

    colour = (0, 0.3, 0)

    # setup the initial parameters for this l-system
    lin = Lindenmayer()
    lin.start_angle = 205
    lin.start_pos = (lin.width*0.25, lin.height)
    lin.str = 'X'
    lin.addProd('X', 'F[+X]F[-X]+X')
    lin.addProd('F', 'FF')
    lin.theta = 20

    # generate the final string
    lin.iterate(iterations=5)

    # render the string using pycairo
    lin.draw(colour)

if __name__ == '__main__':
    main()

In the third part of this series we’ll look at more real­istic approaches to describ­ing plant sys­tems, includ­ing intro­duc­tion of ran­dom­ness through Stochastic L-Systems, and ways to cre­ate more plant-like fea­tures includ­ing leaves and flowers.

Procedural Plants in Python — Part 1

For a fledgling pro­ject idea, I’ve recently needed to work out how to draw plants pro­ced­ur­ally, and of course Python is my lan­guage of choice for some rapid pro­to­typ­ing. Whilst some richly-featured pro­fes­sional applic­a­tions exist for gen­er­at­ing flora in a pro­ced­ural fash­ion for high-end ren­der­ing, there are pre­cious few sys­tems avail­able for the kind of bulk task that I require.

At first the task of gen­er­at­ing plants might seem like a massive endeav­our. The vari­ous per­muta­tions of stem widths, lengths, branch fre­quency, col­our vari­ation, leaf size, shape, fre­quency and mind-boggling. And that’s not even con­sid­er­ing the prob­lem space rep­res­en­ted by flowers.

Fortunately, the prob­lem of rep­res­ent­ing self-similar bio­lo­gical devel­op­ment is one that attrac­ted Aristid Lindenmayer, who in 1968 intro­duced L-Systems:

… as a the­or­et­ical frame­work for study­ing the devel­op­ment of simple mul­ti­cel­lu­lar organ­isms, and sub­sequently applied to invest­ig­ate higher plants and plant organs. After the incor­por­a­tion of geo­met­ric fea­tures, plant mod­els expressed using L-systems became detailed enough to allow the use of com­puter graph­ics for real­istic visu­al­iz­a­tion of plant struc­tures and devel­op­mental processes.

(Lindenmayer’s book The Algorithmic Beauty of Plants is no longer in print, but is avail­able for free in PDF.)

L-Systems are remark­ably power­ful, simple sys­tems that express the devel­op­ment of a formal gram­mar through a par­al­lel rewrit­ing pro­cess. In effect, you start with a string that rep­res­ents cer­tain graph­ical oper­a­tions, and a set of rules for rewrit­ing the string. You iter­ate over the string a cer­tain num­ber of times, and are left with a final product.

For example in a 2D sys­tem, the sym­bol F rep­res­ents draw­ing a straight line in a for­ward dir­ec­tion, the sym­bol + rep­res­ents rotat­ing left, and — rep­res­ents rotat­ing right. If you start with the string “F”, and a rule that says “(F → F+F−F−F+F)”, then you’d see the fol­low­ing iterations:

  1. F+F-F-F+F
  2. F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F
  3. F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F+ F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F– F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F– F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F+ F+F-F-F+F+F+F-F-F+F-F+F-F-F+F-F+F-F-F+F+F+F-F-F+F

If you then plot those rules using a unit vec­tor length and an angle of 90 degrees, you end up with the fol­low­ing iterations:

  1. Single iteration of a Koch Square
  2. Second iteration of a Koch Square
  3. Third iteration of a Koch Square

There are sev­eral more sym­bols that can be applied to the graph­ical oper­a­tions, includ­ing push­ing the cur­rent pos­i­tion and ori­ent­a­tion onto a stack and pop­ping it off again, rota­tions in 3D, and oper­a­tions to define closed poly­gons and mov­ing through col­our and size lookup tables.

The most basic type of L-System is the determ­in­istic context-less sys­tem, D0L, where each oper­a­tion is inde­pend­ent and will always pro­duce the same out­put. The Koch Square above is one of the most simplistic examples, but with a couple of vari­ations much more com­plex pat­terns arise:

  • Start: “X”
  • Rule 1: “(X → F-[[X]+X]+F[+FX]-X)”
  • Rule 2: “(F → FF)”
  • Angle: 25 degrees
  • Iterations: 5
  • Sample 1

In Part 2 we’ll look at a first approach to imple­ment­ing L-Systems in Python, and a rudi­ment­ary ren­derer for 2D sys­tems using PyCairo.

Ten Things I’ve Learned in a Start-Up

Ten things I’ve learned as a developer at WooMe:

  1. Be clear what your dir­ec­tion is. Identify your key pro­pos­i­tion and focus on it. Don’t get dis­trac­ted by unre­lated fea­tures — some­body else is prob­ably already doing those bet­ter than you.
  2. Be mer­ci­less with fea­tures that don’t cut it. No mat­ter how much you like it, if your audi­ence doesn’t like it, either change it or kill it fast. Wasting time on your per­sonal pet pro­ject when there’s no clear demand for it is time you should be spend­ing on fea­tures users want.
  3. Use what’s already there rather than invent­ing your own. Need a mes­sage queue? Video pro­cessing? Load bal­an­cer? Email dis­tri­bu­tion? Payment sys­tem? Use some­thing that already exists rather than waste time you should be spend­ing on fea­ture development.
  4. Quick and dirty is bet­ter than late to mar­ket. At the rate this industry moves, cap­tur­ing your audience’s atten­tion first is more import­ant than hav­ing a well-engineered solu­tion. For the most part, users don’t care about your code qual­ity, as long as it works.
  5. Don’t optim­ise until you need to. Sure, you may end up chas­ing your tail for a bit when you need to scale, but until you hit that point you’re wast­ing valu­able time that should be spent on fea­ture development.
  6. Scaling is most often a data prob­lem. When you do need to optim­ise, scal­ing won’t be about the per­form­ance of your applic­a­tion. Scaling web applic­a­tions hori­zont­ally is easy. Scaling data­bases ver­tic­ally is expens­ive. Scaling data­bases hori­zont­ally is very, very hard. Don’t worry about what web applic­a­tion frame­work you use, worry about what data­base you put behind it.
  7. Abstraction is great, as long as you under­stand what you’re abstract­ing. This has been a con­stant pain with Django’s ORM. Yes, it’s a power­ful and use­ful abstrac­tion, but without a solid under­stand­ing of what the data­base is doing behind it, it’s easy to get into very deep trouble, very quickly.
  8. Log everything. Nothing is harder than try­ing to find prob­lems on a live envir­on­ment without detailed logging.
  9. Storage is cheap. It’s bet­ter to store everything than to throw away data. You never know when you might need something.
  10. Communication is crit­ical. Not just out­side your organ­isa­tion, but within it. Tell people what you’re doing, fre­quently. Ask them what they’re doing just as often. Not only do ideas per­col­ate through this pro­cess much more effect­ively, but when things go wrong, clear and fre­quent com­mu­nic­a­tion will stop them from get­ting worse.

Tactile Computing

Every now and then someone at TED presents a tech­no­logy or an idea that’s so utterly amaz­ing, or ridicu­lously simple that it can’t help but change the world. David Merrill shows off an MIT pro­ject called Siftables in this talk, and even though I’ve been mess­ing around with com­put­ing for 25 years my jaw is still drag­ging along the floor. Check it out.

Bingo We Can Believe In

Alcohol and polit­ics are two things that should rarely mix, but once in a blue moon there’s cause to throw cau­tion to the wind and get a little crazy.

One such occa­sion is tomor­row night, when the US elect­or­ate takes to the polls and — with a little luck — elects Barack Obama their next President.

There are all sorts of reas­ons why this is such an import­ant elec­tion, both in terms of America’s cur­rent stand­ing within the world com­munity, and a num­ber of immin­ent domestic prob­lems that the next President will find him­self deal­ing with prob­ably before he’s even inaugurated.

That’s per­haps a post or two for another time; once the res­ults are in and the res­id­ents of Awesome Manor have either cel­eb­rated or com­mis­er­ated until they drop.

For now, I simply offer my humble con­tri­bu­tion to the elec­tion fest­iv­it­ies. I call it “Bingo We Can Believe In”.

Django Decorator for View Rate Limits

This is a handy little snip­pet that allows you to set rate lim­its for views in Django.

Notes From a Small Internet

Beautiful Soup is a great little Python mod­ule that will read just about any HTML page and give you back a struc­tured parsed tree. It’s awe­some because you can pass it just about any mangled markup — I’ve never known it to choke on any­thing. For some web ser­vice con­sumers I’ve had to write over the years Beautiful Soup has saved me many, many hours of slog­ging through crappy HTML pars­ing. Great soft­ware deserves appreciation.

Whilst brows­ing my good friend Rachel’s web­site I happened to notice that her brother Leonard wrote Beautiful Soup. He also wrote RESTful Web Services, which is part of my (recently pruned) dead tree col­lec­tion, and which I’d heart­ily recom­mend to any­one who has to work with REST web ser­vices. The Django examples were espe­cially useful!

Google’s AppEngine Beat Me To It

Recently I’ve been put­ting some time into writ­ing a data­base adapter for Django that uses Amazon’s S3 and SimpleDB ser­vices as a stor­age layer, whilst try­ing to retain as much of Django’s QuerySet func­tional layer as pos­sible. The gen­eral goal is to provide a stor­age back-end for Django that isn’t depend­ent on the tra­di­tional vertically-scaling data­base server, but can scale hori­zont­ally in the same way as the EC2 com­put­ing cloud does. My even­tual goal being the abil­ity to deploy Django in the cloud with no external depend­en­cies. Just throw out a Django machine image, deploy your app’s code and con­fig, and you have a scal­ing solu­tion that takes minutes rather than days or weeks.

It’s a non-trivial exer­cise that is both stim­u­lat­ing and frus­trat­ing in equal meas­ure, and pro­gress has been steady, if not exactly rapid. It’s worth it to me though, as the abil­ity to roll out scal­ing infra­struc­ture is dra­mat­ic­ally hampered by the data­base layer.

Imagine my delight then to find that Google have launched AppEngine, their own cloud-based web applic­a­tion sys­tem. It’s Python without any messy machine-based lib­rar­ies, uses WSGI so you can use pretty much any Python web app, and with GFS for a dis­trib­uted file stor­age and BigTable as a data per­sist­ence layer. Google even throws in Django 0.96.1 with instruc­tions on how to use their stor­age lay­ers by doing away with Django’s own model  (more on this later).

There’s a lot of whin­ing about how Google’s solu­tion cripples Python (which is crazy when you look at how trivial it is to refactor code to use Google’s sup­plied altern­at­ives), and locks you into their solu­tion. I sus­pect that this is mostly from people who have never even con­tem­plated build­ing an applic­a­tion that needs to really scale, and are there­fore still think­ing in terms ser­vices provided by the under­ly­ing OS. That’s a big prob­lem for scal­ing, because disk, IO, threads, sock­ets, etc are finite resources that are hardware-bound. Abstracting access to these things is tough. Most scal­ing solu­tions these days are about provid­ing mul­tiple hard­ware instances, but unfor­tu­nately that only solves the hard­ware prob­lem. Building an app that scales trans­par­ently over mul­tiple hard­ware instances is a huge chal­lenge in com­par­ison to pro­cur­ing more servers.

Google’s approach is to do away with the concept of hard­ware entirely. That means a change of mind­set towards every request being an atomic oper­a­tion. Persistence occurs (cor­rectly) in your per­sist­ence layer and not in tran­si­ent stor­age avail­able to an instance of your applic­a­tion. Google have provided extens­ive Python lib­rar­ies and API calls to enable applic­a­tions to take advant­age of this, but it seems that a fairly vocal group aren’t inter­ested unless their applic­a­tions work on AppEngine without any addi­tional effort. Considering the paradigm shift that AppEngine rep­res­ents (from machine-centric pro­gram­ming to dis­trib­uted pro­gram­ming) it’s not unreas­on­able to expect some small effort to be required. Especially when you take into account that AppEngine is cur­rently in a very lim­ited trial phase.

I’m extremely optim­istic that Google’s approach will work well for a num­ber of reas­ons. As an applic­a­tion pro­gram­mer I spend huge amounts of time work­ing around hard­ware and plat­form lim­it­a­tions that I should be spend­ing on core func­tional areas. If Google can provide a solu­tion that means I never have to worry about spe­cific hard­ware prob­lems ever again, I doubt I’ll look back.

iPlayer Flash — Embedded (with a little work)

The BBC has just released Flash stream­ing for iPlayer’s 7 Day Catch-Up fea­ture. This is just about the best news to come out of the fiasco that is iPlayer, and for those of us unwill­ing or unable to install the bandwidth-destroying Kontiki cli­ent it’s the only way we can get our on-demand BBC programming.

Unfortunately, the BBC only makes the streams avail­able on their site. I checked the iPlayer terms and con­di­tions, and there’s noth­ing stop­ping remote embed­ding, it’s just a tech­nical hurdle.

Fortunately, the inter­net eats tech­nical hurdles for breakfast.

So, in the half hour before I go out to the pub, I’ve knocked together a little WordPress plu­gin that will take an iPlayer URI, and embed the stream­ing video into your page for you.

Like so:

Streaming full-screen Doctor Who on my web­site. Excuse me whilst I geek-out a moment, this is the coolest thing I’ve played with all week!

I’ll tidy up the code and release it to the com­munity as soon as I get back from a night of drink­ing and dancing!

Powered by WordPress | Theme: Motion by 85ideas.