I Am Seb

At WooMe, we’ve been using Django and Postgresql in anger for coming up on three years, scaling up to tens of thousands of queries per second means we’ve had to look long and hard at Django’s database support to meet our needs.

Our performance tweaks have run the gamut from simple queryset changes, through changing Django’s transactional behaviour, right the way up to writing our own table partitioning and master/slave replication support long before multidb support hit Django trunk.

Of biggest impact so far was rewriting Django’s transactional layer to allow database transactions to commit automatically and run with PgBouncer as a connection pooler.

(more…)

The last few weeks I’ve been experimenting with different Django setups, which virtualenv makes trivial. (If you’re not using virtualenv and virtualenvwrapper for Python development, the secret is to bang the rocks together guys!) My goal has been to create a reusable project framework for quick social experiments that provides the following features:

• External authentication, to exploit as wide an audience as possible without the hurdle of requiring registration.
• Useful debugging information, both through logging and in-page debug.
• Database schema versioning and migration.
• Asynchronous task queuing and control.
• A lightweight WSGI HTTP server to sit between Django and Nginx.
• Straightforward deployment.
• Messaging, so that users can communicate with one another.
• Friends / social groups, so that users know who they want to communicate with.
• A rewards system, to reinforce and drive positive activity.

(more…)

Ten things I’ve learned as a developer at WooMe:

1. Be clear what your direction is. Identify your key proposition and focus on it. Don’t get distracted by unrelated features — somebody else is probably already doing those better than you.
2. Be merciless with features that don’t cut it. No matter how much you like it, if your audience doesn’t like it, either change it or kill it fast. Wasting time on your personal pet project when there’s no clear demand for it is time you should be spending on features users want.
3. Use what’s already there rather than inventing your own. Need a message queue? Video processing? Load balancer? Email distribution? Payment system? Use something that already exists rather than waste time you should be spending on feature development.
4. Quick and dirty is better than late to market. At the rate this industry moves, capturing your audience’s attention first is more important than having a well-engineered solution. For the most part, users don’t care about your code quality, as long as it works.
5. Don’t optimise until you need to. Sure, you may end up chasing your tail for a bit when you need to scale, but until you hit that point you’re wasting valuable time that should be spent on feature development.
6. Scaling is most often a data problem. When you do need to optimise, scaling won’t be about the performance of your application. Scaling web applications horizontally is easy. Scaling databases vertically is expensive. Scaling databases horizontally is very, very hard. Don’t worry about what web application framework you use, worry about what database you put behind it.
7. Abstraction is great, as long as you understand what you’re abstracting. This has been a constant pain with Django’s ORM. Yes, it’s a powerful and useful abstraction, but without a solid understanding of what the database is doing behind it, it’s easy to get into very deep trouble, very quickly.
8. Log everything. Nothing is harder than trying to find problems on a live environment without detailed logging.
9. Storage is cheap. It’s better to store everything than to throw away data. You never know when you might need something.
10. Communication is critical. Not just outside your organisation, but within it. Tell people what you’re doing, frequently. Ask them what they’re doing just as often. Not only do ideas percolate through this process much more effectively, but when things go wrong, clear and frequent communication will stop them from getting worse.

This is a handy little snippet that allows you to set rate limits for views in Django.