I Am Seb

Perl on Rails is a pro­ject by the smart chaps over in BBC Audio and Music Interactive that rep­lic­ates the Ruby On Rails MVC frame­work in Perl. They’re obvi­ously rather proud of them­selves, and I under­stand that intern­ally the pro­ject is mak­ing waves. Whilst I applaud the tech­nical achieve­ment of the indi­vidual developers, I deplore the situ­ation that has forced them to do this.

The prob­lem is that the BBC doesn’t con­trol its own tech­nical infra­struc­ture. In an act of stag­ger­ing short-sightedness it was out­sourced to Siemens as part of a much wider divest­ing of the BBC Technology unit. In typ­ical fash­ion for the BBC, they man­aged to select a tech­no­logy sup­plier without inter­net oper­a­tions exper­i­ence. We can only assume that this must have seemed like an accept­able risk to the tower­ing intel­lects run­ning the BBC at the time. Certainly the staff at ground level knew what this meant, and resigned en masse.

Several years later this puts the BBC in the unen­vi­able situ­ation of hav­ing an incum­bent tech­no­logy sup­plier which takes a least-possible-effort approach to run­ning the BBC’s inter­net ser­vices. In my time at the BBC, crit­ical oper­a­tional tasks were known to take days or even weeks des­pite a con­trac­tual ser­vice level prom­ising four hour response times. Actual code changes for deploy­ing new applic­a­tions were known to take months. An upgrade to provide less than a dozen Linux boxes for addi­tional server capa­city — a pro­ject that was over a year old when I joined the BBC — was still being debated by Siemens when I left, eight­een months later.

The BBC’s infra­struc­ture is shock­ingly out­dated, hav­ing changed only by frac­tions over the past dec­ade. Over-priced Sun Enterprise serv­ers run­ning Solaris and Apache provide the front-end layer. This is round-robin load bal­anced, there’s no man­age­ment of ses­sion state, no load-based con­nec­tion pool. The front-end serv­ers proxy to the applic­a­tion layer, which is a hand­ful of Solaris machines run­ning Perl 5.6 — a lan­guage that was super­seded with the release of Perl 5.8 over five and a half years ago. Part of the reason for this is the bizarre insist­ence that any nat­ive mod­ules or any­thing that can call code of any kind must be removed from the stand­ard lib­rar­ies and replaced with a neutered ver­sion of that lib­rary by a Siemens engineer.

Yes, that’s right, Siemens forks Perl to remove fea­tures that their engin­eers don’t like.

This means that developers work­ing at the BBC might not be able to code against doc­u­mented fea­tures or inter­faces because Siemens can, at their sole dis­cre­tion, remove or change code in the stand­ard lib­rar­ies of the sole pro­gram­ming lan­guage in use. It also means that patches to the lan­guage, and widely avail­able mod­ules from CPAN may be sev­eral major ver­sions out of date — if they are avail­able at all. The recent deploy­ment of Template Toolkit to the BBC serv­ers is one such example — Siemens took years and objec­ted to this con­stantly, and when finally they assen­ted to provide the single most pop­u­lar tem­plate lan­guage for Perl, they removed all code exe­cu­tion func­tions from the language.

So tal­en­ted, under­paid, and frus­trated soft­ware engin­eers at the BBC are forced to make a decision. Either they can pro­duce web­sites using static HTML, and make a few remote calls to lim­ited Perl func­tions, dec­or­at­ing their page with SSIs, or they can fight against a reti­cent and incom­pet­ent tech­no­logy sup­plier to make use of a crippled and out­dated lan­guage on serv­ers that more than likely are unable to meet the capa­city require­ments of a dynamic applic­a­tion being used by the BBC’s audi­ence. Software engin­eers at the BBC must become mas­ters of the sleight-of-hand, using every smoke and mir­rors tac­tic they can to con­jure the appear­ance of dynamic web­sites, not exactly what you would expect from one of the largest media cor­por­a­tions in the world. Oh, and if you’re an external agency work­ing for the BBC and hop­ing to write a new applic­a­tion or build on tech­no­lo­gies that the rest of the world has taken for gran­ted for the best part of a dec­ade, you might as well for­get it. There’s only one extern­ally avail­able devel­op­ment server, and it’s not in syn­chron­isa­tion with the live environments.

It doesn’t have to be this way. If, instead of for­cing its teams to waste valu­able license fee pay­ers’ money on duplic­at­ing exist­ing free soft­ware, the BBC decided to take con­trol of its tech­nical infra­struc­ture and provide a viable plat­form for com­plex, dynamic applic­a­tions, then that cre­ativ­ity, effort, and time could be dir­ec­ted at mak­ing more of the kind of applic­a­tions that make the BBC great.

Some work is already pro­gress­ing in this dir­ec­tion. A large part of the BBC’s Creative Futures pro­ject is what the BBC calls “BBC 2.0″ (often mis­takenly referred to by exec­ut­ives and television-types as “Web 2.0″). The last I heard this was plan­ning to deploy an archi­tec­ture based around Java, Tomcat, Hibernate, Velocity, and MySQL. Whilst I dis­agree with the choice of tech­no­logy for many reas­ons, this is at least an import­ant step in the right dir­ec­tion for the BBC — as long as they exert con­trol over the infra­struc­ture from end to end.

It’s a ridicu­lous situ­ation, and I know that many tal­en­ted and respec­ted tech­nical staff have left the BBC in the past few years cit­ing frus­tra­tion at the insuf­fi­cient tech­nical infra­struc­ture, and the inab­il­ity of both Siemens and BBC man­age­ment to keep up with the pace of tech­no­lo­gical change. Unfortunately, unless some­thing dra­matic changes with the upper levels of BBC man­age­ment to recog­nise the nature of the prob­lem, it’s a situ­ation that will remain the status quo for a long time to come.

Tags: bbc, tech­no­logy, rant