Perl on Rails — Why the BBC Fails at the Internet
Perl on Rails is a project by the smart chaps over in BBC Audio and Music Interactive that replicates the Ruby On Rails MVC framework in Perl. They’re obviously rather proud of themselves, and I understand that internally the project is making waves. Whilst I applaud the technical achievement of the individual developers, I deplore the situation that has forced them to do this.
The problem is that the BBC doesn’t control its own technical infrastructure. In an act of staggering short-sightedness it was outsourced to Siemens as part of a much wider divesting of the BBC Technology unit. In typical fashion for the BBC, they managed to select a technology supplier without internet operations experience. We can only assume that this must have seemed like an acceptable risk to the towering intellects running the BBC at the time. Certainly the staff at ground level knew what this meant, and resigned en masse.
Several years later this puts the BBC in the unenviable situation of having an incumbent technology supplier which takes a least-possible-effort approach to running the BBC’s internet services. In my time at the BBC, critical operational tasks were known to take days or even weeks despite a contractual service level promising four hour response times. Actual code changes for deploying new applications were known to take months. An upgrade to provide less than a dozen Linux boxes for additional server capacity — a project that was over a year old when I joined the BBC — was still being debated by Siemens when I left, eighteen months later.
The BBC’s infrastructure is shockingly outdated, having changed only by fractions over the past decade. Over-priced Sun Enterprise servers running Solaris and Apache provide the front-end layer. This is round-robin load balanced, there’s no management of session state, no load-based connection pool. The front-end servers proxy to the application layer, which is a handful of Solaris machines running Perl 5.6 — a language that was superseded with the release of Perl 5.8 over five and a half years ago. Part of the reason for this is the bizarre insistence that any native modules or anything that can call code of any kind must be removed from the standard libraries and replaced with a neutered version of that library by a Siemens engineer.
Yes, that’s right, Siemens forks Perl to remove features that their engineers don’t like.
This means that developers working at the BBC might not be able to code against documented features or interfaces because Siemens can, at their sole discretion, remove or change code in the standard libraries of the sole programming language in use. It also means that patches to the language, and widely available modules from CPAN may be several major versions out of date — if they are available at all. The recent deployment of Template Toolkit to the BBC servers is one such example — Siemens took years and objected to this constantly, and when finally they assented to provide the single most popular template language for Perl, they removed all code execution functions from the language.
So talented, underpaid, and frustrated software engineers at the BBC are forced to make a decision. Either they can produce websites using static HTML, and make a few remote calls to limited Perl functions, decorating their page with SSIs, or they can fight against a reticent and incompetent technology supplier to make use of a crippled and outdated language on servers that more than likely are unable to meet the capacity requirements of a dynamic application being used by the BBC’s audience. Software engineers at the BBC must become masters of the sleight-of-hand, using every smoke and mirrors tactic they can to conjure the appearance of dynamic websites, not exactly what you would expect from one of the largest media corporations in the world. Oh, and if you’re an external agency working for the BBC and hoping to write a new application or build on technologies that the rest of the world has taken for granted for the best part of a decade, you might as well forget it. There’s only one externally available development server, and it’s not in synchronisation with the live environments.
It doesn’t have to be this way. If, instead of forcing its teams to waste valuable license fee payers’ money on duplicating existing free software, the BBC decided to take control of its technical infrastructure and provide a viable platform for complex, dynamic applications, then that creativity, effort, and time could be directed at making more of the kind of applications that make the BBC great.
Some work is already progressing in this direction. A large part of the BBC’s Creative Futures project is what the BBC calls “BBC 2.0″ (often mistakenly referred to by executives and television-types as “Web 2.0″). The last I heard this was planning to deploy an architecture based around Java, Tomcat, Hibernate, Velocity, and MySQL. Whilst I disagree with the choice of technology for many reasons, this is at least an important step in the right direction for the BBC — as long as they exert control over the infrastructure from end to end.
It’s a ridiculous situation, and I know that many talented and respected technical staff have left the BBC in the past few years citing frustration at the insufficient technical infrastructure, and the inability of both Siemens and BBC management to keep up with the pace of technological change. Unfortunately, unless something dramatic changes with the upper levels of BBC management to recognise the nature of the problem, it’s a situation that will remain the status quo for a long time to come.
Tags: bbc, technology, rant
Although I agree with the overall sentiment with your article (the BBC infra is definitely way behind the times and completely out of date, Perl on Rails seems insane to me even if it is a commendable technical achievement, the BBC and Siemens JOINTLY move way too slowly), there are a couple of points worth correcting:
- The BBC absolutely does control this infrastructure. Siemens just commission hardware/software according to BBC spec. i.e. the BBC mandate that Siemens run Solaris and a home brew version of Perl (e.g. due to InfoSec requirements). There’s an architecture team in the BBC (FM&T) who specify this stuff and are completely empowered to change the stack if they want to. If you asks the Siemens’ architects that support the BBC they absolutely would not agree with the current stack and would love to be moving this stuff forward.
– You can’t say Siemens as a technology supplier does not have internet ops experience because then you are implying that the BBC does not have that experience. Almost all of the staff there were transitioned from the old BBC Technology — they just changed employer, and the BBC very much have control of the steering wheel.
Maybe part of the problem is the half-heartedness of the outsourcing deal in the internet area (I can’t comment about other parts of the outsourcing deal). It seems they’ve only partly outsourced whilst trying to control everything from a single point. Maybe fully outsourcing or fully insourcing would be a more productive environment?
@maui_project_28: whilst the BBC does have nominal control of the infrastructure, at a practical level it’s taken five years to get Perl 5.8 available (and it’s still not live) simply due to delays in the provisioning of hardware and testing software.
If the BBC were in control, I know for certain it would have taken weeks, maybe months at the outside.
Sure, there are probably some people working for the supplier who want the architecture to move forward just as there are many people at the BBC who want to ditch Perl — the trouble is they’re not the people calling the shots. I spent a great deal of time working with the Architecture team because I was running several large technology projects, and believe me they are not empowered to change the stack. Even with the BBC 2.0 project the new Java stack is something that will go through huge executive review before it gets signed-off. I heard of at least one highly-placed person at the BBC who has assigned themselves veto power over these new developments.
Siemens as a supplier has had almost 100% staff turnover in their internet/unix operations since the outsourcing of BBC Technology. I think there was only one original staff member still at Siemens when I left, though there may still be others lurking in the shadows.
“Yes, that’s right, Siemens forks Perl to remove features that their engineers don’t like.”
That statement is misleading and incorrect. BBC Technology (later sold to Siemens Business Services) was instructed by the BBC to security audit all code before it was installed on the BBC web servers. This included (as mandated and required by the BBC) removing any code that would make system calls or direct unfiltered/unmoderated disk I/O, to name but a couple of requirements.
The problems that you document in this post where there long before BBC Technology was sold.
Bravo! This is exactly the sort of post I wanted to write, but I didn’t feel I had enough details to do so. You probably saw my comments (and the response) on the original post. As a licence payer, this state of affairs seems crazy. The BBC should be a beacon of progressive development, but it seems to be conservative old Auntie on the technology front.
@nicolaw: Yeah, I know that a lot of the mess is legacy and some of it is due to conflicting internal pressures at the BBC. However, having a mandate to tighten security of code doesn’t require stripping out parts of a standard language, without documenting those changes for developers, on servers where code cannot be deployed by the BBC developers because even a single line change has to undergo rigorous review by a Siemens engineer. (We missed you after you left — my raised tasks went from an average of a couple of weeks to an average of a couple of months to close.)
Regardless of where the problems originated, wouldn’t you agree that after four years something should have been done to address the problems?
“…planning to deploy an architecture based around Java, Tomcat, Hibernate, Velocity, and MySQL. Whilst I disagree with the choice of technology for many reasons” Just out of interest, what are some of these reasons and what might you suggest?
Hasn’t anyone @ BBS or Siemens ever heard of Perl’s taint checking feature??
This is a brilliant article. I’d scratched my head until my scalp became raw over how the world’s largest and most respected news broadcaster could simultaneously have one of the stupidest and ugliest web sites imaginable — made to look all the more Web1.0 by the advertisements they insert randomly on the international version and their baffling continuance of Real (fake) Video.
This piece makes it clear as to why the beeb has failed so miserably on iPlayer too. I dread to think what appalling deals with utterly inappropriate companies have been done using licence fee payers money on that debacle.
@rickbradley: Yes, both the BBC and Siemens have heard of taint checking. It doesn’t solve everything, and in some situations it makes the problems in hand worse.
The infrastructure behind the BBC’s Journalism websites — http://www.bbc.co.uk/news and http://www.bbc.co.uk/sport being the big ones — is set up in quite a different way. Most of the operational work is handled by BBC admin teams.
A lot of the “Siemens are the bad guys” points are, as some others have already suggested, a bit misdirected. Its the beeb which is quite frankly a slow oversized unwieldy beast. Let me reminisce about my time..
I joined a certain web team (not radio — who, I have to say, like IF&L were far whizzier with their tech than the rest of the beeb*. I worked alongside some really interesting and talented folk. Most had degrees in design or computer science. So I figured they must be doing really cool complex stuff? No sadly not. From the programming side of things you spent your days restricted to 4 year old technology working purely in HTML and lame restricted javascript. The SSI tweaking as mentioned above was done for two reasons I always felt: 1. To make a clever dynamic looking thing dynamic and most importantly 2. To stop brain rot. From the design side of things, when a interesting project did pop up on the radar where did it go? No, not the 31k salaried, very experienced designers but outsourced to some company willing to way undercut the inside team purely to say “we work for the bbc”. Shocking state of affairs.
I clearly remember a conversation with a line manager “I can code in PHP, Perl, Java.. can I have a tinker with some of these things instead of poxy SSI’s” “Afraid not.. you don’t have the correct job title”. What a dull place. So I left..
*NB: If you think “bbc” and think “bbc news online” and nothing more ignore about 70% of everyones comments/this article. BBC news works outside of the rest of the corp and does things completely differently — a lot of the times better..
Excellent article. Just looking to spec a BBC project that in the real world would be straight forward and scalable, and realising that in the BBC environment it’s going to be an absolute mare and seriously compromised.
Having also worked on the JAM project and used the ridiculous M$ IIS fudge it ran on — it seems that (for whatever reason) the BBC are determined to make things as difficult as possible for external suppliers.