The Audio Factory

In September 2014 the BBC announced the “Audio Factory” project for the BBC Radio iPlayer. This involves replacing the equipment and systems used to deliver all the BBC radio streams over the internet. There has been a tendency in the past for the iPlayer to develop in an ‘ad hoc’ manner. Various new formats, transfer methods, etc, had been adopted as and when there was an opportunity and they were felt to be useful. As new types of user device became popular the system had been modified to bolt on ways to stream to them. However this had gradually led to a system that had become over-complex and was difficult for the BBC to maintain.

Figure 1 gives some idea of just how complicated the Radio iPlayer arrangements had become by 2014. It is worth pointing out that this brain-bafflingly complex diagram is actually a very simplified version of the full system, with many details omitted. Otherwise it would become even harder to follow!

When the then-new Coyopa system was introduced the old radio system (“BOB”) was re-tasked to provide the BBC ‘Nationals’ (i.e. Radio Scotland, Wales, etc) and BBC Local radio stations. Coyopa took over provide the UK stations (Radios 1 - 4, etc). In the above diagram, blue connecting lines show the ‘On Demand’ (‘Listen Again’) audio dataflows, and red lines the ‘Live’ radio. The solid lines show the UK station signals from Coyopa, and the broken lines the Nationals, and Local ones from BOB.

Gradually over the years leading up to 2014 various add-ons and modifications had been made to provide streams for new kinds of user device. The results worked amazingly well, but it did grow into what looks like a classic case of “A camel is a horse designed by a committee”! Although it worked, it became costly to run, and prone to problems that took head-scratching to fix because of the complications. And led to it having various odd quirks like users of Android devices not being able to listen to the Nations and Local programmes. In engineering terms, this complexity also meant that any future attempts to add new services or improve performance might accidentally break some other part of the setup.

Hence by 2014 the BBC decided that it was time to move to a system that – as far as possible – would be able to support all kinds of user devices with uniformly high levels of sound quality and accessibility, based on a common core of BBC hardware and software. It is perhaps significant that the BBC continues to have its income cut. So the system also needed to be one where operating cost mattered. One of the biggest consequences of these factors was that it made sense to have a system that is based on one audio codec, so far as possible. And from the BBC’s point of view, it also made little sense to pay to go on using codecs like WMA when the fraction of accesses that used it was a dwindling minority.

Figure 2 shows the totally new system the BBC devised to act as a complete replacement for the older ones. This was given the “Audio Factory” name. There have been parallel developments for the the BBC’s TV output, but here my interest is only in the sound radio arrangements. So I won’t go into the complications of the TV side.

The Audio Factory is largely ‘cloud based’. i.e. it uses internet servers and services in a way that allow for the BBC to expand or modify the system easily. So far as possible Audio Factory uses HTTP ‘chunked’ delivery techniques. This means the BBC can make use of a wide range HTTP systems and methods. It doesn’t require specialised firewalls or routing rules, etc. It also allows the use of a wider range of delivery networks, and isn’t limited to those that specialise in media delivery. The result is much more easily scalable and adaptable than the previous BBC systems. It should also save cost!

For reliability the Audio Factory operates with two chains of dataflow to maximise the ability to operate continuously. The system follows an established engineering practice at the BBC where any critical parts of the distribution chain is duplicated and run in parallel with various crossover points. It has become traditional to call these the ‘P’ and ‘Q’ chains for reasons that have become lost in the mists of time! (The terms seem to have originated from the old BBC London Control Room. But the reasons people give for choosing ‘P’ and ‘Q’ vary.) In this case it means that, in normal operation, when you listen to the iPlayer some of the chunks of audio may come to you from one encoder, and others come from the other. You can’t tell. But it allows equipment to be repaired, modified or upgraded without listeners noticing any problems.

In terms of what the listener receives, the main changes the new system provides are designed to lead the Radio iPlayer into a situation where the AAC (mp4) audio codec becomes the format for all output streams. That will mean the system will be simpler to operate and the money and effort required for other codecs can be saved. Output is provided at four bitrates. Users need only access via the Content Delivery Network (CDN) machines. 320 kbps and 128kbps AAC-LC are provided for live and listen again for UK listeners. The rest of the world will be streamed using 96 kbps and 48 kbps HE-AACv1.

Eventually, all the BBC Radio stations - UK, Nationals, and Local - will be provided at 320 kbps in the UK. This will be the case for both live and on demand. However during the initial operation of Audio Factory the old RTMP arrangements had to be used. That complicated the situation for a while. Initially only Radio 3 will stream 320 kbps because that was already implemented for RTMP, with the other stations temporarily limited to 128 kbps (UK) or 48k (World). This applies to both HLS and HDS. When MPEG-DASH is launched for browsers, etc, these temporary limits will be removed and 320 kbps will become the UK norm. Players can then select the best rate they can sustain.

The BBC have chosen a software encoder by Omnia that uses the current Fraunhofer AAC encoder libraries to generate the streams. The only significant exception – at least for now – is that Shoutcast will provide 128 kbps mp3 via the ICY stream container. That’s why the Shoutcast part of Figure 2 shows its data flow as a broken line. It helps make clear that its an exception to the use of AAC.

Along with the decision to ‘de-clutter’ the number of audio formats and bitrates the BBC also decided to try and do the same for the transfer methods employed. In this case their long-term aim is to move towards adopting the openly defined standard methods that should eventually be offered by HTML5/MPEG-DASH using the DVB (Digital Video Broadcasting) profile. It may seem strange given the reliance that web-browser access has had on Flash, but the BBC would much prefer to use ‘open’ methods. The challenge being to find ones which they can use in practice, particularly given the pressure put on them regarding Intellectual Property Rights from commercial owners of the programmes they’d like to broadcast.

Unfortunately, the HTML5/MPEG-DASH proposals didn’t crystallise into a firm, useful standard as early as the BBC had hoped. And for contractual reason they had to cease using Coyopa and BOB by the end of March 2015. So their current plan is to use three different transfer methods. The Audio Factory uses the USP (Unified Streaming Platform) packaging system which can ‘wrap’ the audio into these delivery containers. All three of these are designed to send ‘chunked’ audio (and video) data for streaming. Initially, they will use HLS and HDS.

HLS stands for HTTP Live Streaming. It is an Apple proprietary streaming format. For obvious reasons that means it is supported by iOS devices. It is also available for Android.

HDS stands for HTTP Dynamic Streaming. This is an Adobe proprietary system. That gives it the advantage of being a ‘drop in’ replacement for the current RTMP transfer method for devices that use Flash. It should ‘just work’ with desktop computer web browsers. All being well, people listening via a home computer this way won’t actually notice the change unless the user interface of the browser plugin is altered. Indeed, I’ve already had a chance to try a test HDS stream using a Flash plugin with Linux and it worked. So, all being well, those who use Flash to access BBC internet radio (and TV) with a web browser running on Mac, Windows, and Linux computers will continue to have access.

The obvious snag is that both HLS and HDS are proprietary. But they’re available and usable right now for a fair part of the listening audience. However HTML5/MPEG-DASH will be added as soon as possible. This won’t actually replace HDS. From that point on, browsers which can accept the MPEG-DASH can use it, whilst browsers that can’t will be able to go on using HDS via the Flash plugins. As I write this (April 2015) the BBC’s plan is to go on providing the remaining Flash-mediated RTMP/aac radio streams until at least after the 2015 Proms. But at some later date, once it has been established that people can easily use HDS instead, the RTMP will cease.

The first significant practical consequence of the changes was that support for WMA streams ceased at the end of 2014. The decision to do this was made public in September 2014, and the intent was that the makers of net radios, etc, would switch to the new streams. In fact, the BBC had actually initially contacted a number of net radio manufacturers, etc, back in January 2014 to discuss the changes with them. But this didn’t happen in all cases. As a result many users of commercial ‘internet radio’ devices lost access, much to their annoyance!

It is unclear in some cases why this occurred, and the actual reasons probably vary from one manufacturer to another. It seems to have been a mixture of lack of clear communication and understanding between the BBC and manufacturers, along with some makers lacking interest in updating their ‘older’ equipment. And in some cases the hardware in the radio may not have been capable of being updated. Whatever the reasons, the BBC got the flak!

One of the problems the BBC had faced during this period is that many net radio requests to their servers don’t actually identify the device’s details. So although they can see how many requests they get for a stream, they can’t always work out what maker’s device is at the other end. That makes it harder for the BBC to tell what devices might need updating or replacing, and which maker to talk to about it.

During February 2015 the changes proceeded and all the BBC Radio Nationals and Local radio stations were moved onto the new Audio Factory system. The ‘Live’ system began handing out HDS from 26th February, and ‘Live’ RTMP was turned off on 4th March 2015. However at present for on demand, RTMP continues to be provided for some devices, including desktop machines. It is intended that this will continue until the BBC can launch on demand using MPEG-DASH/HTML5. The aim is that this will happen in time for the 2015 Proms, but RTMP won’t be ended entirely until both HDS and MPEG-DASH have been established for a while. So there will be a period when RTMP runs in parallel with the other methods. So as of the first week or so of March 2015 when you use a Web Browser like FireFox with the Flash plugin you will be listening to an HDS stream for ‘Live’ radio but still getting RTMP for ‘Listen Again’. But this will change in the coming months as things develop. At present, the dates for the other changes aren’t fixed.

Because Flash can accept HDS you may not notice any difference, and have already been accessing the new HDS streams without being aware of the fact! However for Linux users there is a subtle snag...

Internally, the BBC audio systems are based on using a sample rate (as distinct from a bitrate) of 48k samples/sec. However the output sample rate you get may depend on how you access the iPlayer. When the previous (Coyopa) system was installed some years ago it was discovered that the then-current versions of Flash apparently couldn’t actually cope with the 48k sample rate. Instead, Flash ‘downsampled’ the audio to 44.1k. Unfortunately 48k to 44.1k conversions need to be done with great care to avoid the risk of audible degradation of the sound. So to work around this the BBC servers converted the audio to 44.1k before it was sent. This allowed the BBC to ensure that the conversions were done to a high quality standard. Since then, new versions of Flash have been released, and these can cope with 48k. Now here comes the “However”... The Audio Factory no longer performs these conversions. They ceased in early March. Since current versions of Flash can apparently cope with 48k this means people who use them can now get the ‘closer to the source’ 48k and hear it without any 48k to 44.1k conversions. Good news for them. Unfortunately, Adobe stopped bothering to produce updated versions of Flash for Linux machines some years ago.

[ Please note: At the time I write this (April 2015) discussions on the issue are under way between Adobe and the BBC and the matter is under active investigation. Should there be any developments I'll update this web page accordingly. But at this point the problem seems to be present.]

The result being that the Adobe Flash version available for Linux apparently still can’t cope with 48k. So it downconverts the radio stream. This means that if you use Linux, and FireFox plus Flash to listen to the BBC you will now be relying on an ancient Flash system that may insist on downconverting the audio before you can hear it. Although I doubt most listeners who aren’t HiFi enthusiasts would notice or be upset, this is a snag if you are an audiophile and want to be able to use Linux to get the same high technical quality as people running other operating systems!

To put this into context, it is worth remembering that it common for most computer operating systems to install and run an audio ‘mixer’ out of sight of the user. This often means that the sound you hear is being resampled anyway! If that hasn’t bothered you, then it seems quite plausible that the Linux Flash limitation won’t, either.

Although this seems to put Linux users “on the naughty step”, it should be temporary. In the longer term the BBC will provide HTML5/MPEG-DASH and browsers like FireFox will be able to use that. Until then according to the BBC there is an alternative – although some Linux users won’t like it. This is that a Chrome (browser only) is expected to be available and to cope with the Audio Factory output without the limitation that handicaps those relying on the old version of Flash. I haven’t myself been able to test this yet. But if it works it will be a possible solution for those who want to avoid the Flash limitation.

So for a while, Linux users may have to decide which they’d prefer: Ancient Adobe Flash giving slightly downgraded audio, versus a Chrome browser that should match the quality available on other OSs. Either way, MPEG-DASH should come to the rescue later.

I’ll say more about these developments as and when I obtain any new info, or can do measurements on the quality of the results.

Finally: I’d like to thank the people at the BBC who have kindly provided me with a lot of useful information and comments on what I’ve written. The information came from them. Any remaining errors are mine!