The HDCD Enigma
During the last couple of decades there were three attempts to devise ‘better than Audio CD’ optical disc systems – DVD-Audio, SACD, and HDCD. For most practical purposes DVD-Audio died out. A pity, since it was arguably the best solution in terms of technical quality. The others have continued despite being ignored by the mass market. One is SACD (Super Audio CD) which is actually based on a modified form of DVD. The other is HDCD which is based on audio CD.
The vital distinction between these two systems in practice is that to play an SACD you require a hardware modified version of a DVD player. A normal DVD drive won’t work. So SACD is ‘locked’ to special hardware. Whereas an HDCD should play on any normal Audio CD player. Indeed, although HDCD is sometimes referred to as ‘High Definition CD’ the letters actually stand for ‘High Definition Compatible Digital’ according to an AES paper presented by its creators[1]. If you read this paper and the original USA patent[2] for HDCD it makes a number of impressive claims about what HDCD can do. But how does it actually work? Is an HDCD really much ‘better’ than a conventional Audio CD? And are HDCDs really ‘compatible’ beyond the basic fact that you can put them into a normal CD player and hear music?
HDCDs are quite rare. But some makers and artists have been enthusiasts. Until recently the only HDCDs I’d encountered are dual-layer ‘hybrid’ classical discs produced by Linn records. These have both a SACD layer and a ‘CD’ layer that is HDCD. But I then bought what I thought would be some remastered Joni Mitchell Audio CDs and discovered that two of them are labelled as HDCDs. Listening to them I was disappointed to find that they seemed to be victims of the ‘Loudness War’. They seemed too loud and gave a sound that in places seems uncomfortably compressed. This set me wondering: was this because I was playing them on a ‘normal’ CD player (actually a Cambridge Audio 851C) rather than an HDCD player? Or was it simply the common witless imposition of level compression by those remastering the CDs? So I decided to investigate...
What HDCD is said to do
HDCD is claimed to improve Audio CD in two general ways. One is to increase the available dynamic range - from the 16bit standard of Audio CD to (a claimed) 20bits. This nominally represents an increase in dynamic range from around 90dB to 114dB. The other is to mimic a wider effective bandwidth and provide an audible effect that makes musical transients sound more like you’d hear using a sample rate higher than the Audio CD standard. It is important to bear in mind each of the various types of alteration it can apply are provided as options from a ‘menu’ that those recording/mastering a disc can choose to apply (or not!) when making a specific HDCD. Up to those doing the recording and remastering which of them to deploy in a given case.
The expansion of dynamic range is said to be achieved using two methods. The first is a form of soft limiting for musical peaks. The second adjusts the gain for long quiet passages of music to make the recorded sample values larger. This make them bigger compared with the quantisation level. Effects like these may well be applied when recording and mastering anyway. But the key feature of HDCD is that it should embed ‘control codes’ into the audio data that an HDCD player can detect. They then tell it how to ‘correct’ the alterations done during recording and mastering the CD.
This has two obvious snags for those playing the discs on a normal player. Not being able to decode the peak soft limiting, a normal CD player will output distorted peaks. This may affect pop or rock discs in particular as they do tend to be (re)mastered to be loud. So it may be just another cause of audible compression and squashing of peaks. And the added control codes themselves take away some of the information bandwidth that could otherwise be used for musical details. So despite the claims, listeners playing an HDCD on a normal CD player may wonder if they’d agree with the makers that these discs are ‘compatible’ with CD players beyond the most basic use of the word!
Let’s look first at the peak ‘soft limiting’ in detail...
The values recorded onto Audio CD have 16bits per sample. For a normal CD these are assigned as Linear PCM (LPCM). This means the loudest parts of the music can make use of all 16 and get a reasonably high resolution. However quiet passages of music may only use some of the lower-significance bits. Hence quiet sounds get recorded with relatively low resolution. (Although if the recording is properly dithered and/or noise-shaped this should simply show up as background noise.) However if all the music in a recording was at low level we could decide to ‘wind up the gain’ and make the signal delivered to the digital encoder larger. This means we could make use of more bits for each sample and improve the result. Increasing the input signal level by a factor of 2 makes the result 6dB louder and lets us use another bit per sample. The snag is we may then run into trouble when we come to any loud passages of music. Too much gain means that – if we stick with Linear PCM – these become too big to represent and the recording clips the peaks off the waveforms!
The trick that HDCD employs is to apply a gain of x 2 when recording, but only to samples with a low value. According to the paper and patent this is applied to input signals that are 9dB below the maximum we can represent with 16 bit LPCM. Any input above this is essentially ‘bent’ to squeeze the top 9dB of the input into just a 3dB range on the CD. This alteration is done on a sample-by-sample basis and uses a transfer curve of the kind illustrated in Figure 1.
The graph on the left of Figure 1 compares normal LPCM recording with the HDCD peak limiting behaviour showing the behaviour in linear terms of the magnitude of the values. The graph on the right shows the behaviour in terms of the levels being represented in decibels relative to the maximum level for a 16 bit system. You can see that any input musical waveforms that stay below -9dBFS are simply amplified by 6dB when HDCD recorded. But larger waveforms (often transient peaks) are squashed into a smaller range. By using this trick HDCD can give another bit per sample to quiet sounds and hence get a nominal 6dB improvement in how far they are above the quantisation noise level compared with normal CD.
An HDCD player should be able to detect when this trick has been applied by recognising the control codes that HDCD encoding adds to the stream of sample values that are recorded on the disc. It knows the shape of the bending applied. So it can then expand the top 3dB of the range as presented by the disc out to cover 9dB again. This should ‘correct’ the squashing of the peaks and loud passages of music. The HDCD spec for players is that they should be able to output peak voltage levels that are about double that for a normal Audio CD. So the aim is that they play quiet passages from an HDCD as being about the same audible volume for the same music on a normal CD. But the ‘corrected’ peaks will emerge up to 6dB bigger than any output from a normal CD. Potentially quite a dramatic difference that may well impress listeners.
Alas, a normal Audio CD player has no idea any of this is going on. So it simply plays the CD assuming that the HDCD nonlinear representation of peaks or loud waveforms is an LPCM defined waveform which the listener should hear from their loudspeakers. The consequences can be seen in Figure 2. The upper graph shows an example of an input audio waveform that includes a sudden loud transient that then fades away. If we simply double the amplitude the tops and bottoms of the waveform become too large to be represented by the range of values available for an Audio CD and the result is clipped. The HDCD ‘solves’ this problem by remapping the sample values to let them cover the extended range by crushing the peaks into a nonlinear form.
The lower graph in Figure 2 shows the output we’d get when the resulting HDCD is played. An HDCD aware player can stretch the samples back into the original waveform shape. But a CD player treats the samples as LPCM and will output a distorted version of the original waveform. The result may then sound distorted and level-compressed. Given that most audio enthusiasts probably don’t have an HDCD player this means that an HDCD is an odd bargain given that the creators of the HDCD could have produced a CD without this distortion simply by keeping the level down to normal limits.
One curious consequence of this is that if the listener tries an HDCD with soft peak limiting on two players – one HDCD and the other CD – they may not like the uncorrected distortion (introduced by HDCD recording) when played on the CD player. So such a comparison may give an impression that HDCD is inherently ‘better’, when in reality a normal CD made of the same source material might also have sounded much better than the HDCD played on the CD player. In essence, the use of the peak alterations may cause CD replay to sound ‘worse’ rather than show HDCD as genuinely ‘better’.
This makes it hard for most listeners to make a fair comparison and tell if HDCD actually represents an improvement in practice. It is also unfortunate if the disc was purchased as being a normal Audio CD by someone who only has no HDCD-aware player. Given this peak distortion it also seems reasonable to question if HDCDs that have this effect applied are really that ‘compatible’ after all. Audio CD is based on LPCM, but HDCD soft peak limiting doesn’t conform with this. I guess this all depends how you view the matter or choose to define the word ‘compatible’, so is something each user would have to decide for themself!
Unfortunately it isn’t always easy for a buyer to tell when an ’Audio CD’ is actually an HDCD and when it isn’t! As mentioned at the start of this investigation, my curiosity about HDCD was prompted by finding that some remastered ‘Audio CDs’ I’d bought were actually HDCDs. One example is “Don Juan’s Reckless Daughter” as released on Warner 8122 74664-2. The outer packaging gives no sign that the disc is an HDCD. No logo or wording for this is visible. However examining inside the leaflet that came with the disc I found a mention in small type that said: “CD mastered... using the Pacific Microsonics HDCD system”.
Not having an actual HDCD player or knowing the details of the control code system I decided to experiment by writing a computer program that simply took for granted that material had been peak-crushed using HDCD peak soft limiting, and corrected for this. I then used the program on a 15 minute portion of the content of the disc and generate a 24-bit result with the 6dB increase in range. When I listened, the result certainly sounded better, particularly for tracks like “Dreamland” which combine singing with drums. Figure 3 illustrates the effect graphically.
The graph on the left of Figure 3 shows how often the musical peaks on the ‘Audio CD’ reach a given level. You can see that the distribution looks like it has been squashed up against the 0 dBFS maximum possible. The graph on the right shows the result after the material has been expanded on the assumption it was HDCD soft peak limited. This distribution is much more like what would be typical for carefully recorded (i.e. no compression or peak clipping) acoustic music. Both my ears and the statistics seemed to confirm this disc had peak level compression applied, and it may well be that this was done using HDCD soft peak limiting.
The soft peak limiting aspect of HDCD only gives a nominal improvement of 6dB (1 bit). Yet HDCD claims to widen the effective range from 16bit to 20bit. This means we still have to account for another 3 bits per sample (18dB). To do this HDCD claims to employ a form of dynamic manipulation. This time applied when the signal level is low. When the HDCD ADC/encoder detects that the signal level has fallen to around -40dBFS or lower it can be set to gradually “wind up the gain” during recording. This helps shift the music to a higher level and employs more bits per sample. So it can improve the music/noise ratio on the disk.
In practice, such gain adjustments are quite common in recording and broadcasting. But the key feature of HDCD low-level gain adjustments is that they are accompanied by control codes buried in the least-significant bits of the sample data. These codes can be detected by an HDCD player and then should cause it to ‘correct’ the gain downwards again when playing the disc. The result should then emerge at the original low sound level before the HDCD gain was used to boost what appeared on the CD.
Many pop/rock recordings are relentlessly LOUD essentially all the way though. They may never go below -40dBFS – except perhaps during track gaps or occasional silences! So for many pop/rock recordings this low level action is probably not relevant. But for material like some classical music that has a very wide dynamic range these correctable changes in gain could account for the other 3 bits per sample claimed by HDCD.
The HDCD documents also aim at tackling what the creators see as another limitation of Audio CD. The choice of conventional LPCM with a 44,100 sample/sec rate places an upper limit of 22.05 kHz on the range of frequencies that can be recorded without risking problems. Conventionally, engineers employ time-symmetric filtering to avoid problems due to the input being recorded having components above 22.05 kHz. However this has lead to worries, partly about ‘pre-ringing’ effects, and partly due to the loss of high frequencies that may have an audible effect.
HDCD claims to tackle this by using a choice of filters when recording. As with the peak limiting and low level gain adjustments, the aim is that an HDCD player can then process the values on the disc and be told – via the hidden control codes - which of a menu of replay filters to use for each section of music. So, for example, sustained notes and chords can be replayed using a filter similar to the classic time-symmetric (sinc-like) form. But transient spikes can be replayed using a filter that causes the output to contain some frequencies above 22.05 kHz, so trying to mimic reproducing a wider bandwidth.
Unfortunately, although the public documents give a good over-view of the general ways in which HDCD is claimed to work. they skate over many of the real practical details you would need to be able to assess how HDCD actually works in practice. So to go further we need to see what results it produces and compare them with using a conventional CD player. My report on what I discovered is on the next page...
[1] Compatible resolution enhancement in digital audio systems
Keith O. Johnson & Michael W. Pflaumer, 101st AES Convention Los Angeles 1996 Nov 8-11
[2] Compatible signal encode/decode system. Filed 1993 Issued 1995 Keith O. Johnson et al
USA Patent US5479168 can be viewed via http://www.google.com/patents/US5479168
