Jun 6, 2011
Can you trust your ears ?
Can you trust your ears ? Can anyone ?
In particular – do data-identical audio files always sound the same ?
I’ve just been taking parting an interesting conversation on the LinkedIn Pro Tools User group about the best type of lossless file compression to use for moving audio around.
Some people, including me, have replied “any, they’re all the same”, but several more very experienced, respected and successful engineers have commented there that they never use data-compression on their audio files, because they think it can affect the sound.
Now when it comes from people you respect, this is a fascinating claim, because the there is no theoretical way that this can be true, unless there is a fault with the playback system, or the testing method.
Because lossless data-compression has to be that – lossless – or it’s broken, by definition. And, if two files are data-identical, they are effectively the same file. So they must sound indistinguishable – again, by definition. If they don’t, then there is a fault or design flaw with the replay system.
Thats the theory, at least. And in my experience, the theory holds up. Here’s an example:
Sometime in the mid nineties, I came face to face with the ‘data identical files don’t always sound the same’ issue myself. I had completed mastering on a really great-sounding independent production, and both I and the artists were delighted with the results. Imagine my surprise when shortly after the discs were pressed, the engineer called me up, very disappointed with the way the pressed discs sounded. To his ears, the CDR test copy sounded much better !
I took a quick listen, and was shocked to find I agreed with him.
My immediate assumption was that there was a pressing fault, and I loaded up both the test CDR and the manufactured disc in order to do a bit-for-bit comparison. To my surprise though, the disc checked out – the numbers were identical. They should have sounded identical, too !
Now at that time a debate about this subject was raging on the Mastering Webboard (RIP), with people who claimed to have heard differences like this for themselves putting forward quite plausible theories about ways that data-identical discs might sound different because of poor player design revealing variables in the manufacturing process – “pit-jitter” and so on.
I’d always been sceptical, but now I had my own example to check out!
Having been fooled by my own ‘observer bias‘ in the past, though, I knew I had to do a few listening tests myself to make sure I really knew I could trust what I was hearing. So, we set up a test scenario – two identical CD players, playing level-matched into a Studer desk, through B&W 801 monitors.
I found the test harder to repeat listening blind, but after a gruelling 40 repeat listens, I came out being able to hear a difference between the CDR and the pressed disc with about 70% accuracy. The differences were subtle, but that’s a statistically significant result. I was very excited – now, the problem was how to explain this to the plant, and figure out how to solve the problem.
I was literally about to get on the phone to my client, when I suddenly realised – there was a flaw with my testing method.
We had randomised which disc I was listening to, but not which player.
With a sigh, I steeled myself. We swapped the discs over in the players, and I did another 40 of the tedious blind tests. This time was easier though, because I knew what I was listening for, and sure enough, this time I picked one of the discs as sounding better with 80% accuracy.
The problem was, this time I picked the other disc.
The first time I picked the CDR as sounding better, but when we swapped the discs between machines, suddenly I picked the pressed CD, instead.
Which means, I wasn’t hearing a difference between the discs at all – I was hearing a difference between the players. Well actually, I can’t even say that for certain. It might have been a difference between the players, or perhaps I was hearing a minute level-difference between the two analogue signal paths. We had checked the output levels to +/- 0.1 dB, but even that doesn’t guarantee they were exactly the same.
So, that means the CD actually sounded the same as the CDR after all, right ?
No ! It just means I didn’t spot any difference – or rather, that if they did sound different, those differences were masked by the differences between the two players and/or the signal path.
Now in my opinion, that does actually mean there was no audible difference. I’ve tested myself in this way many times and I know I can hear a decent number of fairly subtle differences in digital audio – for example, EQ level differences of 0.3dB, the differences between different flavours of dither and noise-shaping, the difference between dithered and truncated 24-bit audio at 16 bits, absolute phase…
But if there was a difference here, it was too subtle for me.
That’s not really the point, though. (“What, oh what is the point”, I hear you cry ?!?)
The point is, I jumped to a conclusion, because of the opinion of a client I trust and like – and I stayed fooled by my own observer bias, right till the last moment, when I had to admit I’d proved to myself I was wrong, despite my initial instincts.
And, it’s not the first time.
So, I know from my own experience how easy it is to make mistakes when we’re listening to small (and sometimes, not-so-small ! ) differences. And, it helps demonstrate again just how hard it is to create a proper test. It’s even been shown that problems like a small delay or a tiny click when switching between two identical sound-sources can make us perceive a difference in quality.
Now, I have no way of knowing exactly what the engineers I was talking to on LinkedIn heard, how they tested themselves, or what might have caused the differences they heard. But I think there’s a good chance they may be mistaken, especially since the thing they heard – data differences between identical files – are theoretically not possible, in a properly designed digital audio system.
But maybe this is just my observer bias again, although this time from the other angle…!
Can you trust your ears ? When did you last fool yourself ?
Awesome image by Furryscaly
No related posts.
.gif "Dynamic Range Foundation - Active Member")
This is complete BS. Anyone actually knowledgable in electrical/computer/software engineering would understand that lossless compression from a good encoder DOES NOT affect the audio.
Um – which bit is BS ? The bit where I say that there’s no theoretical way that data-identical files can sound different, or the bit where I say it’s a difficult thing to to test ? Did you read the post ?
For those who are interested, the theory of how data-identical CDs might sound different goes like this:
- Poor player design (power supply, grounding) means mechanical noise from the motor bleeds into the audio. I’ve heard this – on some players you can hear a whine as the disc spins up, on the audio output.
- If this kind of thing happens, the theory goes that manufacturing tolerances in the discs could cause changes in the motor behaviour which might therefore affect the audio quality.
Now of course even IF this happened, it’s evidence of poor player design, not a genuine difference in the sound of data-identical discs. And exactly how this theory could be extended to two different files playing from a hard drive is hard to see.
But it is a scenario where discerning listeners might think they have heard those differences…
Ian,
I’ve seen those discussions over and over for years, and since I have all the theoretical knowledge at hand myself, I have become really tired of the subject.
However – you know what? Your article is pretty much the only serious and most comprehensive post I ever read about the topic, and I can actually agree to every point you make.
Thank you!
You’re welcome, and thanks !
Well put, Ian. It makes me happy that you sat through the grueling 40 blind tests TWICE. Most people would rather just stick with their bias rather than put their money where their mouth is.
It’s funny how these debates come up…and it’s all about super-subtle differences in sound that my wife (or any other normal, non-engineer listener) would never hear.
Awesome artice.
Thanks Joe ! Do I regret spending that particular six hours of my life on something that turned out to be a mirage ? No.
Would I spend another six hours testing it next time someone makes this comment to me ?
No
Will I spend a similar amount of time testing some analogue summing/tape saturation/console emulation things sometime soon ?
Probably…!
Hi Ian!
There is one thing I do not understand: why did you randomly swap BOTH the players and the discs in your second time through the blind test?
It is a known fact that different CDPs have different sound. Alright. When listening to two different CDPs, your personal preference will lead you to prefer one over the other – that’s fine. BUT – when swapping two discs randomly in an attempt to pick differences between them, they should be swapped ALL ELSE BEING EQUAL. In fact, differences in the sound produced by the two CDPs may make up for, or more than make up for, the difference in quality between the two records. Swapping both things simultaneously means keeping track of two different variables, which is NOT what happens when someone buys the CD and listens to it at home.
Or am I missing something?
Hi Andrea,
Sorry, I should have been clearer. The players and discs weren’t swapped simultaneously, either time.
The first 40 trials were made with one disc in one player, one in the other. I never knew which player I was listening to.Then I realised the flaw in this (same disc in same player) and swapped the discs over, so in the second 40 trials everything was the same, except the discs were the other way round in the players. Again, the listening sequence was random, and I didn’t know which player+disc I was hearing.
I’m still not sure that was clear ! So it was: 40 trials with Player A/Disc A, Player B/Disc B and then 40 trials with Player A/Disc B, Player B/Disc A. In both cases, I consistently chose “Player A” as sounding “better” – ie. I identified the difference between the players, or the signal path, or something…
Does that make sense ? The conclusion is that if there were any differences between the discs, they were masked by the differences between the players – and were therefore *extremely* small, in my opinion.
Does that help ?
Ian
Hi Ian,
yes, it did
Thank you very much!
Andrea
Interesting study you did.
I wonder if you have done something different from what Dennis, Dunn and Carson did in their study “The Numerically-Identical CD Mystery-A Study in Perception Versus Measurement” (Audio Eng. Soc 1997), http://www.prismsound.com/m_r_downloads/cdinvest.pdf . They have explanations in terms of interactions between analog and digital parts of CD-players etc, but I don’t recall them saying anything about differences between CD-players of the same brand.
Hi Sverre,
Thanks for the link, very interesting ! And, the folks at Prism can certainly be trusted to both listen acutely and test well
So yes, it looks like my little informal test was looking for the same differences they were, and both failed to hear them.
Fwiw, I know one person who went to far as to tape a small coin to the outer edge of a CD, in order to introduce a very major stress on the player’s servos, but still couldn’t hear any differences… I’ve never wanted to try this one though !
Ian
PS. The “brand” issue isn’t really what I was after – the players were identical models. Having two was just an easy way to compare without constantly loading and unloading discs – I find these comparisons need to be as instantaneous as possible, to stand a chance of hearing anything.
“Lossless” is indeed lossless, but that lossless-ness is guaranteed only as long as we keep the focus of our judgment within the digital domain. That is, one truly lossless compression schema will deliver identical results to another when it comes to re-constructing a digitally mathematical version of the data.
A digital reconstruction in not what our ears are hearing, however. What we hear is an analog synthesis of that digital model, and any differences that may or may not exist would have to be measured after the analog conversion in order to cover any potential variability in the way the schema codecs convert the data.
We see/hear this happen all the time with lossless schenas. MP3 codec differ, not only between “brands” of codec (e.g. ISO vs. Fraunhoffer), but often between two different implementations of the supposedly same “brands” of codec (e.g. LAME vs. Blade). Even in MP3, the “losslessness” should theoretically exist only in the encoding. Once encoded, the playback should not differ from one player to another – i.e the quantity and quality of the lost data should stay the same because the data was discarded in the encoding, not in the decoding. But we all know this is often not the case, that one file can indeed playback different in different codecs.
We also see this constantly in the ultimate examples of “lossless” format; WAV and CDA. Play back the exact same CD in three different CD players and you’ll often hear three different and identifiable results. Not because the data has changed, it hasn’t; it’s the exact same CD with the exact same data. What’s changed is the analog reconstruction of the digital data.
The question in this light is not so much, “Can we trust our ears?” as it is “Can we trust our playback?”, or maybe even finer, “Can we properly translate what we hear from our playback?” It like our studio monitors; we have to know how to translate what things sound like through them to what they will likely sound like in “the real world”. We have to be able to do the same thing with our DAC playback devices.
G.
Hi Glen,
You’re absolutely right – I’m convinced that well-made hardware will show no audible differences between identical lossless files.
But not all hardware is well-made, and every single model will differ in sound because of the analogue stage, as you say. Even that effect is tiny compared to the differences between speakers, though – let alone speaker placement in a room !
I think that’s one of the points I’m trying to make here – spending time and attention on apparent differences that theoretically don’t exist makes very little sense, to me. There are so many other more significant areas where we can make a huge difference in getting things to sound better…
Ian
Ian, I agree on all points; the difference in DAC coloration is an order of magnitude less than the typical kind of difference in monitor coloration. I didn’t mean to compare the two in terms of degreee of effect so much as just a simple analog comparison.
I also agree that DAC coloration, like ADC issues (e.g. sample rate quality) is probably somewhere down on the second half of page 2 on the list of things to worry about regarding the quality of our sound. In essence, if you’re worried abut either one of these, chances are there’s something else far more important that you havn’t crossed off the list yet.
Yet at the sane time it is something to be aware of, at the very least. You used the example yourself of listening to a mastered copy that didn’t sound as expected. It’s obviously important to understand when that is due to the playback system and when that is actually a problem with the source itself.
G.
Hi Glen,
Absolutely – and as a mastering engineer, it’s something I keep an eye on – just this week there is a debate about whether a similar “identical data sounds different” issue is happening with Blu-ray discs. So far though, I’ve seen no convincing evidence (meaning, rigorously designed blind tests) of these kind of effects happening. Which is not to say it won’t, of course…!
Ian
I have no knowledge of reliable data in that regard, either. But I think we can all rest assured that as we are entering the days where Blu-ray players become a disposable commodity right up there with pocket calculators and laser pointers and we can buy Blu-Ray players for $29 at the checkout counter of our local grocery store (true story, I witnessed that myself at my local Food-4-Less last week), that we can expect design and build (and, potentially, output and performance quality) to be of varying and dubious quality in at least some models out there, just as has happened with virtually every other format of audio and video in the past 50 years or so.
We have all seen (if old enough) record players, cassette players, videotape players, CD players and DVD players that have all in their own way colored or diminished the quality of sound or picture, especially – but not only – as the format becomes more popular and less costly. I see no reason why the same thing wouldn’t happen with Blu-ray.
Bottom line, no matter the quality of the actual source media technology, that is no guarantee that the quality of the delivery mechanism technology will always match that level or deliver it transparently.
But it’s often a tempest in a teapot. Make a crappy-sounding recording or phone in the mix or master quality, and the playback quality won’t matter. If the recording sucks, it’ll suck whether listened to on a Blu-ray disc or a wax cylinder
.
G.
G.
Great article, fascinating discussion.
One small point on mp3s specifically – a few comments back ‘lossless’ mp3s were mentioned. There is no such thing. The mp3 format is inherently lossy. Players are not required to be able to decode mp3s recorded at any bitrate of more than 320kpbs – you might find some that will, but most don’t.
I’m not sure it would even be theoretically possible to make a lossless mp3. Since creating files of mp3 format involves processing the original data in various ways, including applying a psychoacoustic model to remove masked frequencies, in order to produce a compressed file that fools the ear into thinking it is hearing the original, it’s highly likely that applying these processes to the original data for reproduction – even at the same bit rate – would significantly change that data, depending on the audio encoded and on the specific psychoacoustic model and other algorithms applied.
This leads me to be reasonably confident to assert that you’re only ever going to be dealing with lossy mp3s.
As mentioned above, the thing about mp3s is that they aren’t just lossy, they are differently lossy depending on the exact encoder used. This arises because the mp3 format is not fully specified. The vast bulk of it is, but the standard does not specify exactly how various parts of the encoding process should be implemented, including exactly what psychoacoustic model should be used. People writing encoders are encouraged to and do substitute their own (hopefully) improved versions of that model and other key steps – these will definitely vary from encoder to encoder and may also vary between different versions of the same encoder as developers (hopefully) improve their models.
Playback, however, is fully defined – if the same mp3 plays back differently in different players, this is likely to be due to the same kind of thing that leads to the same CD producing different output in different players.
@ Wayne – Agreed, on all counts. You’ve seen my rant about mp3, right ?
http://productionadvice.co.uk/why-mp3-sounds-bad/
It’s just wrong