Stephen Silver, Apple Insider:
After years of controversies over Consumer Reports’ assessments of Apple products, AppleInsider paid a visit to the organization’s headquarters for an inside look at the testing process.
The whole thing was fascinating, but start things off with a read of this actual Consumer Reports press release, specifically targeted at Apple Insider.
Glad those fences are mended.
Per that article, regarding speaker testing and the HomePod, it’s simply impossible to do a fair sound test if the tests are not blind. If you know which speaker you’re listening too, your personal bias and expectations will factor into the sound, regardless of how subjective you may try to be. The tests HAVE to be blind, I’m not sure how CR justifies their current methodology.
I’ll just stick with CR for reviews about washers, dryers, refrigerators and the like.
They don’t enjoy wearing blindfolds?
They don’t enjoy wearing blindfolds?
I’ll be interested to see the follow-up to this piece.
And because I enjoy beating dead horses (and don’t want to register with AppleInsider), “[t]he first major tiff between Apple fans and Consumer Reports” wasn’t 2010’s “Antennagate.”
Whether it was their review of the Mac Plus or of the first iMac, CR’s reporting on tech occasionally set standards for a category without fully examining whether a test device actually fit that category. Grading a starter Mac on its lack of Windows-box features, or on whether it had upgrade capabilty for connectivity that was already built-in.
I’d love to know if the magazine archives from back then are available for perusal.
The other issue I’d have with their testing of the HomePod is that it’s incomplete. One of the advantages of HP is that it adapts to the room it’s in. Other smart speakers don’t do this IIRC, so in a real world setting I’d expect the gap between HomePod and others to widen even setting aside the blind testing issue.
A couple of weeks ago I walked into a fairly large Apple store here (this one – https://www.apple.com/retail/universityvillage/) to pick up a repaired MBP. There was music playing pretty loud and it sounded great… I looked up to the ceiling since the sound was filling the front part of the store. There was nothing on the ceiling. It was a single HomePod on the front table.
The other issue I’d have with their testing of the HomePod is that it’s incomplete. One of the advantages of HP is that it adapts to the room that it’s in. Other smart speakers don’t do this IIRC, so in a real world setting I’d expect the gap between HomePod and others to widen, even setting aside the blind testing issue.
A couple of weeks ago I walked into a fairly large Apple store here to pick up a repaired MBP. There was music playing pretty loud and it sounded great… I looked up to the ceiling since the sound was filling the front part of the store. There was nothing on the ceiling. It was a single HomePod on the front table
Yes, that would be a very worthwhile test. It would require CR to reconfigure their entire sound testing setup, with different room configurations, and to re-test and compare against other products, so I doubt they’d do it. However, I’d absolutely mention it in the review as an area the HomePod may have a significant advantage if it works as advertise.