Andy Ihnatko, writing about fake news:
I’ll use Apple as an example. They had a problem on their hands in the form of a deeply-negative Consumer Reports headline: “New MacBook Pros Fail to Earn Consumer Reports Recommendation.” CR had tested the new MacBook Pros and concluded that the battery life of all three models were insanely inconsistent.
Response Option 1: Apple calls the report “Fake News” and dismisses Consumer Reports as “failing, sad, and pathetic.” Next question.
Response Option 2: Apple disagrees with CR’s findings and tries to substantiate their results. An examination of CR’s testing methodology — done with the publication’s help — reveals no fudging, but identifies many quirks in the test protocol that probably contributed to a suspicious result.
Apple did exactly the right thing. If the original CR review was totally screwy, it shouldn’t be hard to demonstrate why, and Apple certainly has the resources to put in that kind of effort. Moreover, doing so indicates that they want to earn the trust of their customers, instead of demanding it.
And:
Apple emerges from all of this looking great, and everybody (Apple, Consumer Reports, and consumers) walks away with a better understanding of the issue.
This is exactly where the bar should be set. Truth seeking behavior.
[H/T John Kordyback]
Don’t forget that it’s not that the review was “totally screwy,” but that it uncovered a bug in Safari. It’s easy to blame this on CR, but it’s as much Apple’s fault.
Truth seeking behavior is only possible if those involved are seeking the truth. Impossible if they’re primarily interested in publicity and ratings.
Thank Dog Drump is the new president. He surely will follow option two every single time. Beaus he’s a man of integer… oh screw it. We’re doomed.
Andy sometimes writes pure magic. It’s usually inversely proportionate to the time he spends with Leo Laporte.
It’s more CR’s fault because they didn’t treat this as an obvious fault in the testing scenario (bug or no), and instead suggested it was defective batteries. The batteries were fine. The scenario experienced (cache off, bug hit) is not one a typical user will encounter thus the message was questionable. The correct thing would be to easily identify that this was a testing issue, and work w/ Apple on resolving it before publishing.
But, CR mindshare. Mission accomplished.
Do we know whether they used this cache off setting before? There is logic to it.
I think they do, their post implied it, but to be honest I don’t fully understand the CR test methodology. They say themselves in the post “our battery tests are not designed to be a direct simulation of a consumer’s experience” and they specifically go out of the way to disable the things that macOS does to save battery, such as screen dimming and caching. What use then is the review if you’re then recommending the product based on scenarios normal users won’t encounter? You wouldn’t test how fast a car can go by disabling some of the features designed to help it go faster…
They might as well not bother loading 10 web sites repeatedly and just create a program that uses all cores at 100%. At least that would be more even for testing between platforms wouldn’t it?
No, they didn’t make any conclusion about the battery, just that the results varied wildly. And if you read the article they notified and sent logs to Apple before publication.
The thing is, absent the bug if they run their tests cache off and get acceptable battery life it’s actually a better result because the machine was stressed slightly more yet still performed well. It appears that this is the way they have always run them for all laptops.
On CR’s end it still doesn’t explain the 18 or 19 hr result, and on Apple’s end it does not explain the many people experiencing disappointing battery life with this machine from its release, unless they are all running an unpatched Safari with cache off.