Consumer Reports is an irresponsible organization that rushed to publish its faulty MacBook Pro results before Christmas only to be caught and have to backtrack. I don’t trust a word these people say about anything.
Consumer Reports now recommends MacBook Pros
Wow. I don’t agree with your assessment. While they probably should have reached out to Apple before they published the results, there was nothing wrong with their testing methodology. There was a problem with Apple’s software. The initial inconsistent battery life problems are purely Apple’s own fault. Now everyone has done the responsible thing and fixed the issue, retested, and updated the results. Good stuff. I appreciate that Consumer Reports might uncover stuff like this that other reviewers did not.
no, because few are likely to experience the scenario they were tested in. the bug was only hit in such a dramatic fashion due to the disabled cache and automation. the batteries weren’t defective. so because it was the test scenario, and obviously so, the responsible thing to do would have been to work with Apple first, before publishing, not after. instead they suggested the notebooks had defective batteries.
the fact that normal users didn’t uncover this bug and only automated tested did is proof that it’s of lesser concern.
The Consumer Report inadvertently exposed bad coding inside the Safari and forced the Apple to engage
Agreed, it would have saved some drama if they had worked with Apple before publishing the results, but there was absolutely nothing wrong with their test scenario. In fact, it’s probably one of the better ways to accurately and consistently test battery life for web use cases in an automated fashion. You are incorrect in stating that “they suggested the notebooks had defective batteries”. They noted that battery life was highly inconsistent.
wrong. There was BOTH something wrong with their testing AND a bug. Yes, the testing uncovered a bug and that was a good thing. But a methodology that deliberately defeats battery conserving features, then complains about battery life is illogical. Test that way if you want but be VERY transparent that you’ve altered the settings from what they are out of the box. Even better, test with out of the box settings too.
Caching web pages is not designed to be a battery saving feature, it’s designed for page load performance. Disabling the cache may adversely affect battery life, but it also better simulates loading of new web pages in an automated test. At any rate, I think we’re focused on the wrong things here. Cache or no cache, the fact they are still getting 18 hours of battery life under any condition is more of a red flag to me.
Consumer Reports were behaved despicably for deceiving their audience in order to garner more publicity, that is obvious. There is no reason whatever to defend them.
which would indicate a defective product since they dipped to absurdly low levels. the entire point of CR is to warn of unsatisfactory products, so a non endorsement is saying exactly that.
the problem was in the testing — non real world, and not likely to affect actual users, even with the bug.
You don’t trust a word CR says because they tested a MacBook Pro using an available developer setting… but you’re damning them even though this ‘fix’ is still in beta and not widely available?
It’s a correct assessment that most consumers don’t use this weird developer-facing feature, but it’s also true that most consumers don’t join Apple’s beta program. Are you going to slam CR for testing on beta software, too?
It would be one thing if CR’s test results were found to be off-base once the setting was disabled in the macOS build it originally tested on, but there’s also a problem on Apple’s end. It’s not ‘backtracking’ if the subject issues a fix to a problem you encountered.
Instead of turning the volume up to 11 on how ‘irresponsible’ CR is, maybe it’s best to examine the landscape. It’s not as if CR was the only entity complaining about battery life – Apple went so far as to re-issue laptops to some tech news outlets when they complained.
I don’t have that option enabled and still have wildly inconsistent battery life on my specced-out MBP. It’s a widespread problem that will be fixed when Apple issues a software update.
You’re just like those videogame fanboys who bombard reviewers because they gave “their” game a low score.
Apple (and other companies like Microsoft and Google) engage with reporters and magazines when they read something that doesn’t make sense. Jim “Heineken” Dalrymple and John Gruber discussed this on the most recent episode of the Talk Show.
As for bad coding, I work in software and I’m generally impressed by quality of immensely complex pieces of software are from companies like Apple, Omni Group, and even Microsoft. Bugs always happen because it’s impossible to predict every circumstance, but the important thing is how quickly and well do companies respond when one is found.
Here’s a cool podcast which includes an interesting bug in the New Horizons Pluto mission: http://a16z.com/2017/01/01/radio-new-horizons-pluto-linscott/
To summarize:
You should really read CR’s current statement. It’s extraordinarily douchey. Rather than admit this won’t affect anyone, THEY ACTUALLY RECOMMEND PEOPLE SWITCH TO A BETA OS.
Sigh.
Where tech is concerned, CR has always reminded me of a room full of Windows NT sysadmins.
They tested using a developer setting. I remind you that the name of the magazine is “Consumer Reports.”
I would say the obligation to work with the manufacturer behind the scenes is fine if they were working on a pre-launch review model. But they bought them in the retail store, so they were the same ones customers were using.
Caught? Backtrack? For what? Cache on or cache off apparently didn’t make a difference for the bottom line results except to expose a bug in Safari. This whole episode has really brought out the drama queen in a lot of people.
In one of the articles CR says they reached out to Apple before publishing and sent logs.
CR’s proud that they buy everything they review, so you know they’re not biased toward/against any particular maker. But they’re always going to be biased toward their own business of reporting on this stuff. When there are dozen different companies churning out a dizzying morass of products, that’s very good for a company like CR. When one company’s products stand head and shoulders above the rest as the obvious choice, that’s very bad for a company like CR. And this same observation applies to media in general; it’s a huge part of why so many tech journalists were in love with the ’90s Windows heyday, and are correspondingly dismayed by Apple’s record-setting resurgence.
Consumer reports notwithstanding, with broad-brush statements like that Jim, why should anyone trust a word you have to say?
Damage done.
Apple loyalists will view this like every other Apple “scandal” and remain in the fold, unpersuaded.
Apple-neutral to Apple-haters will use the original results to fortify their position(s).
Now we need a consumer reports for consumer reports.
Oh, CR, get your shit together.
Consumer Reports is not irresponsible. The problem reported is not a bug in CR. It’s a bug in Apple software. That’s Apple Software. You know, the company that stopped caring years ago. Oh, the iPhone 8 will come with free ponies.
Ummm no.How did they deceive their audience. They stated their methodology, ran the test, reported the result. No testing organization is beholden to any manufacturer regarding the results of a test. That’s what makes them independent.
What you seem to be missing is that they are now misleading people by stating it was a bug and they got Apple to fix it, when it was their incorrect setting in the developer menu that caused the problem, and should have been caught before they did the click-bait declaration that they couldn’t recommend the MacBook Pro. They should have had Apple check what was wrong before such an egergious error on their part made it look like this was Apple’s fault. And then in their “correction” not one word on their fault in the tests. That’s dishonest.
But even today, they don’t admit their own error and try to make it sound like they pointed out the error to Apple who then corrected it. Misleading and in their case, damning to their business model.
They continue to deceive by not taking responsibility for their part in the error of their testing, and making it look like they caused Apple to fix a bug that was in all MacBook Pros. It’s clear deception. We’re not denying Apple had a bug in Safari. We’re criticizing how they have handled it subsequently.
Once again, CR deceives their readers by not admitting their own error and making it look like it was them compelling Apple to fix a bug.
4: They continue to avoid admitting their own part in this problem.
If they are not irresponsible, why won’t they admit their part in the error now, and try to make it all Apple?
What error? Testing with cache off is not wrong.
Apple’s published battery life test are performed with a display brightness set to 375 Nits. CR’s test are done with the display at 100 Nits.
Sure. But today they’re actually suggesting people upgrade to a beta OS… rather than just not fuck up their settings intentionally like CR did.
Why would a fix for a developer only mode (and, further, a developer only mode that is intended for short term use, and you’d have to be stupid to leave on) be pivotal in their recommendation?
It just doesn’t make sense. And they should NOT be suggesting users upgrade to a beta OS for that fix, since the fix won’t affect them. (On the other hand, saying “there may be other factors, so you may need other aspects of this update to reproduce our results” would be completely rational.)
It’s not a mode users will enter. Hell, it’s not a mode developers will enter (it’s a developer feature). The mode probably shouldn’t even exist.
But it does, and sure, it’s fair to test against it. But when a bug exists only in a mode users will NEVER use, CR should have been more forward about that in the follow-up.
Sigh. Hi, web developer here. I know what caching is and does. And turning it off is NOT representative of the real world.
IF they wanted to get a better real world simulation it should be done by leaving the client setting alone and constructing a server that simulates what people actually do… which is visit some sites several times during the day, some once. Some of the revisited sites might have minimal changes. Some might have more substantial ones. This isn’t that hard – get 10-20 sites, capture their pages throughout a weekday, build a server that emulates what happens over the course of 8 hours, run.
AND NOT configured the same way as the customers would see use the MBPs out of the box.
Cache on and this wouldn’t have shown up. That’s the point.
Weren’t you moving to Windows? Are you one of those people who just can’t quit a thing and continues to troll the sites that talk about it long after they’ve stopped actually use the product?
Gruber actually captures my feelings better than I’ve expressed them:
This makes it sound like CR found a problem with the batteries. They didn’t. They found a bug in a Safari developer mode. It’s a real bug, but it’s clear now that it didn’t justify the initial sensational “Wow, first ever Apple laptop not recommended by Consumer Reports!” report. There’s no way they would’ve published that rushed initial report for a laptop from any brand other than Apple. Clickbait, pure and simple.
All the outrage in the world at Consumer Reports still doesn’t make my 15″ Touchbar MBP get any better than about 3 hours’ worth of battery life on a charge. My brother in law reports similar usage. This issue is real, for some of us.
The only fanboys here are the ones who seem to believe that a developer setting is somehow relevant to test simulated consumer behavior.
Apple’s own statement is not much better. The fact of the matter is that there is a bug, and the so called hidden setting is not hidden. The testing methodology might not reflect common everyday use but it was not totally unreasonable. CR is CR, but we should expect better from Apple in how they deal with these situations. It goes beyond the finger-pointing statement. Their removal of the time-remaining reporting feature is indicative of poor handling of the whole situation.
CR should not be given a pass on this. But let’s leave Dalrymple to pretend Apple is not at fault as well.
It’s absolutely hidden. And we’re discussing this bug, not some grand conspiracy you see.
No Jim, you are merely attempting to cover up their duplicity.