On Friday, we posted news about Consumer Reports very publicly changing their Microsoft Surface opinion to “Not recommended”. From our post:
Take this change with a grain of salt, give the experts a chance to dig through the methodology. But no matter the ultimate resolution, this certainly gives a black eye to Microsoft’s Surface brand.
Over the weekend, I ran across this take from Ben Bajarin (it’s from a subscription-only newsletter, but Ben graciously gave me permission to repost it here):
I want to briefly comment on a consumer reports article that came out yesterday. CR officially does not recommend Microsoft Surface hardware based from their interviews and survey of 91,741 laptop and tablet owners. They claim 25% of Surface owners reported a form of breakage after a few years.
What bothers me most about this is the flawed methodology. Surface sales are not huge, and the installed base is relatively small. So even though CR covered over 91,000 people, there is no way they had what any respectable statistician would consider a representative sample. The fact CR is not further detailing their methodology or telling us how many Surface owners they had is a red flag. Based on some other source data I’ve come across, I’d bet CR had less than 50 people respond who owned a Surface and it was probably more like 30. So 25% of my estimate means five people had an issue with Surface after two years.
The flawed methodology is disingenuous, and sadly Consumer Reports has become a bit of a click-bait outlet. They do this with Apple and now with Microsoft. Media outlets endorsing the CR report and flawed methodology only further affirm that CR can do this in the future. Which means, for all the Apple fans in my Twitter timeline mocking Microsoft and Surface based on this flawed process by CR, this may come back and bite you when CR needs another click-bait headline and targets Apple with their disingenuous agenda and methodology.
Personally, I’m not a fan of Windows, but I am a fan of fairness. When I saw the Consumer Reports flip, something just felt off. Consumer Reports has an outsized reputation, which gives them clout. And, as we all know, with great power comes great responsibility.
If Ben’s take is correct, I’d love to see Consumer Reports either completely reveal their methodology or take the change back and redo their testing in a way that would stand up to public scrutiny.
Precisely why I stopped support Consumer Reports over 30 years ago.
Good Lord, but tech journalism really fails miserably where Consumer Reports is concerned.
With the Apple laptop test, they used a perfectly reasonable methodology and 1) reported their findings, 2) reached out to the manufacturer to get help figuring out what was going on, and 3) announced the results of that investigation pretty quickly (and note that it was Apple’s fault, not CR’s).
What were they supposed to do? Not report their findings? Cover them up?
With this case, the situation is similar: they got some results, reported them and gave the manufacturer a chance to comment (note the weaselly way that MS responded: “We don’t believe these findings accurately…capture the performance and reliability improvements made with every Surface generation.” (ie the first generation was terrible but we’ve gotten better). Note also that CR didn’t just run out and report the results after the first year. They waited: “Microsoft is relatively new to the hardware business, and this is the first year CR had enough data to estimate predicted reliability for the company’s laptops.”
And now random guy (Ben B) comes along, speculates wildly about stuff (number of folks they had, etc), demonstrates he knows nothing about getting a representative sample (“there is no way they had what any respectable statistician would consider a representative sample.” Oh, ye Gods.), and everyone is jumping up and down.
So — answer me this: you’re CR, you get these results on a product that you’ve RECOMMENDED (ie you’ve said is a good product). What do you do?
(Note: I have no connection with CR; I don’t have a subscription to CR etc etc.)
There’s no secret about the methodology. Subscribers to CR receive these questionnaires where they can report their experiences with a number of consumer products, from cars to computers to washing machines. They check boxes that describe those experiences in broad generalities.
Obviously, such a survey is deeply flawed since no effort is made to reach a selected demographic sample as polling companies do.
I agree that the number of Surface users in this batch has to be extremely small. Don’t take the reliability rating of your car seriously either.
Peace,
Gene Steinberg
While I’m all for fairness, Bajarin is engaging in rank speculation. It does not make me think well of his capacity for reporting or analysis.
Obviously, such a survey is deeply flawed since no effort is made to reach a selected demographic sample as polling companies do.
Er, that’s not how polling works and let’s not propagate this misunderstanding of how surveys happen any further. Pollsters absolutely have methods for dealing with a non-representative sample; they weigh the responses so that the sample as a whole represents the population. CR absolutely does this (they talk about using “national probability samples to accurately represent the entire U.S. population” in their ‘How We Survey’ page).
He also doesn’t consider that many folks have returned Surfaces because of issues.
My dad is a Windows user, so we tried to get him a Surface Book. First one in the MS store locks up during setup(!) so they have to get a second one from out the back.
We take it home. Over the next few weeks we run into multiple distinct reliability issues: stylus is not recognized; display does not reliably detach; and worst of all, the sleep-wake issue. All are issues discussed extensively on MS forums. The last one was incredibly widespread with Skylake Surfaces for over a year as MS tinkered with the firmware to improve things. This reached an uproar and there was a lot of press around this, with MS admitting they hadn’t gotten it right.
My dad returned it and we got him a X1 Yoga instead. Much more reliable device.
I get this is a sample size of 1 (2?) but I’ve heard plenty of reports like this. Unfortunately, this is very hard to get data from if you’re not MS, and they’re not going to be forthcoming about this.
Ben is not a random person, he’s been an industry analyst for some time. Nevertheless, I agree with the rest of your observations.
Also important to take into consideration is what they do when it breaks. May depend on where you got it, but as our Surface Pro 3 was clearly bricked for no discernable reason (no damage, cracked screen, internals looked normal) Best Buy provided us with a new one. The version we had was no longer available so we were given the choice of a different version of the 3 Pro or a 4 (non-Pro) plus some store credit, as it was cheaper. We chose the latter and it has worked great ever since.
Sure, it was a pain to have to deal with, and we were without it for a couple days, but generally the situation was handled very well and I have very few complaints despite the fact that it broke in the first place.
I stand corrected for having taken that bait.
CR’s methodology was reasonable-sounding, at first.
https://daringfireball.net/linked/2017/01/12/consumer-reports
The hysterical thing is that Gruber then goes on to say “There’s no way they would’ve published that rushed initial report for a laptop from any brand other than Apple. Clickbait, pure and simple.”
Looks like he needs to amend that comment.
Looks to me like CR tried it first with Apple. It got them the desired attention, so they expanded their scope.
I expect Gruber will continue to comment on this.
Yeah, no. Looks like they stayed consistent to reporting their results.
But they’re not talking about customer service. They’re simply talking about reliability and you were an example of how bad that is. I mean, kudos to BB for taking yours back and doing the right thing but the issue under discussion is that “…our Surface Pro 3 was clearly bricked for no discernable reason…”.
If that’s true, fine. But it doesn’t address the small sampling concerns, or the fixed and highly general nature of the questions that fail to deliver the sort of specificity you need to truly judge a product’s reliability.
But it’s also been reported that a very high number of Surface units have been returned to Microsoft, and that does indicate a potential problem. Janak’s message also provides anecdotal confirmation, but he’s not alone.
Peace,
Gene
But it doesn’t address the small sampling concerns,
The only small sampling concerns I’m aware of are the completely speculative ones raised by Bajarin, who has no actual evidence behind them. So, unless you have better evidence than that?
xed and highly general nature of the questions that fail to deliver the sort of specificity you need to truly judge a product’s reliability.
I don’t know what that means. Perhaps you could explain specifically?
Generally speaking, perhaps you could not speak so certainly (“such a survey is deeply flawed”) when it turns out you haven’t really done much investigating at all?
There is a lot of selection bias in the survey – and this is coming from a recovering statistician. And anecdotally, I’ve know lots of people who genuinely like their Surface Pros/Laptops. Personally not my cup of tea it seems to work for them.
Lastly, this is Consumers Report after all. If they had their way we’d all be driving baby poop brown Dodge Darts with 8 tracks blasting Ace of Bass. But I digress…
<
blockquote>my roomate’s mother in law gets $86 an hour on the internet. she’s been unemployed for five months.. last month her pay was $16631 just working from home for 2 hours every day… ➤ look ➤ at➤ this➤ site
➜➜➜http://www.GoogleFinancialCashJobs10GreatCare/Home/Wage….
❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂❂::::qj10.
And running WordPerfect in MS-DOS on a Compaq 386.
There is a lot of selection bias in the survey – and this is coming from a recovering statistician.
A recovering statistician should know better than to talk confidently on a survey’s methodology when that methodology hasn’t been released yet.
<
blockquote>my roomate’s mother-in-law gets $95 an hour on the internet. she has been unemployed for ten months but last month her pay was $12291 just working from home for a few hours every week. ➤check ➤out ➤this
➜➜➜http://www.GoogleFinancialCashJobs38SolutionGroup/Home/Wage….
❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉❉::::ae38