Streaming Media:
Apple’s live stream of the unveiling of the iPhone 6 and Watch was a disaster today right from the start, with many users like myself having problems trying to watch the event. While at first I assumed it must be a capacity issue pertaining to Akamai, a deeper look at the code on Apple’s page and some other elements from the event shows that decisions made by Apple pertaining to their website, and problems with how they setup storage on Amazon’s S3 service, contributed the biggest problems to the event.
Interesting take on the utter disaster that was the streaming video of this morning’s event.
Interesting. I figured it was hackers. Korean hackers.
Trying out new things at the biggest Apple event in years was not smart.
Ignore that article. The author has no clue what he is talking about. It’s a layman’s guess at the cause of the problem and his theory is full of holes including a lack of basic understanding of the difference between JavaScript, JSON and AJAX.
The author tries to come off as an authority but he is actually quite clueless.
Yeah, I have trouble believing that anything that Apple did with their website was responsible for the feed on my Apple TV glitching, or it having a Chinese translator speaking along with everyone.
This may or may have not contributed in any way to the problem, but as my AppleTV didn’t stream correctly either from the beginning, there has been definitely a problem on the streaming side.
That still might be the right answer, but whatever it was, It’s certainly not what the article said.
Samsung was unscrupulous when they were raking in the dough. Now that they’re slumping I imagine they’d resort to any dirty trick they could think of.
I wouldn’t be at all surprised if eventually they get tied to the “iCloud leaked my pictures” brouhaha.
That article is complete and utter BS. His theory boils down to: the page had dynamic AJAX calls in it, so no part of it was cacheable. That’s just patently false. A web page is delivered in many individual HTTP requests for different parts of the site (HTML, CSS, Javascript, AJAX requests, web sockets, video streaming and whatnot). Each of these things can be delivered by different servers and can be cached or not cached individually.
Apple certainly seems to have had some fails on their side, but that was certainly not one of them.
as others in the comments here and on that site have pointed out, the author has no idea what he is talking about.
A lot of people here, and on that website, discrediting that article. And anyway, nothing in that article actually explains why the pre-event feed had 2 audio tracks playing different music styles, why we actually got Chinese translation here in the USA (he touched on that), or why the “TV Truck Schedule” kept showing up.
Goodness that event was painful to stream. Betting at least one person gets fired over it. DDoS attacks may have contributed to some of the server load problems, but the human production problems were much more serious.
Hopefully, Apple learned their lesson befor their next announcement (iPad) in ~October. I personally, think stacking all their announcements and product launches for the same time each year (SEP/OCT) is a recipe for disaster, but it’s the direction Cook & Co. have chosen. What happens when they have another Antennagate? It leaves them exposed without much time for correction before the holiday onslaught. Pissing off customers at Christmas could be a fatal blow. This video may be indicative of how Apple is cracking at the seams, trying to manage all these various announcements and product launches, simultaneously. I hope I’m wrong, but we shall see.
The Chinese translation was almost certainly Apple’s fault, but the lag and stalling were almost certainly not.
I don’t agree. I can very easily imagine it being the streaming company’s fault that the Chinese translation was going out with the wrong feeds.
Considering what a load of shit Antennagate actually was, I don’t think that’s a primary concern for them.
No matter the reason, it was inexcusable. They — whoever was the person responsible — had MONTHS to prepare and rehearse for the biggest part of the biggest announcement in years. Somebody dropped the ball in a big way. Cook should be livid. Heads should roll. Remember Steve’s reaction to the MobileMe screw-up? If they don’t, then expect creeping incompetence to grow at Apple.
I imagine Apple doubles down on secrecy when rolling heads, too. Really, we barely heard a peep about Scott Forstall and he was a name we all knew. (But what we did hear got repeated, and sounded like it was a big leak.)
A streaming failure is par for the course w/ Apple but yesterday was extremely bad.
Dan definitely clarifies everything and confirms some of my assumptions. He’s been a beast on that front for years now so I trust he’s spot on or close.
Layman? Do you know Dan’s credentials?
Apple has to push the translation. The CDN doesn’t do that.
Someone on Apple’s side had the feeds coming into a box that pushed the feed to the CDN and the CDN pushed it to everyone else. Apple blew this event on their end.
The cache/json was explained as to why the site wouldn’t load. That’s why we couldn’t get anything on the site to load properly.
If you think he’s wrong, provide your expertise and credentials on why it isn’t correct.
I don’t need credentials to know that how a website is coded is irrelevant to how the stream performed on my Apple TV.
It definitely could be them. Their upstream could’ve been slow which would put strain on the down.
He never once mentioned AJAX. He said it had dynamic json, which meant each user had to get the page served dynamically (not cached).
That’s scaling 101 problems they failed to properly address. Ajax could’ve helped them here, tbh.
You completely missed it then.
The site crashed. <– code problem
The stream sucked. <– streaming problem
If you have never done live streaming or tried to scale a site, please step aside from bashing someone who is an expert in the field.
He conflated a lot of terms regarding JSON/Javascript/AJAX there, so I’m going with the interpretation of AJAX. But, since I’ve seen it happening live, the website itself was fine! The “dynamic JSON updates” were the part that worked great! The video was a mess, which has nothing at all to do with whatever JSON/Javascript/AJAX is embedded in the page that also embeds the video.
“the website itself was fine! “
Actually, it wasn’t. I got many errors on the site trying to reload that page.
He never once said AJAX. How’d he conflate anything?
This is JSON: {property: “some value”}.
The page absolutely was not fine:
https://twitter.com/johncblandii/status/509391389465911296
https://twitter.com/dalerankine/status/509390595744231424
The video was definitely a mess. Read his explanation of why Akamai couldn’t cache the page (back to JSON) and how the player made too many requests causing a degradation in service.
Ah, right, I’m an idiot, obviously. Thanks for the clear explanation.
Dick.
Not Richard, I’m John.
I still disagree with you, but that did make me chuckle.
😀
I did this for a lot of years (biggest project was CNN Live for 3 years). That’s why when I read this article I know Dan is either spot on or very close.
Quoting:
“Apple decided to add some JSON (JavaScript Object Notation) code to the apple.com page which added an interactive element on the bottom showing tweets about the event.”
JSON cannot “add interactive elements”, he means Javascript.
“As a result, this was causing the page to make refresh calls every few milliseconds.”
The page itself did not “refresh”, if anything it made AJAX requests.
“By Apple making the decision to add the JSON code, it made the apple.com website un-cachable.”
That statement in itself is nonsense. If Apple indeed served the whole page with the JSON data embedded in it dynamically for each individual request, that would make sense. I have not confirmed this at the time, but what I experienced was that the dynamic parts where loaded after the fact, they did not exist when the page loaded. Seems more likely that it was purely fetched via AJAX. But again, we can’t really confirm this anymore at this point.
“And since Apple embeds their video directly in the web page, any performance problems in the page also impacts the video.”
That’s nonsense. If the page managed to load once, it has no impact on the video performance anymore. For me the page loaded fine, the video didn’t.
Your first link shows resources failing to load, nothing to do with JSON, and your second link is likewise some failure with the CDN. You don’t run streaming video from the same servers as your static resources or your “dynamic” content. And as far as that “dynamic” JSON: it wasn’t. There was nothing unique to any user, so that gets cached like everything else. As far as AJAX, you don’t get JSON without making asynchronous JavaScript calls. The article is pure BS.
Any thoughts about the non-server-related aspects of the streaming problems? The fact that they had two competing musical soundtracks during the pre-show portion, went to color bars several times early on, and there was a Chinese (I think) translation clearly audible for the first 40 minutes or so?
This is disappointing. I’m going to copy and paste a couple of my posts from over there. First, What Really Happened™. With no inside knowledge, it looked like a run of the mill CDN failure. Why is that surprising? They hyped this thing up as the biggest thing since the iPhone, what with the ABC exclusive and everything. All of the problems are easily explained by an overwhelming amount of traffic.
• Couldn’t load the page or page resources.
Simple CDN problem. The server couldn’t keep up with requests.
• Didn’t get updates or updates failed to load
Simple CDN problem. JSON is a page resource, see above.
• Weird access denied page.
Simple CDN problem. The server crashed or was otherwise overwhelmed.
• Couldn’t load video.
Simple CDN problem. The client couldn’t load the HTTP Live Streaming playlist file because the server couldn’t keep up with requests.
• Video died while playing
Simple CDN problem. The client tried to load the next video file in the playlist over HTTP and the server couldn’t keep up with requests.
• Video skipped around
Simple CDN problem. The client has to reload the playlist file during streaming and got a stale version, causing it to load earlier video files.
• Chinese audio
That’s on Apple. They either fed the wrong stream to the CDN (wrong input) or assigned the Chinese stream to English clients (wrong config).
Granted, I wasn’t “Inside Apple’s Live Event Stream”, but sometimes the obvious answer is the correct answer.
It would seem that his credentials don’t translate into knowledge. Anybody who refer to JavaScript AJAX calls to fetch JSON content as “JSON Code” lacks a knowledge of the basic vocabulary of this discipline.
Further, the content that was dynamically loaded via JavaScript to add to the area below the stream was NOT dynamic at all and 100% cacheable. You simply put another snippet of static of HTML in the bucket and the next polling request from the browser will fetch the same one that millions of other people would fetch — they were not personalized snippets of content. The page was not being dynamically generated on the server or else it would be constant refreshing and the stream would have reset with a page flash every time the page loaded.
Maybe you feel this guy knows what he is talking about, but I’ve been writing web applications since 1996 and was doing asynchronous fetch and dynamic page updates using hidden iframes and forms before they even called it “AJAX”.
Ultimately, Apple screwed this up and they are responsible, but trying to pin it on an automatic AJAX call that likely polled for updates every 3 minutes and could have been served from a cache almost instantly seems like a poke in the dark. What’s more, neither Akamai nor Amazon are going to provide Dan the Man with actual customer’s configuration because that would completely violate their contract with the customer. The mere fact that he called and asked shows a lack of knowledge of how these things work.
Don’t always believe the “credentials” — a lot of Bozos in the industry have plenty of “credentials”. This article of his exposes him as a Bozo.
Do you even know what you are talking about? Please define “Dynamic JSON” without AJAX? Are you saying the server was constantly generating dynamic JSON content and it was magically being transported to a non-refreshing browser page without an AJAX call involved?
The mere fact he did not mention AJAX or JavaScript but called it JSON is why he comes off as a bozo.
I think your level of knowledge on this subject is close to that of the author of the referenced article if you don’t see why AJAX is being referenced here.
The live stream sucked. It was Apple’s fault. But the authoritative tone of the article referenced while spewing out misinformation is laughable.
Your comments here defending it seem equally laughable. JSON doesn’t magically appear on a web page and mess things up. JSON is a transport format, similar to XML, but faster to parse by a JavaScript client.
JSON has replaced XML in 99% of all AJAX calls on the web. And AJAX should be renamed to AJAJ since the “X” is now a misnomer. Your reply looks like you just googled “JSON” and you are refuting references to “AJAX” because you believe that JSON was used instead of XML. But anybody in this industry who writes dynamic Web apps knows the vocabulary, and they know that JSON is the accepted transport format for request and response data from an AJAX call (which is incidentally done in JavaScript).
Dan completely misunderstands the vocabulary of dynamic web applications, as do you. Stop trying to come off like an authority when you don’t know what you are talking about.
Apple had live stream issues. Their last three events came off without a hitch (yet you say “par for the course”). It may have been too much traffic or may have been something they did that was conflated by too much traffic.
Either way it is on Apple to ensure they have enough bandwidth if they are going to hype an event that way.
But Dan’s article only confirms he is a poser Bozo who does not understand how Dynamic Web Applications function. The JSON content which was retrieved via AJAX calls via periodic polling from the browser client was the same content for every user. You put a static snippet of HTML in a bucket with a timestamp and you serve the same thing to everybody when their browser requests any updates since timestamp X.
The web page itself was dynamic. The JSON was pretty much static. The server side content could have been cached and served from multiple mirrored CDNs.
What Apple did wrong, none of us will know without looking at their actual configuration and their code. But to try to pretend that you know while failing to understand the difference between JSON, JavaScript and AJAX is beyond conceit.
That seems like a far more plausible theory than the periodic AJAX polling for static JSON content bringing the CDN to its knees.
If Apple’s upstream from Flint Center was somehow congested then it would have messed with the downstream which was trying to download data faster than Apple provided it. This could also theoretically cause the loop-back issue.
This is the most sensible comment you have made here. But your praise for analysis in the referenced article is severely misguided.
Sure you get JSON without ajax requests.
Here is dynamic, server generated JSON in a couple languages:
Ruby – https://gist.github.com/johncblandii/b7ca60989696e58adc46
PHP – https://gist.github.com/johncblandii/4d724e71b5def25cd885
Of course your static resources are on a different server. That’s obvious. Did someone say otherwise?
Caching them on a CDN is very common though.
LMBO @ googling JSON and XML. What in the world are you talking about? When did I ever mention XML? 😀
You’d be shocked to know how wrong you are about the 99%. XML is still heavily used (see RSS and a boat load of legacy SOAP).
If you’re so upset about him being authoritative, why are you doing the same? Oh great BC2009, explain to us exactly what happened since this opinion piece is so wrong.
(I’ll wait)
Makes some sense given how fast Samsung got this ad out there. http://youtu.be/vA8xPyBAs_o
Yeah, the twitter stuff would’ve definitely been done via AJAX requests. That won’t affect scale since none of that needs to be dynamically rendered in the html.
I’m more than sure the live blogging stuff was as well but I was having too much fun to care to look at their source/network requests.
Any request can affect the player. Having built numerous players in the past with similar functionality, I can 100% tell you network conditions can make the player think the connection is slower than it really is.
Well built players control the stream quality, when VBR (variable bitrate) is in use, through the following quality of service (QoS) metrics:
– CPU usage
– Network quality (bandwidth detection)
– Packet loss
– etc, etc, etc
I only spit out acronyms for googling purposes (nothing more), in case you were curious.
So…with all that said, you take a lot of long running requests, high cpu use, etc, etc and the player could think you need a lower quality, subsequently drop you and cause a lot of stream switching on the CDN.
NOW…I have zero idea if that is what happened, TBH. If you look at my original comment, I said he’s either “spot on or very close”. It seems he most likely meant Javascript in some places for sure though.
Yeah, ’cause I’m new to the web: http://www.amazon.com/John-C.-Bland-II/e/B002G57DZ2/ref=sr_ntt_srch_lnk_1?qid=1410408587&sr=8-1.
Have a good one BC.
That was pretty bad. I thought I had issues only on my end but it turned out everyone I followed had the same problems.
The translation seemed like a player issue, at first, but it autocorrected without a player refresh. That means it was an issue within the stream for sure and most likely from their encoder, which they could correct without any users refreshing.
TBH, they probably just shut off the translated feed. There was still a quiet one in the background (male voice) you could here for a while after the loud one turned off.
You’re right on the “no inside knowledge” part. I read it as an opinion piece, as that’s mostly what I read from Dan, but clearly others are taking it as a major issue.
My thoughts on your points:
• Couldn’t load the page or page resources.
Doubt it. This is a terribly simple page without any need for long running requests (#1 cause of scaling problems). Amazon has handled much worse (see his explanation of the S3 bucket).
• Didn’t get updates or updates failed to load
You can almost guarantee the JSON was not a simple page resource. This was most likely dynamic data [ex – the live blog data].
• Weird access denied page.
Yep.
• Couldn’t load video.
It’s a static file. Akamai has zero issue serving these playlists.
• Video died while playing
Possible.
• Video skipped around
It’s a live stream. I doubt they had recording on or dvr (which saves files as well) since it would be a lower quality than they get at the encoder. The skipping is most likely due to packet loss.
• Chinese audio
Yep, on Apple. For sure an encoder issue.
Server generated JSON. The JSON could be in the head tag with any amount of info. If they did that, it’d be a pretty terrible failure.
I’m only talking about the initial config, things like configuring the player. I didn’t look to see what player they were using but it needs configuration, most configure through JSON now.
^ speculation
Oh and you’re prob right, I don’t know Dan’s full history but he never struck me as a developer. He’s a streaming expert and that’s why I focused on his thoughts on the stream issues, not the web dev parts.
CDNs are, pretty much, dumb pipes. All they handle is caching and routing to edge servers. They send out what you send in.
They are not infallible but Akamai is #1 for a reason.
Upstream is the most important piece of streaming. It has to be right from the encoder or it will suck otherwise.
I never praised the article. I never took it as the truth either. Y’all just went crazy like the stuff he said was absolutely impossible and attacked his character/rep. That’s just bonkers to me so I provided another view.
[tired of responding, tbh, taking too much time; lol]
LOL. They have like 6 videos (most they could’ve prepared from the leaks but some had to be fast work).
Oh Sammy! 🙂
I thought it was obvious, but you don’t get JSON on the client without AJAX. You can generate it on the server with whatever you want. But nothing gets to the client without JavaScript.
Now that’s a much more realistic and nuanced theory than the original. TBH, I’d hope Apple engineers wouldn’t be so amateurish as to kill their own video player simply by overloading the client’s capacity with super inefficient AJAX updates. Also, that theory still doesn’t explain the reported problems on Apple TV; so it’s rather unlikely as well.
But at least it’s an internally more consistent theory, unlike “nothing was cached because JSON”.
Look, you clearly don’t know what you’re talking about, so please just don’t reply to my comments. It wasn’t a “simple page”, in that it had CSS, JavaScript, and image resources. You don’t seem to know what “long running” means (nor do I in your context: slow client? Additional async calls? COMET?). Each JSON document you load is a part of the current page, just like any other file you load. They are all resources.
You still don’t understand “dynamic data”. If the client isn’t providing unique input and doesn’t require unique output, then it is simply recent data. Which servers cache.
The m3u8 file is never static for a live stream. Whoever served it absolutely did have trouble, in that they apparently kept sending out stale versions. Don’t talk about HTTP Live Streaming until you at least read http://tools.ietf.org/html/draft-pantos-http-live-streaming-13. And that goes for all of your video-related suggestions.
No, the Chinese audio wasn’t an encoder issue. It was encoded correctly for Chinese. They just weren’t sending the English encoding to English speakers. So, like I said, that is a config problem, either on the input or the CDN.
It’s funny, because as Dan himself tweeted, his earlier article regarding a previous Apple streaming failure was far more accurate.
Like I said, I’m speculating as much as anyone else, but this is what I do.
“I thought it was obvious, but you don’t get JSON on the client without AJAX.”
Absolutely wrong. Server generated JSON can exist on an html elements attribute (no JS required). Now, you’ll need JS to do something with it but AJAX is not required for JSON.
And are you putting that in a ‘script type=”text/javascript”‘ tag? Then it’s called JavaScript. If you aren’t, it’s called garbage. Even if you’re putting it in an HTML element’s “attribute”.
True, the Apple TV would’ve been a straight streaming problem. Since the slate showed it was an TV Truck, I’d bank on upstream issues causing the problem.
Long running processes are server processes (in apache, nginx or whatever they are using) caused by slow db queries, etc).
I never said a “simple page”. Calm down and read it again.
m3u8’s are definitely static. I’ve done live streaming at a high level for 7 years (building players and deploying/customizing streaming servers). No need to read more about something I’m well versed in.
An m3u8 can, in simplest form, contain a link to the source stream. Clients send me them all the time.
Translation was a config issue? Ok…your opinion and plausible but that config is handled through the encoding process.
Happy to see you at least have some experience, although I don’t see live streaming but I’ll take your word.
Nope, you can put it anywhere on the page then simply access it.
https://gist.github.com/johncblandii/e5d5bb16fa1815fa3bc4
It’s a simple way to get data to javascript, more often than not, when you may be at a different level in your code and unable to add it in the head and don’t want some random, top level variable your js for that section can’t access.
Definitely not garbage. Common practice to use data attributes for passing data. Whether you want to stuff a full json object or not is a matter of preference.
Consider the low budget badly made commercials, they probably started at 10:30am
The problem with the article is that the author assumes he knows what went wrong. Anybody who is knowledgeable at debugging these kind of performance issues understands that you need access to server configuration and code to diagnose these kinda of things. Making random theories and then try to support them with statistics is like trying to measure the thickness of an iPhone with a grade-school ruler rather than a micrometer and giving your results out to 5 decimal places.
I’ve seen impressive resumes (including references to authored books) only to find the person I am interviewing does not know nearly as much as their credentials indicate.
For all I know, you are knowledgeable, but your comments on AJAX and JSON above makes it look like you were completely missing the other commenters point because you did not get how the two are used together.
Maybe you just misread or misunderstood the point the other guy was making.
I agree on the “here’s what happened” bit. It’s purely speculation and yeah…Apple will never tell.
It just seems what he wrote is plausible.
I know how they work together (use them daily) but neither requires the other. They were simply wrong.
True on resumes. I put out requests for contractors this week and told 3 people I refuse to read their resumes. They’re simply not a worthy gauge of someone’s talent [hence me not sending my LinkedIn]. 🙂