A quick search for "latency" in here has one little hand-wavey blurb about Mux working to optimize HLS.<p>>Using various content delivery networks, Mux is driving HTTP Live Streaming (HLS) latency down to the lowest levels possible levels, and partnering with the best services at every mile of delivery is crucial in supporting this continued goal.<p>In my experience, HLS and even LLHLS are a nightmare for latency. I jokingly call it "High Latency Streaming", since it seems very hard to (reliably) obtain glass-to-glass latency in the LL range (under 4 seconds). Usually Latency with cloud streaming gets to at least 30+s.<p>I've dabbled with implementing WebRTC solutions to obtain Ultra Low Latency (<1s) delivery but that is even more complicated and fragmented with all of the browsers vying for standardization. The solution I've cooked up in the lab with mediasoup requires an FFMPEG shim to convert from MPEGTS/h264 via UDP/SRT to MKV/YP9 via RTP, which of course drives up the latency. Mediasoup has a ton of opinionated quirks for RTP ingest too, of course. Still I've been able to prove out 400ms "glass-to-glass" which has been fun.<p>I wonder if Mux or really anyone has intentions to deliver scalable, on cloud or on prem solutions to fill the web-native LL/Ultra LL void left by the death of flash. I'm aware of some niche solutions like Softvelum's nimble streamer, but I hate their business model and I don't know anything about their scalability.
Good overview of all the parts involved! I was hoping they’d talk a little more about the timing aspects, and keeping audio and video in sync during playback.<p>What I’ve learned from working on a video editor is that “keeping a/v in sync” is… sort of a misnomer? Or anyway, it <i>sounds</i> very “active”, like you’d have to line up all the frames and carefully set timers to play them or something.<p>But in practice, the audio and video frames are interleaved in the file, and they naturally come out in order (ish - see replies). The audio plays at a known rate (like 44.1KHz) and every frame of audio and video has a “presentation timestamp”, and these timestamps (are supposed to) line up between the streams.<p>So you’ve got the audio and video both coming out of the file at way-faster-than-realtime (ideally), and then the syncing ends up being more like: let the audio play, and hold back the next video frame until it’s time to show it. The audio updates a “clock” as it plays (with each audio frame’s timestamp), and a separate loop watches the clock until the next video frame’s time is up.<p>There seems to be surprisingly little material out there on this stuff, but the most helpful I found was the “Build a video editor in 1000 lines” tutorial [0] along with this spinoff [1], in conjunction with a few hours spent poring over the ffplay.c code trying to figure out how it works.<p>0: <a href="http://dranger.com/ffmpeg/" rel="nofollow">http://dranger.com/ffmpeg/</a><p>1: <a href="https://github.com/leandromoreira/ffmpeg-libav-tutorial" rel="nofollow">https://github.com/leandromoreira/ffmpeg-libav-tutorial</a>
Good top-level summary of an extremely complicated subject.<p>The containers diagram is surprising since the arrow means "derived from" - the earlier formats are at the top, I initially thought the arrow was in the wrong direction. Containers are kind of a nightmare since there is a lot of hidden knowledge you need. Many of the tools will let you put any video in any container, but only certain combinations actually work properly across the various devices and operating systems you will actually need to use. It's easiest to stick to "H264 + AAC in an MPEG4 container".<p>HLS is a horrible hack that also has the side effect of making it harder to download and store videos.
Not to judge the content quality but I'm curious about the motivation to build this website. Why? Because in the recent days we were discussing the poor quality Google search results and one argument is that the results are poor because the quality of the content has degraded due to the motivations of the content creators, i.e. optimising everything for SEO and views, likes. etc. with guidance of analytics.<p>So how this works? Is the creator of the website doing SEO here? Will this be sold later? Is it a project for a portfolio? Why are we getting this good quality content? Will it be surfaced by Google?<p>Not that I suspect anything nefarious, just curious about how original content(beyond commentary on social media) is made these days. What made that content possible? Why the creator(s) spent time to create graphics and text and payed for a domain name and probably hosting?
Does all the 1.7GB of the decoded video get copied to the GPU? Or is there some playback controller that knows how to read the “delta format” from the codecs and only copies deltas to the framebuffer?<p>It still blows my mind that we can stream video at 60FPS. I was making an animation app that did frame-by-frame playback and 16.6ms goes by fast! Just unpacking a frame into memory and copying it to the GPU seemed like it took a while.
What I'm curious about is the actual hardware behind video sharing sites. Like how can Streamable,Reddit,Twitter encode such a massive amount of videos at scale. Do they have GPU farms? Dedicated hardware encoding hardware that us mortals can't buy? I left out YT on purpose because they have practically endless money to throw at the problem.
Slightly OT, but the OP links to this video which I found very interesting:<p><a href="https://www.youtube.com/watch?v=VQOdmckqNro" rel="nofollow">https://www.youtube.com/watch?v=VQOdmckqNro</a><p>In it two guys discuss how to reconstruct an audio file from its image representation, and it turns out to be pretty straightforward.<p>In the end they are discussing the legal implications of being able to reconstruct audio from an image: if you buy the rights to the image does it give you the right to the audio (probably not!)<p>But what it makes me wonder is how one could maybe draw sound, or have some kind of generative art program that could be used to first draw a wave and then listen to it. Maybe this has been done already?
This is a really good introduction to how video works. The only think I find missing from a production OTT environment is DRM. Also I assume when the author says "http" he means "https" there are very few (I knew only one and even they used https termination by Akamai) providers who use "http" these days for streaming even with DRM.<p>If anyone is building an economically viable OTT platform they should consider building their own CDN. Insiders tell me the Singapore based OTT provider HOOQ went bankrupt paying huge bills to AWS and Akamai.
As good as thread as any to ask: I have a lot of footage that is recorded continuously over several hours. Most of the time nothing notable happens in the video, but I have a set of timestamps of interesting events. (Think security camera with motion detection events, though this is not that.)<p>Currently I extract the frame at each interesting timestamp and serve them as JPEGs. I have a web-based playback UI that includes a scrubber and list of events to jump to a particular timestamp.<p>I would love to upgrade to a real video player that streams the underlying video, allowing the user to replay footage surrounding a timestamp. I have to be able to reliably seek to an exact frame from the event list, though.<p>I've been looking for something web-based, self-hostable, that doesn't require I transcode my media up front (e.g., break down into HLS chunks). I have few users accessing it at a time so I can transcode on the fly with some caching (though I think it is already H.262 or H.264). Is there anything suitable out there?
A quibble I have with this is introducing video as capturing RGB.<p>Digital video very rarely uses that color space. Instead, the YUV family of colorspaces is far more common.<p>Further, the FPS is somewhat wonk. While computer games and some british TV will broadcast at 60fps, most video is instead at 20->30fps.
That's a really nice introduction. One newbie question: how is this influenced by DRM on the browser? Is it all the same plus some security on top or do videos with DRM use a proprietary codecs and packaging?