Scaling up the Prime Video audio/video monitoring service and reducing costs

My word. I&#x27;m sort of gob smacked this article exists.<p>I know there are nuances in the article, but my first impression was it&#x27;s saying &quot;we went back to basics and stopped using needless expensive AWS stuff that caused us to completely over architect our application and the results were much better&quot;. Which is good lesson, and a good story, but there&#x27;s a kind of irony it&#x27;s come from an internal Amazon team. As another poster commented, I wouldn&#x27;t be surprised if it&#x27;s taken down at some point.

I&#x27;m pretty convinced that microservices are one of those things that make sense 5% of the time and the other 95% is cargo culting.

AWS has a great business model of people over &quot;optimizing&quot; their architecture using new toys from amazon and being charged through the nose for it. It&#x27;s amazing how clients that are doing a few requests per second will want a fully distributed, serverless, microservice + dynamodb + s3 + athena + etc + etc, in order to serve a semi-static web app and print some reports off throughout the day and pay 10-50k a month when the entire thing could run on a few nodes and even a managed RDS instance for a thousand bucks a month. I would argue at this point that early optimization of architecture is astronomically worse than even* your co-worker that keeps turning all of your non-critical, low-volume iterable functions into lanes to utilize SIMD instructions.<p>Some irony in my anecdotal experiences is that most places that don&#x27;t have the traffic to justify the cost of these super distributed service architectures also see a performance penalty from introducing network calls and marshaling costs

This really is a click bait title. They are talking about their video quality monitoring service, not their video streaming service.<p>It’s something they use to check for defects in the video stream - hence the storing of individual frames in S3.<p>Original title: Scaling up the Prime Video audio&#x2F;video monitoring service and reducing costs by 90%

&quot;We built a video stream processor by splitting every 1080p+, multi hour long, 30-60fps video into individual images and copying them across networks multiple times.&quot;<p>Not surprising that didn&#x27;t go will. This strikes me as a punching bag example.<p>Anyone who has worked with images, video, 3d models, or even just really large blocks of text or numbers before (any kind of actually &quot;big data&quot;) knows how much work goes into NOT copying the frames&#x2F;files around unnecessarily, even in memory. Copying them across network is just a completely naive first pass at implementing something like this.<p>Video processing is very definitely a job you want to bring the functions to the data for. That is why graphics card APIs are built the way they are. You don&#x27;t see OpenGL offering a ton of functions to copy the framebuffers into ram so you can work on them there only to copy them back to the video card. And if you did do that, you will quickly find out that you can be 10x to 100x more efficient by just learning compute shaders or OpenCL.<p>You could do this in a distributed fashion though, but it would have to look more like Hadoop jobs. I predict the final answer here, if they want to be reasonably fast as well, is going to be sending the videos to G4 instances and switching the detectors over to a shader language.<p>In general, if the data is much bigger than the code in bytes, move the code, not the data.<p>IO is almost always the most expensive part of any data processing job. If you&#x27;re going to do highly scalable data processing, you need to be measuring how much time you spend on IO versus actually running your processing job, per record. That will make it dead obvious where you should spend your optimization efforts.

This is not a discussion of monolith vs serverless. This is some terrible engineering all over that was &quot;fixed&quot;.<p>Some excerpts: &gt; This eliminated the need for the S3 bucket as the intermediate storage for video frames because our data transfer now happened in the memory.<p>My candid reaction: Seriously? WTF?<p>I am honestly surprised that someone thought it was a good idea to shuffle video frames over the wire to S3 and then back down to run some buffer computations. Fixing the problem and then calling it a win?<p>But I think I understand what might have lead to this. At AWS, there is an emphasis on using their own services. So when use cases that don&#x27;t fit well on top of AWS services come up, there is internal pressure to shoehorn it anyway. Hence these sorts of decisions.

Next they will transition to on premises hardware from the cloud to save another 90%.... oh wait...

The title is editorialised to be clickbait. The original title is &quot;Scaling up the Prime Video audio&#x2F;video monitoring service and reducing costs by 90%&quot;.<p>They changed a single service, the Prime Video audio&#x2F;video monitoring service, from a few Lambda and Step Function components into a &#x27;monolith&#x27;. This monolith is still one of presumably many services within Prime Video.

I wish this was a good condemnation of microservices in a general use case but it is very specific to the task at hand.<p>Honestly, the original architecture was insane though. They needed to monitor encoding quality for video streams so they decided to save each encoded video frame as a separate image on S3 and pass it around to various machines for processing.<p>That is a massive data explosion and very inefficient. It makes a lot more sense that they now look for defects directly on the machines that are encoding the video.<p>Another architecture that would work is to stream the encoded video from the encoding machines to other machines to decode and inspect. That would work as well. And again avoid the inefficiencies with saving and passing around individual images.

This is less an example of why serverless was bad but rather an example where using non-suitable services for tasks they were not meant for.<p>In this case they were using AWS Step functions that are known to be expensive ($0.025 per 1,000 state transitions) and they wrote: &gt; Our service performed multiple state transitions for every second of the stream<p>Secondly, they were using large amounts of S3 requests to temporarily store and download each video frame which became a cost factor.<p>They had a hammer - and every problem looked like a nail. In my experience this happens to every developer at a certain stage when he&#x2F;she gets in touch with a new technology; it doesn&#x27;t mean that the tech itself is bad - it depends on the scenario, though.

Sending video frames between services is expensive, also doing per state transition hosting on things doing state transitions multiple times per second in a single stream is also expensive...<p>Like, did they even think about cost when designing this the first time?

&gt; The main scaling bottleneck in the architecture was the orchestration management that was implemented using AWS Step Functions. *Our service performed multiple state transitions for every second of the stream*(???), so we quickly reached account limits. Besides that, AWS Step Functions charges users per state transition.<p>This is so obvious in my head. I can&#x27;t think of a single good reason where a SFN makes sense here.

I&#x27;d be surprised if this doesn&#x27;t get taken down as it casts AWS lambda in an unfavorable light (and rightly so). That&#x27;s the impression I have of Amazon&#x27;s leadership but maybe I&#x27;m wrong.

Storing individual frames in S3??? Insanity! Their initial distributed architecture is unbelievable.

Dead horse and all that but please just stick to Boring Tech, it is better for your mental health, not to mention your business, development velocity, defect rate, etc.<p>Most importantly it&#x27;s good for mental health though.

Microservices are just an architectural pattern, and like all patterns there are places where they are highly appropriate, and others where they are inappropriate.<p>Same for cloud, same for &lt;pattern&gt;<p>If everything is a hammer you&#x27;ll hurt your thumb&#x2F;hand&#x2F;arm.<p>At least now (for some time) the pattern is named, so broadly when talking about this sort of thing, the name conjures up the same&#x2F;similar image in everyones heads.<p>There are all sorts of inputs to the choice of architectural patterns, including budget, scalability (up and down), criticality, security, secrecy, team skills and knowledge, preference, organisational layout, organisation size, vendor landscape, existing contracts, legal jurisdiction ....

As a former AWS employee I can almost guarantee that the person that made the original design got a promotion over it.

Clickbait title. The expensive part was passing around individual frames and the associated S3 operations. It&#x27;s not clear if they could&#x27;ve kept a distributed architecture but made the work units be chunks of frames or even whole videos. Monoliths can inefficiently use S3 and other cloud services to rack up a huge bill.

I don&#x27;t want to come off too harsh on this, but it sounds like the service didn&#x27;t meet the initial design requirements?<p>Some of this would have been really easy to predict (eg. hitting account limits) if they simply took the time to calculate how many workflow transitions they&#x27;d need to execute for the load.

If you came to me with a design that included passing individual video frames through S3 instead of RAM I would honestly think you were joking. What a wild article.

AWS is truly a customer first company. I been AWS customer in its early days (2006-2012) and then recently (2022-now). And they have been consistent in being customer-first. In the last year, they have proactively helped us cut our AWS spend by multiples. I&#x27;m not surprised at all by this article coming from within Amazon. Kudos for maintaining such a culture.

The headline is a bit of a misnomer. This happens in large businesses all the time (which isn&#x27;t to say it&#x27;s &quot;good&quot;, hardly is, but it suggests the causation is incorrect here, which then indicates the conclusion is entirely off-base):<p>1) We have sexy new product! Everyone use it so we have some use-case stories to tell and we look credible! Who cares if it&#x27;s not the right tool for the job! We need a splashy way to use hackneyed business speak like &quot;we&#x27;re eating our own dog food&quot; at the next user con so all the IT middle managers there will fight over early access and adoption. PROFIT! (Screams of technology teams in the background of &quot;a knife is the most expensive, useless pry tool you can buy, but whatever, you are not listening, mmmkay&quot;).<p>2) A few quarters&#x2F;years later (if you&#x27;re lucky and you made it or someone with enough gravity in their title finally saw the light): Why is expense so high in this business unit? This is insane! Let&#x27;s go back to a more sane architecture. (Screams of technology teams going back to what was working in the first place, but was not sexy nor necessarily new now that no one is watching and hype cycle is over)<p>Does this mean that serverless is useless? Dumb? Uneconomical? No way. For bursty, very short running workloads, it can be GREAT and INCREDIBLY economical.<p>What is useless and &quot;dumb&quot; is whomever thought that Prime Video&#x27;s encoding workloads were going to do anything but increase cost and were somehow a fit for a system whose business case specifically necessitates bursty, shorter workloads that are primarily scale-to-zero for significant periods of the day&#x2F;week&#x2F;month.<p>It was a marketing stunt gone horribly wrong: intentional or not, but that doesn&#x27;t repudiate the value of &quot;serverless&quot; for the right workloads, it just proves you better really understand the technology and the business case and the scale economics, and that goes for <i>any</i> technology.

What are the developers doing, though, if they’re not diagnosing why Reaper isn’t communicating with the Zanzibar service registry?

amazon product ditches amazon product for another amazon product?<p>feels very strongly they just moved from one AWS platform to another.<p>delay between asynchronous communicating processes differs in these architectures and I suspect they were unable to orchestrate microservices to match the RPC &quot;inside&quot; a monolith model. Nobody can: It only matters if your IPC is causing delay you can avoid.<p>Most of us aren&#x27;t in a room where the real cost is high: 90% of computers are more than 90% idle 90% of the time. Amazon is not in that cohort.

I think what most people are missing here is that they used AWS Step Functions in the wrong place. Part of the blame here is that in over enthusiasm of trying to get more users, AWS doesn&#x27;t properly educate customers when to use which service. Worse, for each use case AWS has about dozens of options making the choice incredibly hard.<p>In this case, they probably should have used Step Functions Express, which charges based on duration as opposed to number of transitions and they&#x27;re looking for &quot;on host orchestration&quot; like orchestrate a bunch of things which usually are done in small time and are done over &amp; over many times. Step functions is better when workflows are running longer, and exactly once semantics are needed. Link for reading differences between Express &amp; standard step functions: <a href="https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;en_us&#x2F;step-functions&#x2F;latest&#x2F;dg&#x2F;concepts-standard-vs-express.html" rel="nofollow">https:&#x2F;&#x2F;docs.aws.amazon.com&#x2F;en_us&#x2F;step-functions&#x2F;latest&#x2F;dg&#x2F;c...</a>.<p>This also exemplifies the fact that I learned while being at Amazon &amp; AWS that Amazon themselves dont know how best to use AWS. This being one of the great examples. I&#x27;ll share 1 more:<p>- In my team within AWS, we were building a new service, and someone proposed to build a whole new micro service to monitor the progress of requests to ensure we dont drop requests. As soon I mentioned about visibility timeout in SQS queues, the whole need for the service went away. Saving Amazon money ($$) &amp; time (also $$). But if I or someone else didn&#x27;t mention, we would have built it.<p>I dont think serverless is a silver bullet, but I don&#x27;t think this is a great example of when not to use serverless. It helps to know the differences between various services and when to use what.<p>PS: Ex Amazon &amp; AWS here. I have nothing to gain or lose by AWS usage going up or down. I&#x27;m currently using a serverless architecture for my new startup which may bias my opinions here.

I work in streaming video, specialise on AWS, and have enjoyed using Step Functions for certain (non-video) projects. I am _astonished_ that Step Functions + S3 was even considered as a starting point for defect detection in streaming video. Astonished.

&gt; Moving the solution to Amazon EC2 and Amazon ECS also allowed us to use the Amazon EC2 compute saving plans that will help drive costs down even further.<p>So various parts of Amazon have to work through the AWS same pricing programs that the rest of us do?

Everything old is new again.

I quite like the idea of viewing run cost as architecture fitness function <a href="https:&#x2F;&#x2F;www.thoughtworks.com&#x2F;radar&#x2F;techniques&#x2F;run-cost-as-architecture-fitness-function" rel="nofollow">https:&#x2F;&#x2F;www.thoughtworks.com&#x2F;radar&#x2F;techniques&#x2F;run-cost-as-ar...</a>.<p>If your architecture has a high cost to develop, test and run when a cheaper architecture meets your needs, it&#x27;s a sign that you have overengineered. In my experience there is an order-of-magnitude increase in complexity by adopting microservices that only starts to pay off when your org and user base are huge.

I love how some of developers jumped on the serverless bandwagon with some of the <i>least</i> &quot;serverless&quot; workloads first<p>&quot;Let&#x27;s make our entire website serverless now&quot; erm, no?<p>It&#x27;s cargo culting of the worse kind

Somewhat interesting article, but this isn&#x27;t a monolith, at least not by a microservice fanboy definition.<p>The product (Prime Video) is still built using many business oriented services. Furthermore, this service appears to be developed and operated by a single team.<p>That being said, there are some lessons here - there are good ideas in most design paradigms, but if you take them to the extreme, you&#x27;re going to see some weird side effects. Understand the benefits and engineer a balanced solution.

I think serverless has its place, but this problem doesn&#x27;t seem like a fantastic fit.<p>We are looking into serverless as a way to exhibit to our customers that we are strictly following certain pre-packaged compliance models. Cost &amp; performance are a distant 2nd concern to security &amp; compliance for us. And to be clear - we aren&#x27;t necessarily talking about <i>actual</i> security - this is more about making a B2B client <i>feel</i> more secure by way of our standardized operating model.<p>The thinking goes something like - If we don&#x27;t have direct access to any servers, hard drives or databases, there aren&#x27;t any major audit points to discuss. Storage of PII is the hottest topic in our industry and we can sidestep entire aspects of The Auditor&#x27;s main quest line by avoiding certain technology choices. If we decided to go with an on-prem setup and rack our own servers, we&#x27;d have to endure uncomfortable levels of compliance.<p>Put differently, if you want to achieve something like PCI-DSS or ITAR compliance without having to covert your [home] office into a SCIF, serverless can be a fantastic thing to consider.<p>If performance &amp; cost are the primary considerations and you don&#x27;t have auditors breathing down your neck, maybe stick with simpler tech.

I am big fan of django&#x27;s apps model ... what I like to call a &quot;Modular Monolith&quot;.<p>Being an early engineer at most of my stints, I have build and scaled multiple startups using the approach and it has never failed me, the pitfalls of micro-services is not worth it unless absolutely necessary.<p>I always made it a point to group by business-logic rather than separate at whatever curve ball &quot;new-tech&quot; throws at me.

Two naive ideas that may be OK as a going-in position:<p>- granularity<p>- bandwidth negligibility<p>Breaking everything down to a gnat&#x27;s ass might improve testability, but is testability the product? Do I really need a Java stack trace that reads like an Andrew Wiles proof?[1] <i>Maybe</i> I do, at scale.<p>Then there is the non-zero cost of the packet shuffling. Every edge in the aechitctural graph, not just the nodes, costs. But we just throw a waiter into the code and move on to the next line. No biggie.<p>What was most interesting was &quot;It also increased our scaling capabilities.&quot; Granularity was supposed to let &quot;serverless&quot; absorb the entire universe, I thought.<p>At a higher level of abstraction, maybe The Famous Article is a map&#x2F;reduce job: the requirements dissolved into solution, and a proper number of components precipitated out.<p>[1] <a href="https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Wiles%27s_proof_of_Fermat%27s_Last_Theorem" rel="nofollow">https:&#x2F;&#x2F;en.m.wikipedia.org&#x2F;wiki&#x2F;Wiles%27s_proof_of_Fermat%27...</a>

&gt; The second cost problem we discovered was about the way we were passing video frames (images) around different components. To reduce computationally expensive video conversion jobs, we built a microservice that splits videos into frames and temporarily uploads images to an Amazon Simple Storage Service (Amazon S3) bucket. Defect detectors (where each of them also runs as a separate microservice) then download images and processed it concurrently using AWS Lambda. However, the high number of Tier-1 calls to the S3 bucket was expensive.<p>Taking &quot;malloc for the Internet&quot; [1] a bit &#x2F;too&#x2F; literally there.<p>[1] <a href="https:&#x2F;&#x2F;aws.amazon.com&#x2F;blogs&#x2F;aws&#x2F;eight-years-and-counting-of-cloud-computing&#x2F;" rel="nofollow">https:&#x2F;&#x2F;aws.amazon.com&#x2F;blogs&#x2F;aws&#x2F;eight-years-and-counting-of...</a>

<a href="https:&#x2F;&#x2F;grugbrain.dev&#x2F;#grug-on-microservices" rel="nofollow">https:&#x2F;&#x2F;grugbrain.dev&#x2F;#grug-on-microservices</a><p><i>&gt; grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too<p>&gt; seem very confusing to grug</i>

Seems somewhat curious that they didn&#x27;t at least include Fargate. Feels like they jumped all the way from the typical overengineered setup into using AWS in a way that&#x27;s very close to just &quot;I need virtual machines&quot;.

I&#x27;ve never seen successful micro services if the starting point is not a monolith. The most successful ones I&#x27;ve seen are hybrid ones where some parts needed to be scaled are refactored as a micro service to run in parallel.

Lambda , Steps functions,... is just pure marketing scam to me , because the price is ridiculous too high for 99% of real world use case.<p>They&#x27;re though good enough to deliver an MVP quickly, but that&#x27;s all about it.

Shipping around individual video frames between components is really an astonishingly bad idea.<p>Microservices seem to be a decent idea with a terrible name. The idea of running services that are small enough that they can be managed by a single team makes sense - it enables each team to deploy their own stuff.<p>But if you break things down further, where you need multiple &quot;services&quot; to perform a single task, and you have a single team managing multiple services - all you do is increase operational &amp; computational overhead.

The only time it makes sense to use edge&#x2F;serverless anything is lightweight APIs and rendering HTML to end users so they get the page loaded as quickly as possible. That&#x27;s the only use case good for edge. And any supporting infra that can help deliver rendered pages asap (like kv store on the edge for storing sessions, lightweight database on the edge for user profile data, queues etc). Anything that requires decent amount of processing should not live on the edge&#x2F;serverless. It defeats the purpose.

Am I right in understanding this is just their defect-detection system?

I&#x27;ll launch a consulting business focused on migrations from microservices to monoliths and from the cloud to in-house. Pricing would be a % of the saving over the first year.

I&#x27;m happy to see -- in the discussion here-- the continued backlash against microservices and the deleterious effects it has had on software complexity, and data modelling.<p>But I think it&#x27;s interesting that if we took a time machine back to 2014 or 2015 the tone here would be quite different, and microservices were all the rage on this forum as I recall.<p>I like to hope that the industry learns from its failed trends, but I&#x27;m now old enough to see this is rarely the case.

These days when project managers of new products seek my advice as a solutions architect I tend to suggest they create a minimally viable product that is written modularly so it can scale, but deploy it very simply on a few servers just like we used to 15 years ago.<p>Scaling is definitely a good thing, microservices make scaling easier, no doubt about that. But an MVP rarely needs k8s level scaling, it just needs to be written well so it can scale in the future.

I&#x27;ve been having lots of thoughts lately about how you build a) a system that can respond to scale b) for the affordable price possible c) scaling infrastructure spend with income<p>I love the anecdotes about just buying a Hetzner server which can handle a surprising amount.<p>One of my ideas is a company that maintains an incremental infrastructure that can grow to handle extreme levels of traffic - the infrastructure itself mutates over time.

Breaking things into tiny functions and putting them on many different servers incurs tradeoff costs in both complexity and compute. There is a complexity cost in having to deal with the setup, security, and orchestration of those functions, and a compute cost because if the overall system is running constantly it will be less efficient and therefore more expensive than running on one box.

Rarely will I defend Amazon in anything, but I&#x27;ll make an exception.<p>In my experience, AWS&#x2F;Amazon people do not force you or even direct you to a particular architectural choice. They are relatively indifferent about it.<p>Instead, trend-driven architectures seem to come from the tech community themselves. It&#x27;s the customers often making the wrong choice.

Two things:<p>- When people use the solution -&gt; problem path instead of problem -&gt; proposals -&gt; cost analysis -&gt; solution they get what they deserve.<p>- It is possible to optimize most infrastructures and code, it depends how much obviously but I have seen such percentages before<p>The real question is: why didn&#x27;t they chose the right stack for their problem the begin with?

AWS Step Functions are bad for so many reasons. Scaling, pricing, developer experience, etc.<p>It is clearly made by people who don&#x27;t really understand (or does not care) how distributed workflows work.<p>And pricing are prohibiting to run it at scale. In my opinion it should be free to use, provided you glue together other AWS services with it.

Perhaps Amazon reached peak saturation for its video streaming services so it no longer needed unknown unknowns from holding it back from using a more efficient monolithic architecture. Distributing services across multiple machines is certainly more scalable but all those API calls can add up.

Over engineering at its best. I tend to see microservices as a doubled edged sword and in this case, there was no need for them.<p>Also, the pricing of AWS quickly goes up as you go from EC2 -&gt; Fargate -&gt; Lambda. I don&#x27;t know why on earth someone would build microservices at the lambda-level.

They basically underestimated the cost of moving millions of small files to and from S3; it kinda makes sense if they want to save those images for a long time, but in this case it was for semi-real-time error detection, which is much faster to do in-memory.

Micro-services is BS invented by cloud providers to solve problems you don’t have at 10x the cost.<p>The worst software systems I have ever seen were micro-services. One of them is more than 20 years old. The WTF count per minute is exponential.

It’s expensive to store individual video frames in s3 for no good reason? Go figure…

At this point isn&#x27;t the lesson to use serverless stack for fast iterative processes <i>then</i> use a custom solution once you know exactly what you want?<p>I have 0 experience with serverless&#x2F;cloud. Just a thought.

I wonder if this is, in some way, a kind of signalling of where AWS wants to go – maybe they want to shift more towards dedicated hosting rather than all of these separate services?

You can read more of these sorts of posts at <a href="https:&#x2F;&#x2F;www.microservice-stories.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;www.microservice-stories.com&#x2F;</a>

So one team at Prime for one specific application learned, that serverless was not the ideal compute model for their workload. Wow.

I think using &quot;Monolith&quot; for what they ended up with is badly chosen. Basically they just made a service less granular (or less micro, if you will).

Feels more like the initial version was a prototype not meant to scale<p>Wouldn’t have expected prime to be pushing around images on s3

Locality of reference matters.<p>It&#x27;s fine to split things up, but we have to be careful how we do it + aware of the overheads.

This is astonishing coming from Amazon.

They could save millions by migrating to Digital Ocean or Hetzner (+Cloud66).

I&#x27;m waiting for the AWS Lambda team to talk to Marketing to get this taken down ...

Of all the video streaming services I have used, PrimeVideo is the one where the video&#x2F;audio sync becomes terrible progressively.<p>It is pretty bad. It happens in 8 out of 10 movies. There is some misconfiguration in their AV transcoding pipeline.<p>And here, we have an article talking about Monolith vs. Microservices improving user experience.

This article is going to keep me employed for some time yet

What! They changed the title. Tells you something

but .. but .. there&#x27;s no buzz words in this solution. monolith? ew!

they just coded a step functions monolith...

Guess who happy DHH was reading this

hahahahahahaha

I guess what AWS sells is not servers, but software to manage them automatically, to load balance, to replicate etc. Once, in a short time, GPT can write such (pretty standard) software for you, Amazon will, too, go down.

I wouldn&#x27;t call it a monolith as the number of instances could be scaled up. Mono implies single instance. They just combined multiple microservices into a larger one.