In my work I have the opposite experience, that latency is favoured to throughput: People write messages and expect an immediate answer, for things that are not time-pressing. A lot of things are destroying both though: Starting several things at once, so you can't concentrate on either. We spend a lot of time planning sprints, when half of the work is not planned for but arises spontaneously.<p>I think the problem is that nobody really thinks about what they need and how they can achieve it. If you need latency are you ready to give up throughput? If you need a high throughput, do you manage to shield the team from distractions?
The low latency approach is pretty much already encapsulated by the MVP approach to product design and incremental release schedule. Actually, considering the lower overhead of testing & implementation for small incremental changes vs. monolithic releases that are much harder to reverse course if there's a problem, the low-latency approach could also be the higher throughput approach.<p>Anyway, I am literally going to make myself one single cup of coffee now. No one in my house drinks it, so I have tried just about every method of making a single good cup of coffee. Right now, that balances towards very low latency but very low quality: instant coffee, because my tolerance for coffee latency is usually very low. (also low tolerance for cleaning up the equipment needed to make coffee-- garbage collection!) At some point the needle will swing back towards quality and I'll take the extra 5 minutes to do a pour over & deal with cleaning more stuff.
This analogy maybe works if this is your first time ever making coffee, but anyone who drinks several cups of coffee a day has a process that looks very different from the sketches and doesn’t need feedback from the first cup. Ironically, I think too many problems in software development come from treating things that we do every day and on every project as though they’re unpredictable shots in the dark.
> We bias these decisions towards throughput (I blame Taylorism).<p>I think he's ignoring another tradeoff that may be hiding which leads to a bias towards throughput.<p>One of my favorite lessons with regard to video is something that old school Amiga demo scene taught years ago (slightly before my time) is that standard deviation trumps frames per second in video (I'm starting to suspect this is also true for audio). This has ramifications for video pipelines. Naughty dog has blogged about how they were able to greatly increase their throughput by adding latency (especially on the cell processor). In the process they were also able to make guarantees about variance, and their 30fps was a consistent 30 vs the 60fps that they would occasionally miss. This allowed them to have lower frame rates while still looking better than most gamers would expect from such a low frame rate.<p>So, I would say in the context of making coffee, in real life, boiling enough water for 2 cups is definitely going to have lower variance in the finished time of the second cup.<p>Does it matter for a cup of coffee? Not unless you're at barista scale. But it's an important tradeoff in certain contexts. And an important one to at least be aware of as an engineer.
Cute analogy, but kind of begs the question that you know how much you are going to do/want. In many situations, that 2 is unknown.<p>Even the example of a bus begs the question that you know how many people will take it.<p>Instead, I like the view that you maximize your options. If you know you will definitely want two cups of coffee, get something that makes that more automatic. If you may decide that the one cup is enough, go with what lets you change your mind easily.<p>For the bus, you provide the option at a regular interval so that some folks can weigh that against the cost of a car.<p>Throughput and latency, then, fall off in absolute significance to optionality. Note, they don't disappear. Such is the nature of all trade offs playing against each other.
Kent Beck wrote a post that represents exactly how I feel about software development: "By emphasizing latency [of process] we get feedback sooner. Learning and adapting to external changes lead to less waste and therefore greater efficiency. Each piece is inefficient (compared to some theoretical maximum), but the whole is efficient. In my world, latency dominates. Mostly. But it depends." (4 minutes read.)
You don't always necessarily need to choose one or the other.<p>Frequently there exists perfectly good compromise that has <i>almost</i> the benefit of best latency and <i>almost</i> the best throughput.<p>For example, you have a batch process to process 1 million elements to return a response.<p>You can either process all 1 million elements together and then return response (best throughput at the cost of waiting until everything is processed) or stream results of each one item as they are being processed (best latency but you pay additional cost per item).<p>But there is possibility you can process them in batches of 100? 1000? This means all the additional costs of processing items individually are amortized to something like 1/100th or 1/1000th of the original cost but you still are getting first piece of response very soon (1/1000th or 1/10000th of what you would have to wait if you processed everything in one go).<p>I am currently building large scale trade processing systems using those rules to stream what would traditionally be a batch process, and so far this works beautifully and applies neatly to wide range of problems.
As a worker one deals best with large companies by maximizing thru put. I always keep more than one project going and when one is blocked I do something to save state, and pick the next most urgent one to work on. (Where urgent can also mean “fun, no deps and a clear improvement towards the long term goals,” even if relatively unimportant). However for development process, latency is a total killer. Your growth equations vary smoothly between “development work returns simple interest” and “development work pays
off as expinentially with time” as the interval between a change and it being fully known to be good in production drops to zero.<p>Put it this way. I am going to have that second cup of coffee. Making the whole pot at once pays off consistently day over day. Will this slightly different way of doing the load balancers pay off tho is less certain. It needs experimentation. And if I can turn the dial and watch the effect in real time, I learn quickly. If I wait one week for each data point, I learn slowly.
I know the coffee thing is just an analogy to make his point, so there is not much point in arguing about it. But since I face this question every morning, I can't help but say what I think about, and it is not latency versus throughput. It's latency versus risk.<p>For me it is tea, and not coffee. I have an automatic kettle that will bring the water to a desired temperature and then holds it there. I also have a small ceramic tea pot I brew tea in. I often rew 2 or 3 batches (I know which when I start). I heat the entire amount of water by default, even though that takes longer to initially heat.<p>Sometimes I am in a hurry and want the tea sooner. Then I consider if I should heat a smaller amount of water to get started faster. I don't think I have ever done that though. The reason is risk. (1) Will it heat the
water in time. The answer is yes it should, but that would be very bad if it didn't. Then my tea pot would
cool down as I waited. I think that might be bad for the tea quality. To me this is an assymetric bet. (2)
Will I even remember to put more water in the kettle. I am such a creature of habit. That would really mess up the tea pot temperature. (3) Will I even put the right amount of water in for the initial pot. That takes 2+ portions of water. One to preheat the tea pot, a small amount to rinse the tea, and then the portion to actually brew the tea. That would also be bad if I didn't have enough water.<p>Granted, a number of those items are just because of habit/momentum. If I normally heated enough only for he first batch of tea, then I might be hesitant to heat all the water ahead of time.<p>So for me it comes down to latency versus risk. I suspect this is a factor in "real world" decisions to.<p>For what it is worth, when I write software, I would say I dive in sooner and start writing something like an MVP, along the lines ofwhat the author is saying. Admittedly I plan more features than I plan on writing for the first version, kind of like giving yourself a good leve when playing pool.
Other aspects of latency (for delivering software/features):<p>(1) Psychological. Being able to see early signs of progress can build momentum. Momentum and morale matter. If your efforts don't feel connected with a reward, you can get discouraged. (Obviously, perseverance is a good thing, but realistically it doesn't make this a total non-issue.) Nobody wants to spend 40 years in the wilderness before entering the promised land.<p>(2) Political. People like to see visible results. It's good internal PR. Even if delayed delivery is more efficient, telling others in your organization "just be patient" is a hard sell, can be risky, and may require backbone. And it may require someone to stick their neck out for you. (Like a manager telling their manager, "Yeah, it does look like nothing is happening, but my team has it under control.")<p>(3) Competition. Lower latency can mean gaining an advantage by being first to market. Sometimes the race is close and reductions in latency would be valuable, and sometimes not.
Competitors' plans are usually a big unknown, so there's lots of guessing. Excessive fear is a hazard that can lead to overemphasis on reducing latency.
I'm currently taking a small Hacker News break from my Coursera course on project management (the one made by Google!). This is oddly relevant.<p>When compacting timelines because of a schedule deviation, you typically decide to crash or fast track. A crash is your typical nightmare of achieving more throughput (like the author mentions). A fast track is performing previously sequential tasks (high latency) in parallel (more throughput).<p>I realize this isn't groundbreaking information but I wanted to share what I learned of project management lingo :-)<p><a href="https://www.simplilearn.com/fast-tracking-vs-crashing-article" rel="nofollow">https://www.simplilearn.com/fast-tracking-vs-crashing-articl...</a><p>PS - The course is very good. Coming from a world of consulting most of my life I thought I had a good grasp on terms, ideas, etc - but I've picked up quite a bit here, Google did a great job with the course.
If the two cups of coffee are for one person, I don’t think you want them at the same time, one might get cold.<p>I think you can start to heat again sometime during the drip phase. Start to enjoy your first cup while the 2nd cup is heating. Then the time between 2 cups being ready is shorter than option 1 and not at the same time as in option 2.
Is anyone else bothered by the fact that the coffee example as drawn has been contrived to make the throughput of each approach different.<p>In the “low latency” example, heating and dripping are sequential, when you can obviously be heating water for the second cup, while waiting for the first cup to drip. Increasing the total time by two units.<p>In the “high throughput” example, heating double the water only takes one third more time rather than double the time. Decreasing the total time by one time unit.<p>So if you eliminate these contrivances both approaches take the same amount of time. Meaning they have the same throughput, but one has lower latency to first cup. So obviously the better approach.<p>I appreciate the article isn’t really about the best way to make coffee, but using a broken example does detract from his point a little.
In the "optimal latency" case you can start heating the 2nd cup immediately after the first, then it's ready after 7 time units, for average of 3 time units per cup, which is better throughput than the "optimal throughput" case (3.25 t/c).
Interesting that he didn't consider the speed of his consumers. Presumably a single consumer won't drink 2 cups of coffee at the same time, and the 2nd cup will grow cold. I'm sure there's something to his software analogy there too.
I have:<p>An espresso machine<p>A bonavita temperature controlled kettle<p>A baratza vario<p>V60 and chemex<p>I have the espresso machine on a timer for 30 minutes before I wake up. The bonavita has temperature hold.<p>Depending on the day and my wife, I’ll do pour over or espresso. In both cases we are only talking about ~2 minutes of actual time because the setup is optimized and I don’t need it the second I wake up (If I did, I’d do espresso). If it’s pour over, I push two buttons then do a few other things in the morning.<p>A double boiler espresso machine would be the king for optimizing for both, having the most stable temps for any task.
Whet it comes to networks, this is a thought tradeoff because we can have the same latency even though the throughput (mbit/s) is high. Also on modern consumer CPU's it will take some time to throttle up. Also JIT compilers can do different things depending on load. It seems that every step in the hardware/software stack is optimized for throughput.
I found it very difficult to read this as it started out with the whole 'I made a bad analogy about coffee making that nobody understood, and actually is probably not realistic but anyway now let me tell you what my point is referring back to this bad analogy periodically'
The coffee-analogy was such a perfect example for wasting time on irrelevant optimization that I didn’t read on.<p>Even if the authors’ perspective in the rest the article may have been more useful, spending my time with other things felt like an optimization.
Where I work often latency is prioritized at the expense of throughput. People tend to amplify minor issues which left untouched resolve themselves. Just by leaving the respective task untouched, it leads to increased throughput.
Author is not aware most home drip coffee makers have a spring valve under the filter basket. You can put in two cups, pour one cup when it's ready, replace the carafe and continue making the second cup.
Disclaimer: haven't read the article<p>Disclaimer: probably not relevant<p>If you're still drinking drip coffee from a machine, PLEASE acquire and use a french press. It's so much better, easier, cleaner, and more personal. You choose how clean the vessel is, how strong the coffee is, how much coffee to make, etc.