Linux ate my RAM (2009)

<a href="https:&#x2F;&#x2F;archive.ph&#x2F;WmJHf" rel="nofollow noreferrer">https:&#x2F;&#x2F;archive.ph&#x2F;WmJHf</a>

There&#x27;s some really interesting little details here.<p>Linux, by default, is making the very reasonable assumption that the marginal cost of converting empty physical memory into caches and buffers is very near zero. This is fundamentally reasonable, because the cost of converting empty memory into used memory isn&#x27;t really any cheaper than converting a clean cached page into used memory. It&#x27;s a little more subtle when you take accounting into account, or when you think about dirty pages (which need to be written back to clear memory), or think about caches, but the core assumption is a very reasonable one.<p>Except for on some multi-tenant infrastructure. Here, &quot;empty&quot; pages don&#x27;t really exist. There&#x27;s mostly not an empty page of memory kicking around waiting (like there is on client devices). Instead, nearly all the memory on the box is allocated, but each individual guest kernel doesn&#x27;t know the full allocation. In this world, the assumption that the marginal cost of converting empty to full is zero is no longer true. There&#x27;s some real cost.<p>Projects like DAMON <a href="https:&#x2F;&#x2F;sjp38.github.io&#x2F;post&#x2F;damon&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;sjp38.github.io&#x2F;post&#x2F;damon&#x2F;</a> exist to handle this case, and similar cases where keeping empty memory rather than low-value cache is worse for the overall system. These kinds of systems aren&#x27;t super common, especially on the client side, but aren&#x27;t unusual in large-scale cloud services.

I once worked at a government job and took my computer into the IT department for an issue. I can&#x27;t remember anything about the original issue.<p>But I do vividly remember the help desk trying to figure out one last issue. Some process was consuming all my resources.<p>They never could figure out why &quot;System Idle Process&quot; kept doing that.

When I worked in telco we used to run into this a lot.<p>We&#x27;d demand standard alarms for things like memory leaks &#x2F; out of memory conditions &#x2F; high than normal memory usage, as to get 99.999% uptime we want to be paged when problems like this would occur. Except a bunch of platform did the extremely naive implementation and included recoverable memory in their alarm conditions. So inevitably someone would log in and grep the logs or copy some files to the system, and hit the alarm conditions.<p>And there were some vendors who really didn&#x27;t want to fix it, they would argue that recoverable memory is in use, so it should really be part of that alarm condition.

I used to run a Linux workstation in the late 00&#x27;s (sorry FreeBSD folks, I know, the shame...), and I ran a closed source PVR application on it.<p>The memory access pattern was pretty much pessimal for my use of the box as a workstation. I&#x27;d use it from 7am -&gt; 8&#x2F;9pm every day, then when I&#x27;d walk away from the keyboard, I&#x27;d watch HD recordings (which could be 7GB or more per hour). Those would get cached in memory, and eventually my workstation stuff (emacs, xterms, firefox, thunderbird) would start to get paged out. In the mornings, it was painful to start using each application, as it waited forever to page in from a spinning disk.<p>I eventually wrote an LD_PRELOAD for the DVR software that overloaded open, and added O_DIRECT (to tell the kernel not to cache the data). This totally solved my problem, and didn&#x27;t impact my DVR usage at all.

Trust me. If Linux really eats your RAM to the point of reaching an OOM state <i>you will know</i>.<p>(This was of course because of having too many apps relative to my RAM. Not because of disk caching.)<p>The OOM behavior is not pleasant for a desktop system.

My advice in the situation when someone wants to &quot;free RAM&quot;: &quot;You bought it, better use it.&quot;<p>It always felt strange that people buy lots of RAM but want it to be kept unused...

RAM isn&#x27;t user friendly in Linux. Ubuntu Desktop is the most popular distro by far by Google Trends, but it doesn&#x27;t even come with RAM compression set up out of the box, so as soon as you run out of memory the UI totally locks up until the task killer kills a process, which always takes minutes in my experience. Pop OS does come with RAM compression set up, which is Ubuntu based, but then you&#x27;re stuck on xorg instead of Wayland right now, because they decided to make their own DE from scratch in Rust for some strange reason, which isn&#x27;t available yet. You can set up RAM compression yourself, but when macOS and Windows both have it standard, coming to Linux as a newbie so you install Ubuntu Desktop and your whole system locks up as soon as you run out of physical RAM, it&#x27;s really odd and unexpected. I&#x27;m not even sure who would want to run a desktop distro without RAM compression.

&gt; Disk cache can always be given back to applications immediately<p>This is not true, there is a cost to freeing the cache pages and allocating them to the other program. I have seen some very regressive performance patterns around pages getting thrashed back and forth between programs and the page cache, especially in containers which are memory limited. You throw memory maps into the mix and things can get really bad really fast.

Is this the reason Windows Task Manager seems to show Vmmem (WSL2) as gobbling up well more RAM then WSL seems to indicate is in use?<p>I have more then enough RAM on my office workstation to just accept this, but on my personal gaming computer that moonlights as a dev machine, I run into issues and have to kill WSL from time to time.

This feels like a UX problem. If this is a normal and expected part of linux operation, it should be called out in the {T,G}UI.

Windows and Mac use compressed RAM for many many years as standard.<p>Yet on many Linux desktop you have to activate it (namely ZRAM). It solves the problem that a e.g. browser eats all your memory. It&#x27;s much quicker than Swap and yet mostly unknown by many people who are running a Linux desktop. As mentioned by another user it&#x27;s still not standard on Ubuntu desktop and I don&#x27;t understand why.

I think the information there is valuable because questions about memory usage in Linux keep coming up. The answer: &quot;don&#x27;t worry about it,&quot; is probably a good starting point. The page claims things that are just really misleading, though.<p>&gt; There are no downsides, except for confusing newbies.<p>False. Populating the page cache involves lots of memory copies. It pays off if what&#x27;s written is read back many times; otherwise, it&#x27;s a net loss. It also costs cycles and memory to keep track of all these pages and maintain usage statistics so we know what page should be kept and which can be discarded. Unfortunately, Linux makes quantifying that cost hard, so it is not well understood.<p>&gt; You can&#x27;t disable disk caching. The only reason anyone ever wants to disable disk caching is because they think it takes memory away from their applications, which it doesn&#x27;t!<p>People do want that, and they do turn it off. It&#x27;s probably the number one thing database people do because they want domain-specific caching in userland and use O_DIRECT to bypass the kernel caches altogether. If you don&#x27;t, you end up caching things twice, which is efficient&#x2F;redundant.

Does anyone happen to have expertise&#x2F;pointers on how ZFS&#x27; ARC interacts with Linux disk caching currently when using ZFS-on-Linux? It seems like the ARC space shows up as &quot;used&quot; despite being in a similar category of &quot;made available if needed&quot; - is that correct?<p>Is data in the ARC double-cached by Linux&#x27;s disk caching mentioned in the post? If so, is it possible to disable this double-caching somehow?

We published this paper &quot;TMO: Transparent Memory Offloading in Datacenters&quot; last year which covers some Linux memory management mechanisms that may be quite useful for providing reasonable estimations to application memory usage.<p>We observed that the real memory footprint for applications depends on many factors: file access pattern, disk IO speed (especially if swap is enabled), ssd vs hdd, application latency sensitivity, etc. Instead of coming up with some overly complicated heuristic, we use the Linux kernel provided memory.pressure [0] metric via cgroup v2. It measures the amount of time spent waiting for memory (page fault etc). Then by slowly reclaiming memory from the application until its memory pressure hits some target (say 0.1%), we can claim that the steady state usage is the actual memory footprint.<p>This may not be useful for PC but could be very useful for data center to track memory regression, and also to harvest disk swap without concerning too much about the cliff effect when the host runs out of memory and suddenly kernel pushes everything to swap space.<p>[0] <a href="https:&#x2F;&#x2F;facebookmicrosites.github.io&#x2F;cgroup2&#x2F;docs&#x2F;pressure-metrics.html" rel="nofollow noreferrer">https:&#x2F;&#x2F;facebookmicrosites.github.io&#x2F;cgroup2&#x2F;docs&#x2F;pressure-m...</a>

&gt; If your applications want more memory, they just take back a chunk that the disk cache borrowed. Disk cache can always be given back to applications immediately! You are not low on ram!<p>I&#x27;m running RKE2 on my desktop and it&#x27;ll start killing pods due to low memory pressure, even though the memory was only used for disk caching. I wonder if there is any way to make it stop doing that and instead only start killing pods if it&#x27;s due to &quot;real&quot; low memory pressure.

I remember that being the case for early versions of Android, people were surprised all their RAM was used, and of course, we could find apps that &quot;freed&quot; the RAM, generally making things worse.<p>And the response was similar: all that &quot;used&quot; RAM can be reclaimed at any time should an app need some, but in the meantime, the system (which is Linux) might as well use it.<p>I think they &quot;fixed&quot; it in later versions. I don&#x27;t know how, but I suspect they just changed the UI to stop people from complaining and downloading counterproductive apps.<p>As usual in these situations, unless you really know what you are doing, let the system do its job, some of the best engineers with good knowledge of the internals have worked on it, you won&#x27;t do better by looking at a single number and downloading random apps. For RAM in particular, because of the way virtual memory works, it is hard to get an idea of what is happening. There are caches, shared memory, mapped files, in-app allocators, etc...

Unused RAM is wasted RAM. Why people want to see GOBS of empty RAM boggles my mind.

This made me chase red herrings when debugging oom issues in production. Wish free would just remove the &#x27;free&#x27; column and replaced it with &#x27;available&#x27;.

&gt;&gt; If applications want more memory, they just take it back from the disk cache. Q: If there is no swap configured, will a malloc() then take away clean page-cache pages ? Or does that happen only on page-in ?

Okay, if unused memory is wasted and there are truly no consequences for the &quot;free&quot; column reading zero, then why on a busy system do I get UI chugging and otherwise poor (bordering on unusable) performance under this condition that is immediately resolved by forcing the caches to drop and freeing multiple gigabytes?<p>Whatever conceivable speedup there is from 12 GB of file cache as opposed to 11 is obliterated multiple times over from the time lost by having to do this dance, or worse, recovering after the oom killer wipes out my X session or browser.

Another question is about containers and memory limits. Does the page-cache count against my container memory limit ? And if so, then when I hit that limit from doing many reads, does the page-cache start taking from itself without OOM killer getting involved ?

How old is this website? It&#x27;s from a time when a typical computer only had 1.5 G of ram.

I pity the fool who don&#x27;t eat RAM.

It&#x27;s weird that their &#x27;used&#x27; column is accounting for &#x27;buff&#x2F;cache&#x27;<p>&#x27;man free&#x27; states<p><pre><code> used Used or unavailable memory (calculated as total - available)</code></pre>

Why all the fuss??<p>When you can download as much RAM as you want, any time you want.<p><a href="https:&#x2F;&#x2F;downloadmoreram.com&#x2F;" rel="nofollow noreferrer">https:&#x2F;&#x2F;downloadmoreram.com&#x2F;</a>

My Linux ram problems are 1: Ubuntu default install, ends up with xwayland using 5GB ram. 2: when running out of ram it seems to default to crashing back to the logon screen

You can always just download more RAM. <a href="https:&#x2F;&#x2F;www.downloadmoreram.com" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.downloadmoreram.com</a>

Windows ate my hard disk (every year).

Is the website down? I was reading and suddenly started to get timeout.

htop shows this (it&#x27;s the orange&#x2F;yellow bar in RAM)

&gt; Disk caching makes the system much faster and more responsive! There are no downsides, except for confusing newbies. It does not take memory away from applications in any way, ever!<p>No downsides except for massive data loss when the system suddenly loses power&#x2F;a drive crashes and the massive theft of memory from host OSes (e.g., when using Windows Subsystem for Linux).