Something is deeply broken in OS X memory management

Not this again, we already went through this a few weeks ago.<p>Back then, I thought the conclusion was that there is nothing broken about OS X memory management, and that with every 'fix' you come up with, you will just introduce another degenerate corner case. The same holds for any OS, trade-offs are made that may have some negative effect in some cases, to the benefit of the general cases.<p>I don't recognize any of his symptoms anyway, and my OS X computers get pretty RAM-heavy use, with almost always a linux VM open, XCode, Safari with ~10 tabs, iTunes with a few thousand songs, etc.<p>Edit: Just to be sure I read through some of the links he provides that are supposed to explain what is going on and why the fix would be of any help, but nowhere do I see any hard facts that demonstrate what is going on. Only that he 'saw in vm_stat that OS X was swapping out used memory for unused memory'. I'd like to see some actual evidence supporting this statement.

No actual data, barely any technical discussion at all, mention of "the garbage collection algorithm" which most likely isn't even being used by most of the apps running, capped by a total cargo-cult solution... and this is #1 on the front page?

The core reason that this happens is that OS X uses a memory management mechanism called Unified Buffer Cache (<a href="http://kerneltrap.org/node/315" rel="nofollow">http://kerneltrap.org/node/315</a> is the only reference I can find on this).<p>This seems like a good idea to unify paging and disk cache memory, but it actually isn't. This means, that if you do a lot of I/O, resident pages (i.e. your programs) can actually get pushed <i>out</i> of memory to free up RAM for the disk cache. This degenerates pretty badly in scenarios like using VMs, since you're also using large sections of mmap'd memory.<p>This doesn't happen on NT or Linux, because disk cache can only be turned <i>into</i> memory (i.e. making disk cache smaller), not the other way around; the policy is "Disk cache gets whatever's left over, Memory Manager has priority"<p>Unfortunately, the only thing you can really do about it, is have a machine with a huge amount of RAM, which will kind of help.

What really needs to happen is that Spotlight and Time Machine need to use direct i/o (F_NOCACHE) when they read data from the filesystem, this way they won't pollute the disk cache with their reads and OSX won't swap out a bunch of pages in response.<p>I think you could probably hack something together that does this with DYLD_INSERT_LIBRARIES (OSX's LD_PRELOAD) that would would hook the open system call and fcntl F_NOCACHE on the file descriptor before it hands it back to the application.

I did this for awhile and ran into an interesting (if fatal) edge case:<p>I use a 1.5tb external drive formatted in exFAT to minimize cross-platform headaches, and whenever the drive is marked dirty (improper shutdown, eject, etc), OSX will run fsck_exfat on it before I can use it.<p>fsck_exfat isn't a huge deal -- or wouldn't be, if it didn't have a nasty tendency to leak RAM... the moment you plug in, fsck_exfat's footprint climbs up and up and up... never stopping! Pretty soon it's eaten up 8gb out of my 8gb RAM and poor ol' lappy is unusable.<p>I can say with authority what happens when you run out of physical RAM in OSX: <i>it hard locks</i>. Nothing works -- no keyboard, no mouse, nothing.<p>So, if you plug in your large, dirty (you dirty drive you!) exFAT-formatted external drive, with dynamic_paging switched off, and let fsck_exfat do its thing, your laptop freezes! Leaving the drive dirty, only to be re-scan on boot-up... freezing the laptop, leaving the drive dirty, only to be re-scan on bootup...<p>EDIT: this is with Snow Leopard...

I've experienced exactly the same thing that was described in this article. All the way though to installing more ram and disabling the dynamic pager (this is a late 2011 mbp).<p>Like the author I was shocked at how accustomed I was to waiting for an app to become responsive again. I was trained to wait on the OS to do it's business before I could do my work. Now things happen as quickly as I can think to do them, this is how computing should be.

I've been a Mac user since the beginning, and by far my biggest frustration is the perpetual running-out-of-RAM, even when I close basically everything. I have 4GB of RAM, and frequently catch kernel_task using at least half of it.

I also have major memory problems. 8GB RAM total in the system and I have 4GB sitting in inactive and it's paging out? <a href="http://imgur.com/VE4GB" rel="nofollow">http://imgur.com/VE4GB</a> At this point the system pretty much thrashes until I start closing apps or perform a manual `purge`

I had a similar problems with my Mac slowing down to a crawl with certain instances of disk access.<p>I tried turning off spotlight (which was taking a very long time to complete) but it did not help.<p>For me, the problem turned out to be a failing hard drive. After replacing my system hard drive, things returned to normal speed.<p>I'm just posting this in case it might help someone else.

There's a nice script[1] for tweaking OS X's dynamic pager settings to reduce the system's swappiness that helps a bit. Incidentally, if you have both an SSD and HDD installed, you can use it to move the swapfiles to the HDD to reduce wear.<p>[1]: <a href="http://dropsafe.crypticide.com/article/3848" rel="nofollow">http://dropsafe.crypticide.com/article/3848</a>

Thank god for this article. My wife is a photographer making heavy use of Lightroom on her 17" MBP and has been experiencing these exact problems for a year or two. We've tried everything to fix it, rebuilding the system from scratch, to no avail.<p>She had 4 gigs of RAM which we recently upped to 8gigs which reduced the severity of the problem.<p>I really, really hope this is something that gets fixed in Mountain Lion. Tasks that should take 20 seconds take 10 minutes or more.<p>It's good to know she's not crazy.

It occurs to me that HFS might be the real culprit. A lot of the bad behaviors described here involve heavy disk use. John Siracusa has a nice round-up of all the HFS faults:<p><a href="http://arstechnica.com/apple/reviews/2011/07/mac-os-x-10-7.ars/12" rel="nofollow">http://arstechnica.com/apple/reviews/2011/07/mac-os-x-10-7.a...</a>

You can run `purge` in a terminal to free your 'inactive' ram.<p>I've set up a cron job to purge frequently, keeps thing humming.

Does anyone else notice extreme time-based slowdowns using multiple monitors? I've looked through forums and system logs and I can't find an immediate explanation for it. The system tends to hang when using multiple monitors for any extended period of time.

<a href="http://wagerlabs.com/blog/2008/03/04/hacking-the-mac-osx-unified-buffer-cache/" rel="nofollow">http://wagerlabs.com/blog/2008/03/04/hacking-the-mac-osx-uni...</a><p>It's called the Unified Buffer Cache (UBC).

the OS X kernel is open source. so why aren't people reading it to figure out where this bug is?

I've gotten a lot of his symptoms, and there might be another cause:<p><a href="http://reviews.cnet.com/8301-13727_7-20064489-263.html" rel="nofollow">http://reviews.cnet.com/8301-13727_7-20064489-263.html</a><p>Bad blocks in the disk, causing the system to beachball frequently due to disk I/O failures when swapping out to disk.<p>The solution for me was to back up, reformat the disk and zero-ing out everything causing bad sectors to get remapped, and then restore.

Something is quite wrong indeed. I disabled the dynamic pager, and now my system is working as it's supposed to. Snappy and responsive.<p>I opened all of my apps, expecting it to crash miserably: instead, the system started paging as it should, stayed responsive (though slower), and promptly returned to normal once it regained memory.<p>I don't know what's going on, but I can definitely say that this is how I want my computer to work.

I think this is the penalty we pay for the advice from two years ago:<p>"Buy all your developers SSDs. It makes them more productive."

Try "Free Memory"... I looked for a solution to this problem a few months ago. My computer runs better if I free the memory every few days. For example, resuming a parallels virtual machine drops from 30 seconds down to 3-4. Note, this is a 4GB Core 2 Duo with SSD.

My completely non-scientific observations have found that OS X needs plenty of RAM, like any modern OS. However, any disk I/O task has a huge performance impact on the rest of the system, as described by this article. For example, something like unRARing a file will affect the entire system detrimentally, even if CPU usage is nominal. By affect I mean even the cursor can get jittery, which is normally unheard of on OS X.<p>This typically affects me in low memory situations, such as less than 100mb of free memory. The effect is most pronounced when switching between browser tabs, which would cause a lot of disk usage... pulling all of that data in and out of non-ram cache.

I don't have any tests to prove this, but switching from a 64-bit kernel to a 32-bit one and forcing apps to run in 32-bit mode helps a lot with memory usage on OS X. You can use this app to switch apps to 32-bit: <a href="http://www.macupdate.com/app/mac/40405/sixtyfour" rel="nofollow">http://www.macupdate.com/app/mac/40405/sixtyfour</a><p>If you look at Windows 7 memory consumption with the same set of software you use in OS X, you'll notice memory usage is 1/2 or 1/3 on Windows compared to OS X. Maybe someone knows why that is?

Anyone have any nice articles with tips on how to generally optimize OS X? Esp to better handle this paging/memory management problem that the OP is talking about.<p>I have a MBP with 4GB RAM and leave programs open all the time. After a few days, it feels very sluggish.<p>Aside from double my memory and changing my habits (i.e. shutting down every night), how do I fix this?

I'm (fairly) uninformed on this, but from an initial read of the post it seems like memory management may have been optimized for SSD-equipped systems, at the expense of hard disk performance?<p>Whether this is unintentional, part of a calculated tradeoff, or a cynical business/tactical decision is another thing.

The only winners are those "Mac cleaner, keeper" apps whose Google Ads know we are all watching beach balls.

Damm.. I turned swap off, and now I have 3 VMs running concurrently on my 2009 MBP with 8 GB RAM, and it's smooth! Before this, even one VM would cause the system to periodically become unresponsive. Ok, this is my _subjective_ opinion, and you can ignore it, but hey, it works for me.

I fucking hate OS X Lion Vista. I really miss the stability of Snow Leopard. But can't easily go back.

When time machine starts on my MBA ( to an external hd ) it almost freezes the Mac. Is this related?

Disabling dynamic paging as suggested in the fine article seems to have made some apps that constantly gave spinning wheels before (Firefox, Time Machine backups) some extra speed.<p>However, it's still early days. It might just be a "washed car effect."<p>(Mac Pro 1,1 / 7GB RAM / WD Caviar Black)

I too have noticed huge issues particularly when using photoshop or final cut pro. I figured it was the applications, but if it's the OS that's definitely a much bigger issue. I regularly restart every 2-3 hours when using those two programs heavily.

Interesting tangent: Plan 9 never had "dynamic paging" (swapping to disk). It supports virtual memory, but not swap. This information is accurate as of about 6 years ago (when I stopped following Plan 9).

Anyone have a radar number? This entire discussion is incredibly vague.

I've had problems with mtmd (Mobile Time Machine) really slowing down writes (and all disk stuff) since the Lion pre-releases. With that off (sudo tmutil disablelocal) things are pretty smooth.

Awesome. The thread is more informative and entertaining than the article! This is Hacker News!

i don't understand why anyone who cares about performance doesn't at least max out the RAM, let alone use an SSD as their boot disk. sure, the SSD is expensive, but the RAM? dirt cheap. i only wish the MBP could take more.

Please:<p>font-size: 16px; line-height: 1.5em;

I have an 8-way Xeon Mac Pro w/ 20GB of RAM, almost half of which is 'free' at any point during the day unless I'm doing something really out of the ordinary.<p>Yet it still swaps to disk ALL THE TIME and a new Terminal.app window can take up to 5 seconds to open.<p>I really don't give a shit how it's not "technically" broken - that's broken from an experience point of view. And I haven't re-installed the OS (this was an App Store upgrade from Snow leopard) because that's a major pain in the ass as this is an actual workstation used to do actual work.<p>I can't believe this is actually advice, either - that's what Windows users used to say in the 90s. Anyway, I guess I'm just ranting. OS X is wonderful except for the fact that it sucks at managing memory to keep a system snappy.

This claimed problem is not adequately described-- because I don't see it. Therefore there is something else going on here.<p>I have 8 GB of memory on my Macbook Pro, and I've never seen anything like the problems described in this article. I leave my machine up for weeks at a time, I leave the Time Machine drive connected for weeks, and never have a slowdown, even when time machine is backing up.<p>The only time I have memory issues is when I have too many safari tabs open for too long. Eventually safari takes too much memory. So, I shut down safari and start it back up-- it opens up all the tabs that were open and takes a lot less memory.<p>I'm kinda astounded that people with 20 and 32GB of RAM and 8 CPUS are saying it takes 5 seconds to open a terminal window. Have never seen anything like that.<p>I would venture to guess that these people who are seeing the problem are running a kernel extension or possibly have otherwise modified their machine.

&#62; OS X is wonderful<p>ROFL