Linux: Minimizing Memory Usage

Disclaimer: This is just my opinion, vaguely based on my experiences, and there are probably counter-examples that proove me wrong in certain circumstances.<p>Memory footprint is very much of a concern not because we wouldn't have memory -- we do -- but because we still have disks that maybe fast by the 90's standards. And all that bloat gets partially loaded to memory at some point or another. And offloaded to swap or discarded, and then again loaded when needed. Disks kill performance.<p>And if you do anything complicated with your computer you'll end up to the disks eventually, if not for anything else that Linux kernel favors I/O buffers by default so a big file copy can turn much of your running image dirty.<p>That's why it can take seconds to start up a Firefox or a half a minute to start Ubuntu. You can speed that up with hacks but it doesn't cure the underlying reason.<p>I think the magnitude of the problem is pretty much revealed in the blog post where RES-SHR for gnome-terminal is a whopping 9 megabytes. How the FSCK can a simple terminal possibly waste 9 megabytes of actual memory? Why the hell does my Nautilus process that is effectively doing nothing but display a few fancy icons on my desktop consume 12 megabytes of memory?<p>I'm not kidding. There's no reason we couldn't have a system that boots to full desktop in less than a second. There's no reason we couldn't have these programs take tens of kilobytes in size, eating at most hundreds of kilobytes of memory -- with the same user experience. We just need to scoop up all program data in one big disk read and then unleash our horde of 3GHz CPU cores at them.<p>I sometimes wonder how much faster a Linux desktop would be if it was compiled with -Os rather than -O2/-O3. The CPU speed isn't the bottleneck, a typical Ubuntu desktop does nothing CPU intensive. I suspect that even a high-level bytecode environment would probably increase the responsiveness of the system in spite of being interpreted because the program code would take less space and fit better in caches.<p>It's NOT all right to dismiss this as a non-problem just because we have loads of RAM.

First, he forgets about the sharing memory which needs to be subtracted from the resident number, since that memory is shared by multiple applications, so gvim eats not 27mb but only 14mb which is a very respectable number in my opinion.<p>Second, I just have gone through a long-awaited "Arch weekend" and finished building a fully functional ArchLinux machine.<p>It was definitely not "optimizing for the wrong thing". Arch allows you to pick system components you actually want, as opposed to Ubuntu's approach to installing everything you may need. It speeds up your boot/reboot/resume time and it does run about 30% leaner on RAM when it reaches feature parity with my Ubuntu installation.

64-bit made things worse, especially if using interpreted languages which often have convenient but complexly indirect structures.<p>E.g. creating this:<p><pre><code> d = dict((x,x * 42) for x in xrange(100000)) </code></pre> a dictionary with 100k items in Python, takes up 10996 kB RSS memory on my 64-bit system, but 5200 kB RSS on a 32-bit Python (same Python version). You've got your dictionary hash buckets, the integer objects etc, all separately allocated objects. The integers themselves are also 64-bit.<p>Of course, if I really cared about the memory in the above case I'd create an array or even a list that fit the key/value pattern; the array array.array('i', (x*42 for x in xrange(100000))) stores the same information in the maximally compact form increasing RSS by about 500k.<p>But given a production system with 32G of memory memory optimisation is rarely being made.

BTW, Check out smem ( <a href="http://www.selenic.com/smem/" rel="nofollow">http://www.selenic.com/smem/</a> ) for a more accurate reporting of memory usage - saves you from having to make the RSS correction, etc. It's from kernel hacker and the main author of Mercurial, Matt MacKall.