About once a week I hear some poor newbie scream in terror as he discovers that his box is just seconds away from a gruesome death with barely a few megabytes of memory left. How could this have happened, it was fine when I booted it this morning, why does a Linux box need 2GB of memory just to run Apache, more bloated than Vista, etc, etc.
Then you explain about the wonders of disk caching, and invariably the first question is always “How do I disable it?”
All in all, it’s a lot of repetition.
To avoid this, I registered LinuxAteMyRam.com which features a big flashing “Don’t Panic” sign and answers the most frequent questions as reassuringly as possible.
The goal is to allow people to appreciate Linux’s disk cache for the brilliant, unobtrusive and effective optimization it is, so it skips over some details like the swappiness setting.
If you have thoughts or suggestions, do comment.
I love your http://linuxatemyram.com/ site! I’ve tried to explain memory usage in Linux to so many people, but now I can just give them a link to your site. Excellent work!
Do you know if there is a way to Tune the cache buffer?
While I understand the concept of the cache buffer it does not always work to the users benefit. I have a single computer that functions as my desktop as well as a DVR backend server (MythTV), unfortunately when Myth records a TV program it bumps about 3GB of older (worthwhile) cache data and inserts an hour of TV which may not be watched (ie. used again) for a week. Another side effect of this is that my Desktop programs get swapped which causes them to be extremely slow to respond when I return to them. The end result is caching of files system data that will be read and used at a rate FAR lower then drive is capable of to the detriment of the performance of all other processes.
A simple solution to this would be some way to disable the cache buffer for a specific file system. I have looked high and low and cannot for the life of me find any way to do this but I can find MANY explanations such as this one as to why caching is good.
@Paul
You do indeed have a special case there, and Linux does cover such scenarios.
The first and easiest thing to do is to set swappiness to 0, so that your desktop apps won’t get swapped out.
Linux allow apps to specify that no caching should be done on a per-open basis through O_DIRECT. The problem is getting MythTV to use it.
I’m sure other people are in the same situation, so you could submit a patch or feature request for an option to enable O_DIRECT when encoding output files.
If you don’t want to compile your own MythTV or wait for a fix, you can also use an LD_PRELOAD hack to set O_DIRECT on some or all calls to open. This particular example sets it on all calls, which would be horrible for loading thumbnails and such, but it should be simple to modify it to match your output files by name.
I’d be interested to know what you do and how it turns out!
One of the things I love about linux is that you can clear the cache. There are various scenarios like watching movies or other file based apps where the cache is filling up with junk that you are not going to use again.
Unfortunately this does result in real apps being swapped out, and needing to clear the cache. I just spend 20 minutes waiting to get a ssh login to a box with this issue.
The free command took forever, so I created a 2nd ssh session since that felt quicker and cleared the cache from it since the login prompt seemed to come up quicker than the free command.
ronald@ronald-xubuntu:~$ free
total used free shared buffers cached
Mem: 1790976 1711844 79132 0 16852 1355248
-/+ buffers/cache: 339744 1451232
Swap: 1831372 99340 1732032
ronald@ronald-xubuntu:~$
ronald@ronald-xubuntu:~$ free
total used free shared buffers cached
Mem: 1790976 339528 1451448 0 1364 42780
-/+ buffers/cache: 295384 1495592
Swap: 1831372 99576 1731796
ronald@ronald-xubuntu:~$
Result a system that was back to normal again (until it fills up the memory with cached junk.
Contrast that with windows 7 where nothing is free after boot because half the ram is used by the cache (up to 4gb), and if it decides to index your disk you are toast because the cache grows and everything grinds to a halt with no way to get back half your memory. (Apart from writing a memory hog and die app).
Does not look very readable the output.
The point is
free memory increased from 79k to 1.4gig
cache reduced from 1.3gig to 42k
used memory reduced from 1.7gig to 339k
And I still had 99k in swap, but everything started working, since the stuff that was cached was all crap.
@Ronald
You may be encountering another issue here. Having a page cache full of unused pages will not cause a 20 minute delay in logins.
On my system, logins normally take 1-2 seconds. I tried setting swappiness to 100 and filling the page cache with useless junk, and I could still log in in less than five seconds. I tried filling 90% of my RAM with mlock’d pages, and then trashing the page cache. It still took less than 10 seconds.
Perhaps you had some huge, high priority or frequently forking processes running? If so, you could easily see a 20 minute delay in logins, and when it dies down and you’re able to run commands again (such as ineffectually clearing the cache), the system will continue to be responsive.