A customer contacted our support recently, wondering why his Linux servers showed high memory utilization. We’ve talked a fair bit about monitoring Linux memory, and what it means. But this case was a little different. The customer showed that according to free, top, and LogicMonitor graphs, most of his server’s memory was in use – and not as file system cache.
However, the odd thing is that top and ps showed that the sum of all processes RSS (resident segment size) was consuming very little memory. So where was the memory going?
The most common reason people have for complaining that the free memory shown in top does not add up to the resident memory used by processes is because they fail to notice that most of the memory is used by the cache (which is, in effect, free memory.) But that was not the case here.
Looking at the contents of /proc/meminfo showed these two lines:
Slab: 4126212 kB
SReclaimable: 4096172 kB
So – almost 4G of memory was in use by the kernel slab memory structures – but almost all of that memory was reclaimable. (Or, in effect, free.)
So reclaimable slab space is yet another way that Linux memory can be in effect free, but not show up in the free memory statistics.
Does this change the way we should monitor the memory of Linux systems? Not at all. Linux will do a good job of managing the available physical memory and the data structures within it. When physical memory runs low, it will start swapping out to disk backed virtual memory. So long as you are monitoring the rate of blocks being swapped per second, and this is fairly low number – it doesn’t really matter what the physical memory is being used for.
Of course, you can dig into what is using the slab, using slabtop, /proc/slabinfo, and start tuning your /proc/sys/vm stuff, for things like min_slab_ratio…
But, unless you have active paging on this system, I would not bother.
Everything is working fine, and if memory gets low, the kernel will give up the reclaimable slab (4G worth.)
Steve is the founder of LogicMonitor.
Subscribe to our LogicBlog to stay updated on the latest developments from LogicMonitor and get notified about blog posts from our world-class team of IT experts and engineers, as well as our leadership team with in-depth knowledge and decades of collective experience in delivering a product IT professionals love.
LogicMonitor wants to exclusively invite you to TopGolf after Day 1 of CiscoLive. You will be joined by Michael Tarbet (Global VP of Sales, MSP) and Steve Kahn (Area VP, Channel Sales, North America) and we will have 2 bays reserved for a fun night of networking, eating and drinking, and golf!
Join this webinar to learn more about IT automation and how you can use it to drive innovation within your organization.
LogicMonitor wants to exclusively invite you to attend the Charlotte FC soccer game on Thursday, June 30th! Join us for a fun night of soccer and networking, and be prepared to carry on the Charlotte FC songs and chants!
When: June 30th at 6:00 pm
Where: Bank of America Stadium | 800 S Mint St, Charlotte, NC 28202
If you would like to attend, please register so we can lock in your ticket!