[illumin8]

I've tried to educate Oracle DBAs on why top is wrong and their memory really isn't being used. It's painful and they often refuse to believe that I know what I'm talking about, and that they should use the free -m command to see what memory is actually available for use.

Is there any particular reason why Oracle DBAs are less likely to believe this? Perhaps it's because most of them grew up in legacy UNIX environments rather than Linux.

[krobertson]

I think this is a pretty common misconception overall. Working at startups, often find devs with multiple hats sometimes doing ops tasks. Seen many who hop on a system trying to diagnose some issue, fire up top and proclaim "OMG, the problem is we're running out of memory!"

2nd is explaining virtual/resident set size.

[francoisdevlin]

Okay, I'm one of those devs. What is this virtual/resident set size thing you're talking about?

EDIT: Thank you for all the helpful responses!

[illumin8]

Short and overly simple answer: If a program allocates 2GB of RAM, it might not need to use it all right away. The kernel pretends like it just gave the program 2GB of RAM, but it doesn't really need to flush all those unused pages of cache yet. It can still use them for other things like disk cache, and when the program tries to access those pages then the kernel will dynamically page them in and out as needed. This explains why a program might have 2GB virtual but only 128MB is resident in actual physical memory pages.

This is overly simple and there are a lot of nuances such as shared memory segments, etc.

[adavies42]

then there's over-commit, e.g. "you want to malloc 200% of the ram? here you go! enjoy! oh wait, you wrote to the second half? have fun finding the power button, sucker!"

can someone explain to me why so many distros, including "enterprise" server stuff, ship with /proc/sys/vm/overcommit_memory set to 1?

[aliguori]

Most software cannot cope with malloc failing. A little swapping is at least recoverable. A failed malloc almost guarantees that your application will crash--sometimes in a very unpleasant way.

[adavies42]

ime the main alternative is the box crashing, frequently without leaving behind enough information to know what went wrong. at least if the app crashes you have a pretty good idea who was incompetent.

[maushu]

I'm a developer too and this is what I understand (might be wrong):

- Virtual memory is basically "abstract memory" that is linked (mapped) to RAM or HDD, this means that by accessing this "memory" you might actually be accessing the HDD and, because of this, larger than physical ram.

- Virtual set size is the allocated virtual memory (the above) to the process.

- Resident set size is the allocated physical (RAM) memory to the process.

- Shared memory is memory that is shared on multiple processes, meaning that if you have 10 processes using 10mb of resident memory each and have 2mb of shared memory each, the total of resident memory used is not 100mb but 82mb.

[scott_s]

Virtual memory is actually more subtle than that.

At a mechanical level, virtual memory is permission from the operating system to use addresses in your address space. It is so called because, as you point out, it allows us to separate the concept of "memory for a process" from "physical memory on a chip." The reason I further refine the concept is that allocating virtual memory does not allocate actual memory. Let's look at an example:

  void* addr = mmap(NULL, 10 * 1024 * 1024, PROT_READ | PROT_WRITE, 
                    MAP_ANONYMOUS | MAP_PRIVATE, -1, 0);

That call allocates 10 MB of virtual memory. But there is no physical memory backing any of it. All that has happened is that the operating system has now said "Okay, starting at the address I return to you, you can now access 10 MB of memory. I will do all of the work of making sure physical memory backs the virtual memory when you access it." That is, once I try to access the memory, it will trigger a page fault, and the OS will find a page in physical memory to back my virtual memory. But until that happens, no memory - not in RAM, not on disk - backs the virtual memory.

[Steve_Baker]

The virtual size of a program includes all shared objects (shared libraries, shared copy on write memory pages, the read only executable pages and other shared memory) that a process uses in addition to the memory that only that particular process is using (its resident set size.) Shared objects like shared libraries can be mapped into memory by multiple processes and thus don't use any additional physical memory for each additional process that uses it.

The RSS thus usually indicates the amount of heap and stack a process is using that is unique to it.

[mgedmin]

RSS also includes shared memory pages (those that are currently resident).

[yread]

On Windows it's called committed memory

http://blogs.technet.com/b/markrussinovich/archive/2008/11/1...

(the article doesn't have an anchor there... but you know what I mean)

[sciurus]

There's a good overview of this in the first four pages of http://www.atoptool.nl/download/case_leakage.pdf

The rest of that document shows how to use atop to monitor per-process memory usage and identify a memory leak.

[ivan78]

Well, if you don't understand it, you must read some basic book about modern computer architecture. Short answer: It is possible to reserve memory address space without assigning actual physical memory. As your program runs it can dynamically assign physical memory pages to its address space when it needs it. Again - you must read a book if you want to understand it.

[ivan78]

Why downvoting? It may be not the best explanation, but IMO the closest to the way a "developer" should understand it. On Linux a virtual memory is committed with mmap call. Check this for details: http://stackoverflow.com/questions/2782628/any-way-to-reserv... This is basically how shared libraries and other shared objects are attached to your process.

[henrikschroder]

> Why downvoting?

> you must read a book if you want to understand it.

"Go read a book" is not a very constructive or helpful answer to a question that - as it turns out - could be answered with a brief comment.

[onemoreact]

It can be when a short description completely and accurately describes what's going on, but the person creating the comment needed to read a few books on the subject to actually understand both what's happening and why due to other issues.

EX: How does virtual memory get mapped to L1 cache?

[ivan78]

to henrikschroder: I expected that this reaction was caused by mentioning a book. It is my deepest belief that questions like memory management can not be answered with brief comment. You must read at least a basic book if you want to understand it in depth.

[ajross]

That's depressing. Developers are the ones who have to understand this. In IT it's common to find people (even "DBAs") who have made a career of following procedures someone else wrote.

[covati]

I don't think that is necessarily true. Most modern programming languages, as well as the relative cheap nature of RAM, have rendered a true understanding of the inner workings of RAM less important.

Would they be better at their job if they did? Probably. But do they have to? I'm not so sure anymore.

[ajross]

Sure, but we're not talking about the "narrow, career day-job programmers" here. We're talking about the "wear multiple hats and diagnose problems wherever they lie" set. Those people shouldn't be diagnosing problems with their own ignorance. At the very least they should understand what they don't know.

[krobertson]

Agreed. It can sometimes be a side effect of small companies, or young developers living the startup lifestyle. Not saying its bad or the root of the problem, just an observation.

More so, I think it comes more with modern languages that don't force you to manage memory anymore. When you don't need to alloc/free everything you're using, it is easy to get lazy. Factor in never generation who've never worked outside a managed memory language. There are minute details about retaining references, having multiple copies of the same data, or other leaks (file descriptors?).

[Legion]

2 minute fix: get people to use htop, not top.

[sciurus]

Having tried both, I'd recommend atop over htop. There's a great overview of its capabilities at https://lwn.net/Articles/387202/

[mgedmin]

While more powerful, atop is rather confusing. htop is friendly.

[sciurus]

I'm not sure what about using atop is confusing, but I'll admit to printing out and referring to the manpage when I discovered it. That was as much because of the great explanations atop's manual provided for the metrics it could display as it was because I wanted to learn the interactive commands.

Some things about htop that I could see as friendlier:

* Htop scrolls with the arrow keys, while atop uses ^F and ^B.

* Htop displays a reminder for some commonly-used commands at the bottom of the screen (e.g. that you need to press 'F1' for help), whereas in atop you have to remember the commands or look them up by pressing 'h' or '?'.

* Htop displays gauges for system-level activity. I think this is a bad tradeoff, though:

    CPU | sys      0% | user      1% | irq       0% | idle    799% | wait      0% |
    MEM | tot    7.8G | free    5.7G | cache 735.6M | buff  351.2M | slab  248.1M 

is much more useful than

    Avg[|                                        0.2%]
    Mem[|||||||||||||||||                 1008/7967MB]

There are features of htop that I wish atop had. For example, the toggleable display of threads, the tree view, and integration with strace and lsof. Even without those I find atop more useful, but YMMV.

The killer feature for atop is logging per-process performance data and reviewing it after the fact.

[JadeRobbins]

Thanks! htop is MUCH better than plain ol top.

[spudlyo]

You'd be surprised how good and feature rich plain ole top actually is. My guess is you rarely deviate from the default options. I made a video which shows how you can get top to show you some pretty interesting stuff.

http://www.youtube.com/watch?v=yFKRsLj_Jhg

[laurentoget]

thanks for sharing!

[seclorum]

Teach them to interpret and understand cat /proc/meminfo .. one of the most interesting things you can do with this is pipe it into gnuplot and watch it over time as things happen. Try it sometimes .. you might get through to one or two.

[ivan78]

If you want to be more enterprisey, you can pipe it to SNMP counter and then draw a graph with your Network Monitoring System. It is much more convenient if you have more that few servers.

[seclorum]

Maybe that's more 'enterprisey', but its more 'system administrator'-ey to just use an onboard gnuplot and look at real graphs without putting much SNMP/traffic-load on the system of interest .. the purpose is to educate the user on how the system works, not use it as a monitoring device for lazy sysadmins.

[gaius]

I've been an Oracle DBA for 15 years, and no-one uses top for that, as it's well known not to account in any sort of meaningful way for the way Oracle uses shared memory. The only thing it's useful for is seeing which of the sysadmin's Perl scripts is chewing the CPU.

[ivan78]

As a longtime Linux user/admin and beginning Oracle DBA I now know that all memory should be occupied by Oracle, not by OS. :-) Serious mode on: I'm sure it's a big problem to be narrow expert. They can be brilliant specialist in their field, but one step aside and they are absolutely helpless.

[illumin8]

Thanks for the great comment. I really wish more people like yourself at least understood Linux memory management. I think it will truly help you to be an exceptional DBA. I can also understand how an expert DBA might not know or care too much about the OS underneath his software, although I would argue that if you understand the OS fundamentals, you will be that much better at whatever specialty you have.

[ivan78]

Thank you. I'm on my way to become exceptional DBA. Oracle is only 2 years younger than me, but unfortunately it has been developing much faster than I am. It literally takes years just to become familiar with all these layers of Oracle technologies. So, it's really hard to blame DBAs for not knowing how underlying OS works. :-)

[greedo]

I've had the same discussion with Websphere administrators who can't grasp the concept of caching and the role of swapping. I even had one admin think that modifying the swappiness setting to keep memory free would be a good idea...