native memory

the picture

you know how a container running -Xmx1g can still get killed for using too much memory, with no OutOfMemoryError anywhere? the heap is the tip of the iceberg. it's the part you configured, the part every dashboard shows, and the part the GC manages. under the waterline is everything else the JVM needs to run your code, and the container's memory limit counts all of it.

the memory map chapter drew those areas. this one measures them, grows them one at a time, and shows how to find the one that's leaking.

reserved, committed, resident

the JVM gets memory from the OS in two steps, reserve and commit. there's a third number the OS keeps on its own, and it's the one your container limit cares about.

a page only becomes resident the first time something writes to it. so a JVM that commits a 256 MB heap at startup sits at 51 MB of RAM until the heap actually fills up. NMT reports reserved and committed. ps, top and the container report resident.

turning on NMT

# start the JVM with tracking on (summary is cheap; detail adds call sites and costs more)
java -XX:NativeMemoryTracking=summary -jar app.jar
 
# ask it, any time
jcmd <pid> VM.native_memory summary scale=MB
 
# find a leak: take a baseline now, diff against it later
jcmd <pid> VM.native_memory baseline
jcmd <pid> VM.native_memory summary.diff scale=MB

here's part of the summary from the experiment below, at the point where it had 533 threads, 21,065 classes and 128 MB of direct buffers:

Total: reserved=2902MB, committed=583MB
       malloc: 189MB #228253, peak=202MB #228364
       mmap:   reserved=2713MB, committed=394MB
-                 Java Heap (reserved=264MB, committed=256MB)
-                     Class (reserved=1036MB, committed=31MB)
                            (classes #21065)
-                    Thread (reserved=1066MB, committed=42MB)
                            (threads #533)
                            (stack: reserved=1064MB, committed=40MB, peak=40MB)
-                      Code (reserved=244MB, committed=8MB)
-                        GC (reserved=60MB, committed=60MB)
-                     Other (reserved=128MB, committed=128MB)
-                 Metaspace (reserved=67MB, committed=23MB)

NMT isn't free: Oracle's documentation puts it at a 5 to 10% slowdown, and it can only be switched on at startup. turn it on when you're investigating, not forever.

growing it one category at a time

this little program grows one native category per step and asks NMT (and ps) about itself after each one. same JVM, same 256 MB heap, start to finish.

so the same -Xmx256m process went from 51 MB to 486 MB of RAM, and by the end the heap was less than half of what NMT counted (256 of 583 MB committed). the rough costs, from this run:

• a platform thread: a 2 MB stack reserved (1 MB on linux x64), about 100 KB resident while idle, more if it ever recursed deep. a pool of 2,000 threads is a few hundred MB before it does any work. virtual threads keep their stacks on the heap instead.
• a direct buffer: exactly its capacity, plus a little bookkeeping.
• a class: metaspace plus class space plus symbols. tiny classes here cost about 2.6 MB per thousand; real classes, with real methods, cost more.

direct buffers

a direct ByteBuffer is the classic way to hand memory to native code and I/O without copying (see buffers and channels). it's also the classic native leak, because of how it's freed.

the native memory is released by a cleaner when the GC notices the small heap object is unreachable. so the size of the native memory has nothing to do with when the GC runs. a program can churn through gigabytes of direct buffers while the heap stays so calm that no GC ever happens. the JDK patches over this by calling System.gc() itself when you hit the limit, which is why this works:

import java.nio.ByteBuffer;
 
// allocates 1,000 direct buffers of 1 MB and throws every one away immediately.
public class Buffers {
    public static void main(String[] args) {
        for (int i = 1; i <= 1000; i++) {
            ByteBuffer b = ByteBuffer.allocateDirect(1 << 20);   // 1 MB outside the heap
            b.put(0, (byte) i);                                  // use it, then drop it
        }
        System.out.println("done: 1000 MB allocated, 0 kept");
    }
}

$ java -XX:MaxDirectMemorySize=64m Buffers.java
done: 1000 MB allocated, 0 kept
 
$ java -XX:MaxDirectMemorySize=64m -XX:+DisableExplicitGC Buffers.java
Exception in thread "main" java.lang.OutOfMemoryError: Cannot reserve 1048576 bytes of direct buffer memory (allocated: 67108864, limit: 67108864)
	at java.base/java.nio.Bits.reserveMemory(Bits.java:178)

finding a native leak

when RSS keeps climbing and the heap doesn't, take an NMT baseline, wait, and diff. here's a "buffer pool" that only ever grows, five seconds after the baseline:

Total: reserved=1973MB +211MB, committed=597MB +205MB
-                 Java Heap (reserved=264MB, committed=256MB)
-                     Other (reserved=256MB +205MB, committed=256MB +205MB)
                            (malloc=256MB +205MB #256 +205)

Other, +205 MB, in exactly 205 allocations since the baseline: that's direct buffers, one megabyte at a time. the category tells you which kind of memory to look for; a heap dump then tells you who holds the ByteBuffers (they're small, but they're there).