skip to content
everything.deepDives.outOfMemoryZoo

the OutOfMemoryError zoo

every OOM message, what it means, and what to do.

chapter
chapter 17.12
reading time
8 min read

there isn't one OutOfMemoryError, there's a family, and the message after the colon tells you which pool ran dry: the heap (Java heap space), class metadata (Metaspace), OS threads (unable to create native thread), direct buffers (... direct buffer memory), or no memory at all, just a length limit (Requested array size exceeds VM limit).

read the message first. the fix for each one is different, and "add more -Xmx" only fixes one of them.

the picture

you know how an OutOfMemoryError in a log makes everyone reach for -Xmx? that's like hearing "we ran out" in a kitchen and ordering more flour, when it was the gas that ran out. a JVM has several separate pools, each with its own limit, and each one throws its own flavour of the same error.

  • ├── java heapJava heap space · GC overhead limit exceeded
  • ├── max array lengthRequested array size exceeds VM limit
  • ├── Metaspace
  • │ └── compressed class spaceCompressed class space
  • ├── OS threadsunable to create native thread
  • ├── direct buffersCannot reserve ... direct buffer memory
  • └── the container limitno error: SIGKILL, exit code 137
which pool each message comes from. only the first branch is about -Xmx. the last one isn't an OutOfMemoryError at all, which is what makes it the worst.

the zoo

every one of these was reproduced on jdk 25 with a tiny program and a tiny limit (-Xmx16m, -XX:MaxMetaspaceSize=16m and so on), so they die in milliseconds instead of hours. the messages are copied from the real output.

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

ran out ofthe heap. a full GC ran and still couldn't find room for the object you asked for, within -Xmx.

fixeither the live data really is bigger than the heap (raise -Xmx), or something is holding on to objects it shouldn't. if a heap dump says the second, it's a leak.

reproduce
$ java -Xmx16m Heap.java
while (true) keep.add(new byte[1024 * 1024]);
thrown by the JVMHeapDumpOnOutOfMemoryError: firesExitOnOutOfMemoryError: fires
pick an animal. the red line is the real message. the chips at the bottom say who threw it, and whether the two "on OOM" flags react to it: they only fire for errors the JVM itself throws, which is not all of them.

a few patterns in there are worth saying out loud:

  • two of them aren't about memory. Requested array size exceeds VM limit and Required array length ... is too large are about a length past Integer.MAX_VALUE - 2. you'd get them with a terabyte of free heap.
  • unable to create native thread is about threads. this one says "possibly out of memory" in its own message, and it's usually a thread limit: 6,124 threads in on macOS (which caps a process at 6,144), a ulimit -u or a container's pid limit on linux.
  • the collector changes the message. the exact same program said GC overhead limit exceeded under Parallel and Java heap space under G1. only Parallel implements the overhead limit.

what the JVM tries first

a heap OutOfMemoryError is never the first thing that happens. by the time you see one, the collector has already tried everything it's allowed to:

hover a stage.

the path from a failed allocation to the error, for the heap (G1, jdk 25). the soft-reference step is a promise in the spec: every SoftReference is cleared before a heap OOM is thrown.

that's also why a JVM near the edge feels so slow before it dies: it's running full GCs back to back, each one freeing a little less (what that looks like in a log is in reading GC logs). the error is the end of a long, expensive goodbye.

a thread dies, not the JVM

predict the output
run with -Xmx16m. what does main print?
java
import java.util.*;
 
public class Worker {
    public static void main(String[] args) throws Exception {
        Thread worker = new Thread(() -> {
            List<byte[]> keep = new ArrayList<>();
            while (true) keep.add(new byte[1024 * 1024]);
        }, "worker");
        worker.start();
        worker.join();
        System.out.println("main is still alive, worker is " + worker.getState());
    }
}

which is why production services usually want the JVM to give up on the first one:

terminal
$ java -Xmx16m -XX:+ExitOnOutOfMemoryError Heap.java
Terminating due to java.lang.OutOfMemoryError: Java heap space
$ echo $?
3
flagwhat it does on an OOM the JVM throws
-XX:+HeapDumpOnOutOfMemoryErrorwrites an .hprof first (-XX:HeapDumpPath= picks where). the evidence
-XX:+ExitOnOutOfMemoryErrorexits immediately with code 3, so your orchestrator restarts a clean process
-XX:+CrashOnOutOfMemoryErrorcrashes on purpose, writing an hs_err file and a core dump if they're enabled
-XX:OnOutOfMemoryError="cmd %p"runs a command, %p is the pid. older scripts use it for kill -9 %p

HeapDumpOnOutOfMemoryError plus ExitOnOutOfMemoryError is a sane default: you get the dump for later and a fresh process now. but as the zoo shows, neither fires for direct buffer memory, for thread creation, or for the length limits thrown by library code. those still need to be caught by your logs.

the one that doesn't throw

the scariest out-of-memory has no message at all. a container has a memory limit for the whole process: heap, metaspace, thread stacks, direct buffers, everything in native memory. go over it and the kernel's OOM killer sends SIGKILL. the JVM gets no chance to throw, log, dump or run a hook. all you see is the process gone and an exit code of 137 (128 + signal 9). if your service restarts with 137 and your logs are clean, look at the process's total memory, not at the heap.

one level deeper:StackOverflowError and the JVM's own allocation failures+

StackOverflowError is the heap error's cousin: a thread's stack (one per thread, sized by -Xss) ran out of room, almost always from runaway recursion. it's covered in stack and heap. and when the JVM itself can't get native memory for its own internals (malloc or mmap fails), there's no exception to throw: it crashes with an hs_err_pid<pid>.log file that starts with "There is insufficient memory for the Java Runtime Environment to continue", which usually means the machine or container is out of memory, not the heap.

keylines
  • OutOfMemoryError is a family. the message after the colon names the pool that ran out.
  • heap: leak or undersized. metaspace: classes that never unload. threads: OS limits. direct: native buffers.
  • two of them are length limits, not memory: past Integer.MAX_VALUE - 2 elements.
  • an OOM kills the thread that hit it, not the JVM. use -XX:+ExitOnOutOfMemoryError plus a heap dump.
  • exit code 137 with clean logs: the container killed the process. no error is ever thrown.