class loader leaks

the picture

you know how app servers (and some frameworks' dev modes) can reload your app without restarting the JVM? they do it with class loaders: each deploy gets a fresh loader that loads your classes, and undeploying means dropping that loader so the GC can throw the old version away. class loaders has how loaders work; this chapter is about the dropping part, which goes wrong a lot.

think of a hotel. a guest (your app) checks out, and housekeeping is supposed to clear the whole room. but the guest left a coat in the lobby cloakroom, and the room key is in its pocket. as long as the cloakroom (owned by the hotel, not the guest) holds that coat, the room counts as occupied. nobody can clean it, and the next guest gets a new room. check in and out enough times and the hotel is full of empty, occupied rooms.

why one reference holds everything

the references point in a direction that makes this inevitable: every object knows its class, every class knows its loader, and every loader keeps a list of every class it defined. so from any single object of your app, the GC can reach the loader, and from the loader, everything else.

the important direction: a reference from the app into the server is fine (the app dies, the reference dies with it). a reference from the server into the app is the leak, because the server outlives every deploy.

redeploy, ten times

a tiny "app server" that deploys a 301-class webapp in a fresh URLClassLoader, undeploys it (closes the loader, drops every reference it knows about), runs a GC, and repeats. the webapp is innocent except for one line, which depends on the mode:

the heart of the server (trimmed to the registry case):

static WeakReference<ClassLoader> deployAndUndeploy(int n, String mode, URL webapp) throws Exception {
    URLClassLoader loader = new URLClassLoader("webapp-v" + n, new URL[] { webapp }, Server.class.getClassLoader());
    Runnable app = (Runnable) loader.loadClass("App").getConstructor().newInstance();
    app.run();                                                    // deploy: loads 300 Page classes
    if (mode.equals("registry")) REGISTRY.put("listener-" + n, app);   // the one bad line
    loader.close();                                               // undeploy
    return new WeakReference<>(loader);                           // so we can ask: did it die?
}

and what it printed, with and without the bad line (jdk 25, G1):

no leak:   deploy  10: old loaders alive   0, classes loaded  1035, metaspace   1.0 MB
registry:  deploy  10: old loaders alive  10, classes loaded  4045, metaspace   8.3 MB

a real app is thousands of classes, not 301, so each leaked redeploy is tens of megabytes. and metaspace has no limit by default, so instead of an error you get a process that grows by that much every deploy. cap it, and you get the error instead:

$ java -XX:MaxMetaspaceSize=32m Server registry webapp/ 500
...
deploy  41: old loaders alive  41, classes loaded 13377, metaspace  31.0 MB
Exception in thread "main" java.lang.OutOfMemoryError: Metaspace
	...
	at webapp-v42//App.run(App.java:12)

note the webapp-v42// in the stack trace: since java 9, a frame names the loader its class came from, if the loader has a name. naming your loaders is free and makes this kind of bug much easier to read.

finding it

the JVM will tell you how many loaders it has, by name, while it runs:

+-- <bootstrap>
      +-- "platform", jdk.internal.loader.ClassLoaders$PlatformClassLoader
            +-- "app", jdk.internal.loader.ClassLoaders$AppClassLoader
                  +-- "webapp-v1", java.net.URLClassLoader
                  +-- "webapp-v2", java.net.URLClassLoader
                  +-- "webapp-v3", java.net.URLClassLoader

three copies of the webapp, after three deploys and three undeploys. that's the smoking gun. jcmd <pid> VM.classloader_stats adds the class count and metaspace per loader (301 classes and about 0.75 MB each, here). to find who holds an old one, take a heap dump, find the old loader object (Eclipse MAT has a "duplicate classes" report that lists classes loaded by more than one loader), and ask for its path to the GC roots. the first object on that path that doesn't belong to the app is the thing to fix.

the usual fixes, matching the usual roots:

• registrations: anything the app registers with something outside itself (listeners, MBeans, shutdown hooks, DriverManager, a logging framework's appenders) gets unregistered on undeploy.
• threads and executors: every thread the app starts, it stops. every executor it creates, it shuts down.
• ThreadLocals: remove() in a finally, every time, especially on threads the app doesn't own (see ThreadLocal internals).
• libraries in the right place: drivers and libraries that keep global state belong in the server's classpath, not inside each app.