microbenchmarks with JMH

the benchmark that measured nothing

you know the move: wrap the thing in a loop, check the clock before and after, divide. here it is, timing Math.sqrt and a small hash-mixing function, 200 million calls each, once with the result thrown away and once with it added to a sum:

static int mix(int h) {
    h ^= h >>> 16;
    h *= 0x85ebca6b;
    h ^= h >>> 13;
    h *= 0xc2b2ae35;
    return h ^ (h >>> 16);
}
 
void main() {
    int n = 200_000_000;
    for (int round = 1; round <= 5; round++) {
        long t0 = System.nanoTime();
        for (int i = 0; i < n; i++) Math.sqrt(i);          // result ignored
        long t1 = System.nanoTime();
        for (int i = 0; i < n; i++) mix(i);                // result ignored
        long t2 = System.nanoTime();
        int acc = 0;
        for (int i = 0; i < n; i++) acc += mix(i);         // result used
        long t3 = System.nanoTime();
        double sum = 0;
        for (int i = 0; i < n; i++) sum += Math.sqrt(i);   // result used
        long t4 = System.nanoTime();
        IO.println("round " + round + " (ps per call): sqrt ignored " + (t1 - t0) / (n / 1000)
            + ", mix ignored " + (t2 - t1) / (n / 1000) + ", mix used " + (t3 - t2) / (n / 1000)
            + ", sqrt used " + (t4 - t3) / (n / 1000) + "   " + acc + " " + sum);
    }
}

the real numbers from jdk 25 on an apple M4 Pro (rounds 3 to 5 agree to within a few picoseconds; here in nanoseconds per call):

the first round (before C2 had finished with main's loops) printed 6 picoseconds for the ignored sqrt, which is already absurd: a 4.5 GHz core can't do anything in a fortieth of a clock cycle. from round three on, both "ignored" loops took no time at all.

this is dead code elimination: mix has no side effects, so once it's inlined, a call whose result goes nowhere is a call that doesn't need to happen, and an empty loop doesn't need to run. it's a great optimization for real code and a disaster for benchmarks.

what JMH does about it

the same experiment as JMH benchmarks:

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Warmup(iterations = 5, time = 1)
@Measurement(iterations = 5, time = 1)
@Fork(2)
@State(Scope.Thread)
public class DeadCode {
    double x = 12345.678;                         // a field: the JIT can't treat it as a constant
    static final double CONSTANT = 12345.678;     // a constant: it can
 
    @Benchmark public void baseline() {}
    @Benchmark public void sqrtDiscarded() { Math.sqrt(x); }
    @Benchmark public double sqrtReturned() { return Math.sqrt(x); }
    @Benchmark public void sqrtBlackhole(Blackhole bh) { bh.consume(Math.sqrt(x)); }
    @Benchmark public double sqrtFolded() { return Math.sqrt(CONSTANT); }
    @Benchmark public void logDiscarded() { Math.log(x); }
    @Benchmark public double logReturned() { return Math.log(x); }
}

two defences, both visible there:

• return the result, or consume it. JMH takes whatever a @Benchmark method returns and feeds it to a Blackhole, an object designed so the JIT can't prove the value is unused. with more than one value, take a Blackhole parameter and call bh.consume(...) on each. (on jdk 25 JMH uses the JVM's built-in "compiler blackholes", and prints a note saying so at startup.)
• read inputs from @State fields, not constants. Math.sqrt(CONSTANT) was computed by the JIT once, at compile time, and the benchmark measured returning a number.

and one honest surprise. Math.log was not eliminated: discarded and returned cost the same. on this CPU the JIT turns Math.log into a call to a hand-written stub it can't see inside, so it can't prove the call is side-effect free. on another JVM or CPU, the same broken benchmark might get deleted. a benchmark that's only correct by accident on your machine is still broken.

warmup, forks, iterations

the other trap is time. a method's first call and its ten-thousandth call are running different code: interpreted, then C1-compiled with profiling, then C2-compiled with everything the profile taught it. JMH can show this directly with single-shot mode, which times individual calls in a fresh JVM:

that's why a JMH run is shaped the way it is:

Benchmark            Mode  Cnt   Score   Error  Units
Warm.groupByLength   avgt   15  41.585 ± 0.735  us/op

• forks (@Fork): each benchmark runs in its own freshly started JVM, so benchmark A's profile can't pollute benchmark B's. the JIT compiles call sites based on the types it has seen; run a benchmark that passes ArrayList and then one that passes LinkedList in the same JVM, and the second one inherits a megamorphic call site the first one never had. more than one fork also shows you how much the result varies from JVM to JVM (it can, because JIT decisions aren't deterministic).
• warmup iterations (@Warmup): run the benchmark for a while and throw the numbers away, until the JIT and the caches settle.
• measurement iterations (@Measurement): run it more, and keep these.
• modes: AverageTime (time per call), Throughput (calls per time), SampleTime (a latency histogram of individual calls), SingleShotTime (cold, one call per iteration, which is how the curve above was made).

running one

the official way is the maven archetype (org.openjdk.jmh:jmh-java-benchmark-archetype), which sets up the annotation processor and builds a benchmarks.jar. the processor matters: JMH is itself an annotation processor that generates the harness code around your methods. without a build tool, it's two commands:

$ javac -cp jmh-core-1.37.jar -processorpath jmh-generator-annprocess-1.37.jar:jmh-core-1.37.jar:jopt-simple-5.0.4.jar:commons-math3-3.6.1.jar -d classes DeadCode.java
$ java -cp classes:jmh-core-1.37.jar:jopt-simple-5.0.4.jar:commons-math3-3.6.1.jar org.openjdk.jmh.Main DeadCode
...
Benchmark               Mode  Cnt  Score   Error  Units
DeadCode.baseline       avgt   10  0.384 ± 0.001  ns/op
DeadCode.sqrtDiscarded  avgt   10  0.384 ± 0.010  ns/op
DeadCode.sqrtReturned   avgt   10  0.707 ± 0.009  ns/op

useful flags: -f, -wi, -i override the annotations; -prof gc adds allocation per operation (often more telling than time); -prof stack shows where threads spend time; on linux, -prof perfasm prints the hottest generated assembly, which is how you prove the JIT did what you think.

@Benchmark
public void build() {
    List<Integer> list = new ArrayList<>();
    for (int i = 0; i < 1000; i++) list.add(i);
}

@State(Scope.Thread)
public class SortBench {
    int[] data = new Random(42).ints(10_000).toArray();
 
    @Benchmark
    public int[] sort() {
        Arrays.sort(data);
        return data;
    }
}