Muhammed Senussi
Muhammed Senussi
  • Muhammed Senussi
JVM Memory When the Container Lies to YouJava

The ticket said "memory leak". The heap graph was flat. Both were true, and the second one was the clue.

The heap is a minority shareholder

A JVM in a container spends memory on metaspace, code cache, thread stacks, GC structures and direct buffers, none of which appear in a heap dump. Native Memory Tracking is the only honest accounting, and it is off by default.

  • Set -XX:MaxRAMPercentage rather than a fixed -Xmx, so the limit follows the pod.
  • Every thread is a stack. A thread-per-connection server with a 1MB default is a memory plan, not an accident.
  • Direct buffers live outside the heap and are freed when the referring object is collected, which may be much later than you assume.

We found ours in a Netty buffer pool sized for a machine four times larger than the pod. The heap was never the problem.

JVM Memory When the Container Lies to You — Figure 1
JVM Memory When the Container Lies to You — Figure 2
JVM Memory When the Container Lies to You — Figure 3

3 Comments

  • Sanne de Vries

    May 11, 2024

    MaxRAMPercentage is the single setting I wish every Java image shipped with by default.

  • Omar Haddad

    May 12, 2024

    هل تنصح بتشغيل Native Memory Tracking في الإنتاج دائماً؟

    • Muhammed Senussi

      Muhammed Senussi

      AuthorMay 12, 2024

      بمستوى summary نعم، فتكلفته حوالي 5% وقد وفّر علينا أياماً من التخمين. أما مستوى detail فأشغّله فقط أثناء التحقيق.

Leave a comment