## Basic information **Project name:** Eclipse OpenJ9 **Project id:** technology.openj9 ## What are the affected versions? Confirmed present in source at tag `openj9-0.62.0` (current/latest tag as of 2026-09-09). Dynamically reproduced against the official prebuilt `ibm-semeru-runtimes:open-21.0.10_7.1-jdk` Docker image (bundled OpenJ9 0.57.0) — the vulnerable function is byte-for-byte identical across this range, confirmed by direct source comparison against the `openj9-0.62.0` tag. ## Details of the issue **Title:** Pre-authentication remote denial of service in JITServer via uint32 integer-overflow infinite loop in `MessageBuffer::computeRequiredCapacity()` **Weakness:** CWE-191 (Integer Overflow) leading to CWE-835 (Loop with Unreachable Exit Condition) — pre-authentication, network-reachable, High severity (CVSS 3.1 `AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H` ≈ 7.5). JITServer — OpenJ9's optional network daemon for remote JIT compilation — reads a 4-byte, fully attacker-controlled length field as the first bytes of any new TCP connection, before any protocol-version check or authentication occurs. If that value falls in the range `(0x80000000, 0xFFFFFFFF]`, the buffer-growth routine `MessageBuffer::computeRequiredCapacity()` enters a genuine infinite loop due to unsigned 32-bit integer wraparound. The compilation-worker thread that accepted the connection is permanently pinned at ~100% CPU with no possibility of self-recovery. Because JITServer's health/liveness endpoint is decoupled from the compilation-thread pool, this is invisible to standard orchestrator health checks. A handful of ~8-byte TCP connections (one per pool thread) is enough to exhaust the entire compilation-thread pool and cause a full, silent denial of service for every client JVM sharing that JITServer instance. **Root cause** — `runtime/compiler/net/MessageBuffer.cpp` (tag `openj9-0.62.0`), lines 137-146: ```cpp uint32_t MessageBuffer::computeRequiredCapacity(uint32_t requiredSize) { if (requiredSize <= _capacity) return _capacity; uint32_t extendedCapacity = _capacity * 2; while (requiredSize > extendedCapacity) extendedCapacity *= 2; return extendedCapacity; } ``` `_capacity` is always a power of two, starting at `INITIAL_BUFFER_SIZE = 32768`. For `requiredSize` in `(0x80000000, 0xFFFFFFFF]`, the doubling sequence passes exactly through `2^32` and wraps the `uint32_t` to 0; once `extendedCapacity == 0`, the loop condition is permanently true and `0 * 2` stays 0 forever. **Call path (fully traced, `openj9-0.62.0`):** `runtime/compiler/net/CommunicationStream.cpp`, `readMessage()`, lines 81-92, reads the attacker's first 4 bytes as `serializedSize` with no upper-bound check, and calls `msg.expandBuffer(serializedSize, bytesRead)` whenever `serializedSize > bufferCapacity` → `Message::expandBuffer()` (`Message.hpp:354`) → `MessageBuffer::expand()` (`MessageBuffer.cpp:78-97`) → `computeRequiredCapacity()`. Reached from `JITServerCompilationThread::processEntry()` → `stream->readCompileRequest()` (`JITServerCompilationThread.cpp:1070`, sole call site, unconditional) → `ServerStream::readCompileRequest()` (`ServerStream.hpp:149-177`), whose `readMessage(_cMsg)` is its first statement (line 151); the protocol version-compatibility guard at line 152 runs strictly *after* `readMessage()` returns, so it cannot protect against a hang occurring inside `readMessage()` itself. No authentication, session/clientUID check, or size cap exists anywhere on this path, with or without TLS configured. **Dedup vs. CVE-2026-6918 / GHSA-q393-vr4c-969r:** I checked the nearest existing advisory in this exact file, CVE-2026-6918 (OOB read in `Message::deserialize()`, fixed in 0.59.0 via commit `c53b6b93f422b77b3395a3464d5578b706e9a618`). I fetched that fix's actual patch and confirmed it only adds a bounds check inside `MessageBuffer::readData()` (the cursor-advance function used while walking already-received message descriptors during `deserialize()`) — it does not touch `computeRequiredCapacity()`, `expand()`, or the initial buffer-growth call site in `readMessage()`, all confirmed unchanged at `openj9-0.62.0`. This is a distinct root cause, distinct trigger, distinct impact (permanent hung thread invisible to health checks, vs. a crash an orchestrator would restart), and distinct code location in the same file. **Suggested remediation:** Add an explicit upper bound on `serializedSize`/`requiredSize` before it drives buffer growth (reject any declared size above a sane maximum), and/or make `computeRequiredCapacity()` detect the overflow explicitly (e.g. check for `extendedCapacity == 0` after multiplication, or use a 64-bit intermediate) rather than relying on the doubling loop to terminate naturally. ## Steps to reproduce **1) Mathematical proof of the root cause** — a byte-for-byte Python re-implementation of the arithmetic with correct uint32 wraparound: ```python def compute_required_capacity(required_size, capacity=32768): if required_size <= capacity: return capacity extended = (capacity * 2) & 0xFFFFFFFF iterations = 0 while required_size > extended: extended = (extended * 2) & 0xFFFFFFFF iterations += 1 if iterations > 100: return None # never converges return extended print(compute_required_capacity(0xFFFFFFFF)) # -> None (never converges, stuck at 0) print(compute_required_capacity(0x80000001)) # -> None (never converges, stuck at 0) print(compute_required_capacity(0x80000000)) # -> 2147483648 (terminates, boundary case) print(compute_required_capacity(100000)) # -> 131072 (terminates normally) ``` **2) Real execution against the official, unmodified compiled binary:** ``` docker pull ibm-semeru-runtimes:open-21.0.10_7.1-jdk docker run -d --name jitserver-test -p 38400:38400 \ ibm-semeru-runtimes:open-21.0.10_7.1-jdk /opt/java/openjdk/bin/jitserver ``` Confirmed listening on `127.0.0.1:38400` (container log: `JITServer is ready to accept incoming requests`). Baseline container CPU (idle): **0.45%**. Test client (Python, single connection, single write): ```python import socket, struct s = socket.create_connection(("127.0.0.1", 38400), timeout=5) s.sendall(struct.pack("<I", 0xFFFFFFFF)) # exactly 4 bytes: FF FF FF FF # no further data sent; connection left open, then closed ``` **Result:** container CPU rose to **109.86%** within ~10 seconds. Per-thread inspection (`top -H -b -n1 -p 1` inside the container) showed one "JIT Compilation" worker thread pinned at **99.9% CPU** in `R` state, while every other thread (including the listener) stayed at 0.0%. **The spin persisted after the test client fully disconnected** (re-checked ~35+ seconds later: still 99.9% CPU, cumulative CPU time climbing 1:1 with wall-clock time) — proving the thread is permanently stuck regardless of connection state. **Negative control:** a separate connection sending a well-formed small length value (`100`, little-endian) produced no thread spin and no CPU anomaly, confirming the effect is specific to the boundary-overflow value. Container was stopped and removed after the test; no persistent state was left behind. *(Note: dynamic reproduction used the readily-available prebuilt OpenJ9 0.57.0 binary; the vulnerable function is confirmed byte-for-byte identical at `openj9-0.62.0` via direct source comparison, so this corroborates the source-level finding rather than being a different bug.)* ## Do you know any mitigations of the issue? Add an explicit upper bound on `serializedSize`/`requiredSize` before it is used to drive buffer growth (reject, with a normal protocol error, any declared message size above a sane maximum — legitimate JITServer messages are nowhere near `2^31` bytes), and/or rewrite `computeRequiredCapacity()` to detect the overflow condition explicitly (e.g., check `extendedCapacity == 0` after multiplication, or use a 64-bit intermediate) rather than relying on the doubling loop to terminate naturally. [client_output.log](/uploads/fbfcc1891d3d0ded0ee32b5bfb80ecb2/client_output.log) [poc_client_baseline.py](/uploads/79166714ac230c7e3f261d55cc317289/poc_client_baseline.py) [poc_client.py](/uploads/6fe9f387ce26a07c903867c466ae507f/poc_client.py) --- **Reported by:** Duc Anh Nguyen (VinSOC) — HackerOne handle `@gladiator9797` — mra9797@gmail.com I'm also happy to coordinate on disclosure timing, and note that I have a second, related but distinct finding in the same JITServer message-parsing subsystem (an out-of-bounds heap read via a different unvalidated field, also unaddressed by the CVE-2026-6918 fix) that I'll file as a separate report shortly. ## Attachments [POC-openj9-jitserver-dos.md](/uploads/ad3ac84a2524724f295b7b9a6f9d1a39/POC-openj9-jitserver-dos.md)