Okta disclosed the HollowByte DDoS vulnerability in OpenSSL, allowing attackers to disable servers with an 11-byte TLS payload by triggering
Identity verification solution vendor Okta disclosed a security vulnerability named HollowByte, which exists in the widely used encryption library OpenSSL [1]. This is a high-risk Denial of Service (DoS) attack weakness that allows unauthenticated remote attackers to disable server operations simply by sending a malicious TLS packet of 11 bytes [1][2]. Although the OpenSSL development team patched this issue in the 4.0.1 version released on June 9, 2026, the fix was not registered with a CVE number, nor was a CVSS risk score published; it was treated only as a general bug or hardening fix [1][3]. If not addressed promptly, attackers can cause the server to allocate disproportionately large memory space before completing any secure handshake, ultimately triggering memory exhaustion (OOM) and service termination [1][2].
The core mechanism of the HollowByte vulnerability lies in OpenSSL's memory allocation logic error when handling TLS handshake messages [1]. In the standard TLS handshake process, the client sends a handshake message containing a length indicator that explicitly specifies the length of the subsequent message body [3]. Older OpenSSL versions, upon receiving this header information, would directly trust the size claimed in the header and immediately pre-allocate a receive buffer of the corresponding size, even before the actual data was transmitted over the network [3]. Attackers exploit this logical flaw by forging an extremely large length value within the 11-byte packet, forcing the server to pre-allocate up to 131 KB of memory [3][4].
The OpenSSL development team resolved this issue by merging pull requests #30792, #30793, and #30794, changing the buffer growth strategy to an "incremental mode" [1]. The corrected logic no relies solely on the size claimed in the header but instead gradually expands memory only when network bytes are actually received [1]. This change ensures the server does not waste resources before data arrives. Okta's Red Team found during testing that running Nginx on a server with 1 GB of memory and triggering the vulnerability caused 547 MB of fragmented memory to be frozen, directly causing host memory shortage [1]. In a 16 GB memory environment, the attack could lock up 25% of memory, and since the number of connections did not reach the limit, traditional DDoS defense measures (such as connection-based threshold limits) could not block this type of attack [1][2].
OpenSSL, as a foundational encryption library, is widely embedded in various software ecosystems, including web servers (such as Nginx, Apache), programming language execution environments (such as Node.js, Python, Ruby, PHP), and database systems [1]. Due to its ubiquity, the impact scope of HollowByte is extremely broad; any server relying on an unpatched OpenSSL version faces risk [1]. Notably, many containerized environments (Containers) statically bind the OpenSSL version during build time; updating only the operating system's OpenSSL package may not resolve application-layer issues [1]. Additionally, since this vulnerability was not registered with a CVE number and the fix was released as a "silent patch," many automated monitoring systems relying on official CVE lists or standard security advisories may fail to detect the threat promptly, leading to delayed defense [3].
If attackers exploit this vulnerability, they can cause continuous memory consumption, and each 11-byte request can cause the server to pre-allocate up to 131 KB of memory until the process is manually restarted [3]. This not only leads to loss of service availability (Availability) but may also affect overall system stability due to memory fragmentation issues [2].
For the HollowByte vulnerability, the primary protection measure is to immediately upgrade OpenSSL to the patched version. Okta strongly urges users to upgrade to the new OpenSSL version promptly [1]. Specifically, users must upgrade OpenSSL to 4.0.1, or according to their current branch, upgrade to the corresponding backported versions: 3.6.3, 3.5.7, 3.4.6, or 3.0.21 [1][3]. For organizations using containerized architectures, all container base images statically bound to older OpenSSL versions must be rebuilt and updated to ensure the encryption library relied upon by the application layer is updated [3].
Beyond version upgrades, it is recommended to establish proactive monitoring mechanisms to detect anomalies. Strict memory RSS (Resident Set Size) trend alerts should be configured on observation platforms [1]. If the memory consumption of TLS service processes (such as load balancers, Nginx, Apache) shows progressive, step-like, and abnormal growth without significant traffic increase, this should be treated as a potential HollowByte attack indicator [1]. Additionally, it is recommended to review and implement TLS Handshake rate limiting at the load balancer or edge layer; while this cannot completely eliminate risk, it can significantly increase the attacker's cost [1][5].