The 0xDABB of Doom: CVE-2021-25641

Introduction

When I previously wrote the original Dubbo publication, we disclosed that issue as it was mitigated by the vendor. While the Dubbo “HTTP” protocol in that disclosure was trivially vulnerable to the most common Java deserialization attacks (as evidenced by the immediate cropping up of exploits for Dubbo as soon as a very broad description of the issue was published in a mailing list) - the so-called “HTTP” was an elective protocol, and honestly - there was no apparent reason to choose it over the default Dubbo protocol. This made it an unlikely misconfiguration, and most devs would probably just stick to the default Dubbo protocol; this made it very tempting to investigate Dubbo further. The “Dubbo” protocol is a proprietary binary protocol which required some reverse-engineering of its inner logic and how it is being consumed by the endpoint. It actually wraps around the binary of the serialized object, adding meta-data and additional objects to the stream. The underlying deserializer defaults to Hessian2, configured to exclude many of the “known dangerous” classes often used to exploit such attacks. However, there were some ways to manipulate this protocol, such as elect the deserializer, which would then lead to RCE.

Vulnerability Overview

The Dubbo protocol is a proprietary protocol developed for Apache Dubbo for transmission of objects between Dubbo services. Apache Dubbo providers and consumers using certain versions of Dubbo, when configured to accept the default “Dubbo” protocol, allows a remote attacker to send a malformed stream containing a malicious object to the exposed service which would result in Remote Code Execution. This RCE occurs with no authentication, and no knowledge on an attacker’s part is required to exploit this vulnerability – only an open port on a Provider server running Apache Dubbo. This issue applies to default configuration, where the built-in Dubbo protocol is used.

Severity

Checkmarx considers this vulnerability to have a CVS Score of 10.0 (Critical), as it is an unauthenticated remote code execution vulnerability which provides privileges at the Dubbo service’s permission level, allowing complete compromise of that service’s confidentiality, integrity, and accessibility. CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H

What’s Going On – Understanding the Dubbo Protocol

The server-side “Dubbo” protocol on a provider is used to expose a service, and methods within this service, to external consumers. The “Dubbo” protocol has two general modes served over the same port:

1. Telnet-like mode, where a CLI is available to allow invoking a method exposed by the provider service
2. An RMI mode, where the TCP packet is then treated as a byte stream, which is deserialized to be used with the service offered by the Dubbo provider

Fig. 1 - Dubbo Telnet

While default behavior is Telnet-like, the way the remoting protocol is invoked is by providing a header which is built in the following manner:

1. Two magic bytes – 0xdabb
2. A flag byte – this flag determines:
   1. If this stream is a request or a response
   2. If this transaction is one-way or two-way
   3. Deserialization mechanism identifier (Hessian2, Kryo, Java etc.)
3. Padding null-bytes, to allow for dynamic length
4. Length of stream

This method is reminiscent of port-knocking, but instead of sending a stream of bytes to open a port, two preamble bytes are used to switch between protocols (Telnet-like mode and RMI-like mode). The preamble header is then followed a stream sequence containing the following data:

1. Dubbo Protocol version
2. Service name (“Path”)
3. Service version
4. Method name
5. Method argument types
6. Method arguments
7. Request attachments, which may contain additional data

All of the above data is read as a UTF string with the exception of 6 – method arguments; this is because a method argument may contain any Java object which may be passed to the Dubbo Provider, while other data is provided as Strings, to be used with reflections inside the code. The open Dubbo port can be trivially fingerprinted as such by readily available tools, such as Nmap.

Fig. 2 – Dubbo telnetd Service, As It Appears on Nmap

The raw TCP request, including preamble and metadata, sends a request and receives a response over the same TCP stream, as demonstrated via Dubbo’s Quick Start sayHello demo:

Fig. 3 – Dubbo Protocol Request in Its Default Serialized Form

Fig. 4 – Dubbo Protocol Response in Dubbo Demo

Fig. 5 – Dubbo Protocol Breakdown

Attacking the Dubbo Protocol

The most commonly known Java-based gadget chains used for exploiting deserialization attacks rely on several key classes – these classes appear to be blacklisted on the default Hessian2 serializer. The Dubbo protocol relies on the flags byte (3^rd byte of the stream) to determine which serializer\deserializer to use when processing this stream. While the default is Hessian 2, and attacker may tamper with this flag to choose a different deserializer, exposing multiple additional options for serialization. These additional deserializers do not offer the same protection Hessian 2 does. An attacker can modify the flag to choose either of the following protocols:

• FST Serialization
• Kryo Serialization

Both of these protocols are binary serialization protocols, and successfully deserialize the FastJSON gadget-chain.

Fig. 6 – The Majestic, Feral Beauty of a Kryo-Serialized FastJSON Gadget ByteStream

For the purpose of having the gadget chain deserialized, however, a compliant stream should be created containing the required preamble, headers, and metadata. Since the exploit does not actually require Dubbo to successfully dispatch the request (in versions <= 2.7.5), but rather simply successfully read, parse and deserialize it, the stream itself can contain random or arbitrary strings. These strings are only there to serve as padding and satisfy multiple “readUTF” operations, until readObject is called on the stream, at which point FST or Kryo are the chosen deserializer and the exploit is triggered.

Recreating the Issue

An attacker can reimplement the Dubbo protocol with its byte preamble, header flags and content length, as well as dummy strings and a malicious object to satisfy the endpoint’s implementation for reading it. The same gadget chain used in the previous exploit for 2.7.3 HTTP protocol, which allows remote OS command execution was found in the scope of vanilla Apache Dubbo, so long as dubbo-common is also in scope (more on this later).

Proof-of-Concept

Recreating a Victim Dubbo Instance for PoC

Either use the code sample from my previous research blog or follow this guide:

1. Follow the Official Apache Dubbo Quick-Start guide until a functioning provider and registry are successfully created (see here for a full sample)
2. Ensure dubbo-common, or a package where this dependency is not optional, are in scope

 Add the following dependencies to enable Spring Framework, Dubbo and Dubbo Common (presented in Maven pom.xml form for ease of use):

Triggering Vulnerability PoC

See this link for functioning POC code. The gadget chain being exploited here, which is native to Dubbo 2.7.3’s ecosystem, is clearly broken down in Part 1. The rest of this POC takes this gadget, and reimplements the Dubbo protocol to wrap it with the appropriate preamble, flags, and headers to exploit a vulnerable deserializer on a receiving Dubbo endpoint.

Fig. 7 – Deserialization Gadget Writes “whoops!” to Java Console, and Starts Calc.exe

Vulnerability Evolution

Between the disclosure of this issue and the CVE, this issue has evolved across several releases. For versions 2.7.5 and 2.7.6 - the vulnerable deserializers have been removed from the core packages into their own respective packages. This functionality is imported separately, in the form of the dubbo-serialization-kryo/fst packages. In 2.7.7, according to documentation, it is no longer possible to pass garbage values in the Dubbo stream – deserialization does not occur without a proper method and service name.

Conclusions

This exploit, following the previous PoC for exploiting the HTTP protocol in version 2.7.3, further illustrates the need for strict whitelisting combined with class resolution look-ahead within any remote invocation services that deal with serialized objects - not just for Dubbo, and not just for Java. While this issue is glaring for Dubbo 2.7.3 and under, because of the gadget chain relying on an outdated FastJSON discovered and disclosed in the previous research, this only scratches the surface of this attack – other gadget chains may still exist within Dubbo’s internals, and most Dubbo instances will rely on other dependencies beyond those bundled with this software, allowing for a more diverse dependency ecosystem and allowing similar forms of exploitation to persist until a robust whitelist solution is used within Dubbo’s internals. At present, it is safe to assume many (and probably most) instances of Dubbo <= 2.7.3 are vulnerable to remote code execution via deserialization of untrusted data, while any Dubbo version under 2.7.8 has some form of unsafe deserialization going on in the Dubbo Protocol pipeline, 2.7.9 has added a configuration flag to prevent “chosen deserializer” attacks such as this. Whether it is vulnerable is dependent on gadget availability – which is always a likely possibility.

Recommendations

• Upgrade from Alibaba Dubbo to Apache Dubbo; Alibaba Dubbo does not appear to be officially maintained any longer
• Update Apache Dubbo to its latest version
• Ensure to never import unnecessary Dubbo packages, e.g. dubbo-serialization-kryo where Kryo is not actively used, to reduce attack surface against the RMI interface

Disclosure Timeline

2/5/20 – Disclosure of Dubbo RCE Deserialization Exploit 2/19/20 – Reminder #1 to Apache Security Team 4/6/20 – Reminder #2 to Apache Security Team 7/27/20 - Reminder #3 to Apache Security Team 1/22/21 – First Acknowledgement of Issue 2/26/21 – Fix Released 6/4/21 - Public Disclosure

Apache Package Versions

Deserialization RCE with FastJSON Gadget: Vulnerable package: org.apache.dubbo.dubbo <= 2.7.3 Requirements: org.apache.dubbo.dubbo-common <= 2.7.9 org.springframework.spring-web – 5.3.4 Deserialization vulnerability without FastJSON Gadget, requires additional deserialization gadget in scope: Vulnerable package: org.apache.dubbo.dubbo <=2.7.6 – implementation dependent Requirements: org.apache.dubbo.dubbo-common <= 2.7.9 dubbo-serialization-kryo AND/OR dubbo-serialization-fst – <=2.7.9

Alibaba Package Versions

Despite certain signs of life, this is no longer officially maintained – all Alibaba Dubbo resources now point to Apache Dubbo. Deserialization RCE with FastJSON Gadget: Vulnerable package: com.alibaba.dubbo <= 2.6.7 Requirements: Com.alibaba.dubbo-serialization-kryo OR Com.alibaba.dubbo-serialization-fst <= 2.6.7 org.springframework.spring-web – 5.3.4 Deserialization vulnerability without FastJSON Gadget, requires additional deserialization gadget in scope: Vulnerable package: com.alibaba.dubbo <= 2.6.9 Requirements: Com.alibaba.dubbo-serialization-kryo OR Com.alibaba.dubbo-serialization-fst <= 2.6.9 org.springframework.spring-web – 5.3.4

Final Words

Disclosures like this are part of the Checkmarx Security Research Team’s efforts to drive changes in software security practices. Checkmarx is committed to analyzing open source packages to help development teams build and deploy more secure software.