CVE-2022-31358 _ Multiple Vulns in Proxmox VE & Mail Gateway

Proxmox Virtual Environment (Proxmox VE or PVE) is an open-source type-1 hypervisor. It includes a web-based management interface programmed in Perl. Another Proxmox product written in Perl, Proxmox Mail Gateway (PMG), comes with a similar web management interface. They share some of the codebases.

In this article, I will introduce how to debug PVE’s web service step-by-step and analyse three bugs I have found in PVE and PMG.

PVE is a Debian-based Linux distribution. The ISO installer is available at their website. Do note that if you would like to reproduce any of the bugs in this article, please use “Proxmox VE 7.2 ISO Installer” updated on 04 May 2022, which does not include the patches unless you run apt update manually.

In a default installation, the web service should listen on port 8006.

With a few commands, it is not difficult to figure out that the scripts of the web service are located in /usr/share/perl5/:

I choose IntelliJ IDEA and its Perl plugin for debugging. Here are the steps to set it up:

At this stage, you should have correct syntax highlighting and dependency resolving in IDEA.

Run these commands to install the required debug tools:

All set. To start a debug session, click Run > Debug 'PVE remote' in IDEA and run PERL5_DEBUG_HOST=<your PVE server IP> PERL5_DEBUG_PORT=12345 PERL5_DEBUG_ROLE=server perl -T -d:Camelcadedb /usr/bin/pveproxy start -debug on the server. If everything goes well, the debugger should break at line 330 of SSL.pm by default, as shown in the image below.

By logging in to the web interface, it can be observed that a lot of requests are sent to endpoints under the path /api2/json/. Usually, json after /api indicates the format the response data is in, and the server might support various formats for different purposes. For example, xml might be implemented for RPC calls, jsonp for cross-origin <script> tags, or html for setting innerHTML. In PVE, if we change json to html, the server will return an “API Inspector” page containing the json result:

Further testing shows that the server does not properly escape user’s input. If we visit a non-existent API endpoint, the request path will be reflected in the href attribute of an <a> tag. As such, an attacker can inject HTML tags to achieve reflected cross-site scripting.

The function handle_request at perl5/PVE/APIServer/AnyEvent.pm line 1100 is our entry point. If the request path starts with /api2, it will pass the request on to function handle_api2_request.

Stepping into handle_api2_request, we can see at line 865 the variables $rel_uri and $format are extracted from the rest of the request path by a regex. Then function PVE::APIServer::Formatter::get_formatter is called to get a “formatter” for generating the response.

Later on, the $formatter is called at line 946. When generating the “breadcrumb” HTML of the navigation bar, the request path is directly concatenated to the href attribute of the <a> tag.

Since the authentication cookie PVEAuthCookie is set with the Session attribute, successful exploitation requires the victim to be logged in to the web interface in the same browser session before he visits the malicious link.

An attacker can access every functionality in the web interface by executing malicious JavaScript code. One of the features is to execute shell commands. Here is a video demonstrating a possible attack scenario. In the video, the victim logged in to PVE web UI, and then visited a link. A reverse shell of the PVE host was spawned on the attacker’s machine.

This vulnerability is patched by encoding user inputs to HTML entities in pve-http-server version 4.1-2.

While handling HTTP requests, if there is any error, the PVE server will write the error message in the status line of the response.

The corresponding code is located in perl5/PVE/APIServer/AnyEvent.pm:

At [1] the server uses a regex to match the first line of the error message, trying to avoid additional lines breaking the HTTP response at [2]. However, this method only prevents LF(%0a). It’s still possible to inject response headers with CR(%0d) in Chromium-based browsers.

This is what the response looks like in Burp Suite:

At the time of testing, using CR(%0d) to inject response headers only works on Chromium-based browsers (Chrome, MS Edge, Opera, etc.), and it is not possible to inject into the response body using only CR(%0d). Firefox does not recognise CR(%0d) as a valid newline indicator without LF(%0a).

This bug in PVE might seem completely harmless at first sight. Unfortunately, at AnyEvent.pm line 1327, there is a length limit check for incoming HTTP requests. If a request header exceeds 8192 bytes, the server will reject to process the HTTP request.

As such, an attacker could craft a malicious webpage to set long cookies on the victim’s PVE domain multiple times. Once the victim visits the malicious webpage, subsequent HTTP requests to the PVE domain will carry a very long cookie header and thus be rejected by the server.

Here is a video to demonstrate this client-side DoS vulnerability. In the video, the victim was able to use PVE web UI at first. After visiting a malicious link, the victim can no longer access the web UI until he clears the cookies.

One thing to note is that Chrome allows third-party cookies by default. This is a necessary condition to exploit this client-side DoS bug since we are setting cookies from the attacker’s domain to the victim’s PVE domain. However, if the victim has changed their cookie policy to “Block third-party cookies” or “Block all cookies (not recommended)” in browser settings, this attack will not work.

This bug is patched by adding an additional check of \r\n in pve-http-server version 4.1-3.

A PVE server can run as a standalone node or join a cluster to connect with other nodes. This design naturally allows nodes to exchange information with each other. For instance, the api /api2/json/nodes/{node_name}/status is meant for querying the status of a node in the cluster by its name. It can also be used to query on the node itself.

If we change the node_name to a nonexistent value “test”, we will see this error message: HTTP/1.1 500 hostname lookup 'test' failed - failed to get address info for: test: No address associated with hostname. It seems that the server is trying to perform a DNS lookup on the given node_name. A quick test using Burp Collaborator verifies our guess:

By step debugging, we are able to locate the corresponding code in AnyEvent.pm:proxy_request. It turns out that the server resolves node_name to IP address and then relays our HTTP request to https://{IP}:8006/api2/json/nodes/{node_name}/status.

One thing we might want to try here is to setup our own HTTPS server to listen on port 8006 with a valid SSL certificate and observe whether the relayed request could come in. While it does not work like that because there are multiple checks performed before firing the request and one of them is expecting /etc/pve/nodes/{node_name}/pve-ssl.pem to be found for every node in the cluster. Whether we input our own domain name or IP address, the server will never find the cert file since the node_name does not point to any real node in the cluster. So it just throws the error “HTTP/1.1 596 tls_process_server_certificate: certificate verify failed” during TLS handshake and stops there.

Another thing we notice is that $uri is appended to the port (line 699, 703 and 705 in the image above) when constructing the $target URL. The developers might have assumed that $uri will always start with a slash(/). While that is not true as we find out that it is possible to replace slash(/) with its URL-encoded form %2F without breaking the request parser.

We tried to turn the starting part of the URL into userinfo and append our own domain by using the at sign (@), but one of the sanity checks blocked us again. After several attempts, we managed to find a suitable API to exploit this SSRF vulnerability: GET /api2/json/nodes/{node_name}/tasks/{upid}/log. This API accepts any string as upid, which means we can set node_name to a valid node so that it won’t fail for certificate issues. Then we use URL-encoded slashes and @ to control the hostname.

An authenticated user without any permissions in PVE is able to perform this SSRF attack. Due to the large shared codebases between PVE and PMG, an authenticated user in PMG that only has a low privilege “Help Desk” role or “Audit” role can also exploit this SSRF vulnerability using API /api2/html/nodes/{node_name}/pbs/{remote}/snapshot/.

Inside the callback function of http_request, the server looks for the pvestreamfile header in response headers (line 778) and extracts its value to the variable $stream. $stream is later passed to sysopen, and the server will return the content of the file as the response body.

The vulnerable code exists in PMG as well. An attacker can exploit the SSRF vulnerability presented earlier to read arbitrary files on PVE/PMG servers with only a non-privileged account in PVE or a low-privilege account in PMG. The sysopen is called in process “pve(pmg)proxy worker” with uid=33(www-data).

With the ability to read an arbitrary file, hackers might be particularly interested in credentials and secret keys stored on the server. We decided to dig into the implementation of the authentication process to see whether the server stores anything in the database or in the config file, in plaintext or encrypted by some “secret keys”.

Authentication in PVE/PMG is implemented by signing and verifying a string using RSA/SHA-1. Upon successful login, the server will sign a “ticket” for client, as known as “PVEAuthCookie” or “PMGAuthCookie”. Here is a sample of the ticket:

The double colon seperates the plaintext and signature. The format of plaintext is PVE:{username}@{realm}:{hex(timestamp)} . While the signature is generated using private key stored at /etc/pve/priv/authkey.key for PVE, or /etc/pmg/pmg-authkey.key for PMG, only root user has read-write permissions to these files.

However, it turns out that if the backup feature in PMG has ever been used, the backup file will contain the authkey. More importantly, it is readable by www-data users:

The path to the backup file can be extracted from task logs which is also accessible by www-data user. Combining all the vulnerabilities above, an attacker can forge a ticket to achieve privilege escalation from a low privilege “Help Desk” role or “Audit” role to "root@pam" for full access in PMG.

We have attached the python script below and a video demonstrating this exploit. In the video, the attacker logged in to PMG web UI as a “Help Desk” user and was not able to change the current user’s role due to low privilege. After running the exploit, a forged ticket was generated, and the attacker gained access to the web UI as "root@pam" user.

There are several commits applied in pve-http-server version 4.1-3 to fix the bug chain.