Introduction
Containers and microservices have become the modern trend in software development. Managing such infrastructure is a tremendous challenge in production environments. Kubernetes (or k8s), apart from being a Greek word to describe “someone in command“, is an extremely popular container orchestration platform. The modular design makes it great at flexibility and scalability, as almost all components can be individually tuned to meet a specific goal. However, this complexity can be really challenging from a security perspective.
The RedHunt Labs research team, therefore, decided to perform internet-wide research to explore, study and analyse unsecured Kubernetes clusters across the web. This blog post describes our attempt to assess the security posture of Kubernetes clusters scattered across the internet, detailing our research methodology, findings and analysis.
Mapping the Attack Surface
Working Architecture
First things first, let us get a quick understanding of the working architecture of Kubernetes.
Kubernetes Core Architecture
A Kubernetes cluster consists of a set of worker machines, called nodes that run containerized applications. A pod is the smallest unit in Kubernetes, hosting one or more containers. A control plane/master node manages the worker nodes and the pods in the cluster.
The control plane includes components such as:
- Kube API Server: A frontend for Kubernetes that takes RESTful requests from clients and translates the requests to operations on containers, pods or nodes.
- etcd: A key-value database used as a backing store for all cluster data.
- Kube Scheduler: A watcher for newly created pods.
- Kube Controller Manager: Runs controller processes.
A few components on each node maintain the running pods and provide a runtime environment to it:
- Kubelet: Makes sure that the containers are running properly.
- Kube Proxy: Maintains network rules on nodes.
- Container runtime: Software that runs on containers (e.g. Docker).
Default Ports
Kubernetes clusters usually listen on a range of well-defined and distinctive ports. Here is an overview of the default ports used in Kubernetes which might pose a risk from a security standpoint:
Port(s) | Component |
6443/TCP | Kubernetes API Server |
2379/TCP | etcd server client API |
10250/TCP | Kubelet API |
Additionally, we discovered the following ports to be of importance too:
Port(s) | Component |
8443/TCP | minikube API server |
8080/TCP | Insecure kube-api port ( --insecure-port) |
Our Methodology
The initial idea was to cover the basic components of Kubernetes instances. However, the attack surface gradually widened during the course of our research. We also created a scanner in Go called KubeStalk, keeping in mind both speed and accuracy to cover the entire IPv4 space while respecting our exclusion lists. We chose to keep the scanning process as non-intrusive as possible so that there is no disruption due to our study.
An architectural overview of the tool is depicted in the flowchart below:
Flowchart depicting an architectural overview of the scanning process
The scanning timeline spanned over a period of 59 hours distributed across many nodes spread across the internet.
Our Findings
We have broken down our findings into two major parts. The first part covers the core components of Kubernetes, and the second one covers the additional attack surface as discussed above.
Core Components
Over a time span of nearly 3 days, our scanners uncovered 574,913 misconfigured internet-facing Kubernetes core components. Among them, 338,320 were Kube API servers and 232,103 Kubelet APIs. etcd exposure contributed towards the remaining 4,490 instances.
Geographically, the most exposure occurred in the United States, followed by Germany, Belgium, Ireland and Singapore. [Interactive Map]
World map depicting the exposure distribution
The United States alone accounted for almost 57% of the total exposures, followed by Germany, Belgium, Ireland, Singapore and Japan. Below is a graph depicting the country-wise exposure distribution.
Country-wise distribution graph
Cloud Providers at the Top:
We noticed around 61% of the exposures were hosted in public clouds, with Google Cloud Platform (GCP) leading the most exposure counts, followed by Amazon Web Services (AWS), Tencent Cloud, Microsoft Azure and DigitalOcean Cloud. The following chart presents the exposure levels from the top 5 cloud providers.
Exposure statistics from public cloud service providers
Most affected versions – v1.21 and v1.22
While analysing the results, we found that versions v1.21 and v1.22 are the most seen versions among the unsecured Kubernetes clusters, contributing to more than half of the deployments. This also indicates that 60% of the Kubernetes clusters have not been updated for more than a year. v1.21 was released in April 2021, while v1.22 was released in August 2021. At the time of writing, the current latest version is v1.25 which contributed to less than 1% of the existing clusters, totaling just 1,211 instances.
Top versions from the Kubernetes instances
Deriving further analytics and statistics, we mapped out the CVEs from the official CVE feed to the versions on the exposed clusters. The following table depicts a statistic of the most critical vulnerabilities (high CVSS base scores, i.e. scores >= 8) affecting the unsecured instances. Please note that the vulnerabilities have actually not been verified on those instances and are just statistics based on the version numbers.
Kubernetes instances affected with high-severity vulnerabilities
The above statistics shed light on the fact that proper patching is still not being done, and a huge number of deployments still remain at risk to a large number of known vulnerabilities. This observation also highlights the fact that Attack Surface Reduction still has a long way to go.
Additional Attack Surface – Kubernetes Management Components
What else was exposed related to Kubernetes
Apart from the regular default components, we discovered developers using additional software to manage their Kubernetes infrastructure, which also poses a significant security risk if misconfigured. The following section focuses on those risky software components we discovered during the course of our research. The infographic below depicts the overall exposure.
Project Resonance - Wave 9
We now take a look at each component on a case-by-case basis.
cAdvisor
cAdvisor is a container metric engine that provides developers with an overview of their resource usage and the performance of their containers. The software exposes a Web UI for viewing the resource metrics. It includes system processes, CPU, memory, network and filesystem usage statistics.
cAdvisor Web UI dashboard
We found a total of 23,101 unauthenticated cAdvisor dashboards exposed.
Kubernetes Operational View
kube-ops-view is a system dashboard that provides an overview of multiple deployed K8s clusters. Hovering over a pod provides additional information about it, including containers, deployment zones, etc.
Kubernetes Operational View dashboard
Our tool KubeStalk found 929 instances of such misconfigured portals exposed to the internet.
Kubernetes Resource Report
kube-resource-report is yet another dashboard for viewing and downloading a cluster and pod’s resource requests vs usage reports. A misconfigured portal provides very detailed information about clusters, nodes, pods, applications, namespaces, etc, running on that instance.
Kubernetes Resource Report dashboard
A total of 31 instances of such dashboards were uncovered during our scanning process.
Kubernetes Web View
kube-web-view is frontend software that is designed to give developers an extremely detailed overview of the Kubernetes instance (including CRDs). The information exposure includes a detailed overview of all deployments, jobs, DaemonSets, pods, services, etc. A misconfigured deployment of this software can be disastrous, considering the amount of information being exposed via the dashboard.
Kubernetes Web View dashboard
We discovered a total of 11 IPs that exposed this product unauthenticated to the internet.
etcd Viewer
etcd-viewer is a portal that allows users to navigate and modify etcd distributed key-value stores. A publicly accessible dashboard allows unauthenticated users to directly view and edit the values inside the etcd storage database. Misconfiguration of this software can easily lead to the destruction of a cluster as a whole.
etcd Viewer dashboard
Our scanners uncovered 8 IPs exposing this product directly facing the internet, unauthenticated and publicly accessible by anyone.
Uncovering Compromised Clusters
During the process of verification of results, we noticed that several of the exposed clusters had already been compromised.
Many of the clusters were running intentionally vulnerable images, which were abused for container breakouts and getting access to the host. We also uncovered adversaries deploying malicious cryptominers inside the compromised clusters. Many compromised instances had traces and IoCs of the already known Hildegard malware on those compromised clusters.
The script below attributed to TeamTNT (a notorious group previously known for the Kubernetes compromise) was found on one of the instances of the compromised clusters. It has a detection rate of zero on VirusTotal.
Bash script used to discover and infect unsecured Kubernetes clusters
The script initially utilises masscan to discover unsecured Kubernetes clusters across IP ranges by scanning for open ports. It then makes use of zgrab to probe for HTTP services serving the path /v1.16/version. Once an unsecured cluster has been discovered, the script uses the Docker CLI to deploy an intentionally privileged container.
Docker containers are usually limited in terms of resources, but the --privileged flag gives all capabilities to a container, lifting all constraints imposed by cgroups. This basically means that the container has all capabilities that the host does, and has access to all system resources that a normal container wouldn’t have.
The execution chain follows downloading other components, scripts and cryptominers for fully utilising the resources of the system. However, we cannot estimate the number of compromised instances, since this was out of the scope of this research.
An Attacker’s Perspective
In this section, we will take a look at how easily an attacker can compromise an unsecured Kubernetes cluster. It is possible to gain code execution (RCE) in a running container through unsecured API servers or Kubelets. Both the exec API and run APIs allow running arbitrary commands in containers.
Once an unsecured cluster that allows anonymous access to its APIs has been discovered, launching a malicious process in a hijacked container is more stealthy as it doesn’t need to pull new images or run new containers.
For our case, let’s say that Kubelet’s APIs running on port 10250 allows anonymous access. One of the first steps would be to discover the pods running inside the node. A list of pods inside the container can be retrieved by visiting the /pods endpoint.
curl -sk 'https://<ip>:10250/pods' | jq -r '.items[] | "Pod: \(.metadata.name)\tNamespace: \(.metadata.namespace)\tContainer: \(.spec.containers[].name)"'
Initial pods, namespaces and containers discovery
The output from the /pods endpoint provides us with a list of pods and their attributes, including their namespaces and the container to which they belong. This information is enough to achieve code execution through either the run or the exec API.
curl -sk -X POST 'https://<ip>:10250/run/<namespace>/<pod>/<container>' -d "cmd=id"
This command, in our case, would take the following form if we’re to target the pod nginx-6c54bd5869-pfxmw in the default namespace inside the Nginx container:
curl -sk -X POST 'https://<ip>:10250/run/default/nginx-6c54bd5869-pfxmw/nginx' -d "cmd=id"
Code execution within a container
Voila! We have achieved code execution in just two steps. From here on, the attacker can access the servicetoken runtime secrets, deploy an intentionally vulnerable container and escalate privileges by breaking out of it, further, moving laterally between pods and containers and gaining persistence. This blog post details a more detailed attack path after initial access. Tools like Peirates aid post-exploitation and lateral movement through clusters after initial access.
Open Source Tool Release
We strongly believe in giving back to the security community. Therefore we are releasing a community version of the tool used along with this wave. The tool takes a list of hosts as input and generates an output file in CSV format. You can get the tool and read more about it from our GitHub (https://github.com/redhuntlabs/kubestalk). A GIF has been added below to demonstrate the capabilities and working of the tool.
KubeStalk tool demo
Prevention and Mitigation
There has been a significant effort at documenting how to secure Kubernetes clusters on the internet. The Kubernetes docs mention very clearly the aspects that need to be looked at considering security. The CISA and NSA have released a solid hardening guide for the same. The OWASP Cheatseries Project also has a definitive guide to Kubernetes Security.
In short, here’s how to keep your clusters safe:
- Authorization Audits: Enforce RBAC and regularly audit cloud roles (IAM).
- Network Control Audits: Review the network policies and service meshes. Enable mTLS.
- Contain Workloads: Make use of pod security policies and OPA/Gatekeeper.
- Apply Regular Hardening: Follow the CIS benchmark.
- Cluster Upgrades: Upgrade as soon as a stable release is out (watch out for EOL).
- Image Safety: Regularly scan images for known vulnerabilities.
Open-source tools like kubescape, kube-bench and kube-hunter make the process of auditing Kubernetes instances easy.
Impact
The impact of having an unsecured Kubernetes cluster can be devastating. Production environments, if compromised, can cause immense business disruption leading to financial losses and customer distrust.
Conclusion
The research throws a flood of light into the security posture of unsecured Kubernetes clusters on the internet. It is clear that the concept of security for complex systems like Kubernetes is still translucent. APT groups have already stepped ahead, compromising such unsecured instances and using them for their own benefit. Developers need to be made aware of security-first development when it comes to building applications. As security researchers, we continue to strive to raise awareness amongst organisations of such potential threats through Project Resonance‘s R&D program.