Microsoft has released a study describing an ongoing series of attacks on Kubeflow, a toolkit on top of Kubernetes clusters for running machine learning ( ML) operations. According to Microsoft, Kubernetes was at the forefront of a more recent large-scale XMRIG initiative. Misconfigured dashboards are at the center of a widespread campaign for the XMRIG Monero-mining. Closer inspection showed the image running a common open-source crypto-jacking malware that mines the virtual currency of Monero, known as XMRIG. The attacks have been ongoing since April 2020. The end goal of Microsoft is to deploy a cryptocurrency miner on K8s clusters running internet-exposed Kubeflow instances.
According to a security researcher, the business has found such kinds of attacks against ten of Kubernetes clusters running ML toolkit Kubeflow. Although the number of compromised clusters is small relative to previous attacks by Kubernetes (k8s). Server owners’ profits for crooks and financial losses are more likely than previously seen.
Nodes used to perform ML tasks are often fairly efficient, and often include GPUs. This makes Kubernetes clusters a great target for crypto-mining campaigns. They are also for ML tasks, which was the purpose of this attack.
KubeFlow – Cloud-Native Machine Learning toolkit for Kubernetes
It is an open-source project that integrates the most common resources of data science in one place. It’s an exciting toolkit with Kubernetes under the hood which can make life easier for Data Scientists and Data Engineers.
Kubeflow components
When you have a Kubeflow case going, though, you will get an integrated component. This shows that it will allow you to create machine learning models and bring them into production. Kubeflow is a project-based in the cloud. It can be implemented using various providers of cloud platforms: Azure, Google Cloud Platform (GCP), and Amazon Web Services ( AWS). It can also be installed on your local machine or on-prem infrastructure.
- Notebooks: Notebooks are an industry norm for the prototyping and sharing of machine learning models across teams. You can deploy a Jupyter Lab instance easily in Kubeflow, and start building your ML models.
- Model training: Kubeflow has built multiple model training options into it. From local Jupyter Notebook training, through managed services such as the Google Cloud AI Platform (formerly known as the Cloud Machine Learning Engine), to TensorFlow Training (TFJob) custom distributed training. To analyze your model you can use TensorBoard or TensorFlow Model Analysis (TFMA) with ease.
- Fairing: Kubeflow Fairing is a Python package allowing you to train the ML model in a cloud environment that is hybrid. You can choose whether to train the model locally (i.e. in the Jupyter Notebook) or cloud-based.
- Hyperparameter tuning: Kubeflow has incorporated Katib — a hyperparameter tuning system that runs on Kubernetes and implements Google Vizier.
- Pipelines: Kubeflow pipelines are a modern method for the development of end-to-end ML workflow. Every stage of a pipeline is a Docker container and a.yaml file (using Argo) describes the pipeline.
- Experiments: However, kubeflow offers a convenient way of comparing different pipeline speeds.
- Model serving: Options are available for multiple ML models: like TensorFlow Serving + Istio, TensorFlow Batch Prediction, Seldon Serving, NVIDIA TensorRT Inference Server, PyTorch Serving.
Popular attack vectors used to detect compromised Kubernetes clusters
Firstly, Kubernetes is a system for running and coordinating containerized applications across a machine cluster. It is a platform to use methods that provide predictability, scalability, and high availability to manage the life cycle of container type applications and services. The data gets encrypted when writing into etcd. Any newly created or updated secret should be encrypted when stored until your Kube-API server has been restarted. To search, you can use the command line software etcdctl to get back the contents of your file. KMS provider uses gRPC to interact with a plugin unique to KMS. This is responsible for all remote KMS contact.
Popular attack vectors used to detect compromised Kubernetes clusters are mentioned below:
ANONYMOUS ACCESS
The method of mitigation to Anonymous Access is to turn on RBAC (Role-Based Access Control) that requires specific anonymous request authorization. Anonymous access is the most serious kind of misconfiguration. It allows both external and internal attackers to directly hit commands to various k8s components via API and compromise k8s cluster.
We can detect misconfiguration of Anonymous by verifying the user object for the system: anonymous username and system: unauthenticated group in the logs.
Compromised Service Accounts
There are 2 types of accounts that exist in Kubernetes: regular users and service accounts. Therefore, implementing the least privilege principle for service accounts is important so that an attacker can’t use them to compromise the k8s cluster anymore.
While for service account misuse there is not any universal signature detection, there are many techniques like anomaly detection that can be to identify potential abuse. Service accounts are for a particular purpose by automating systems, their operation should be mention and consistent.
REMOTE COMMAND EXECUTION
Widely and commonly used exec command is “/bin/bash”, opening the interactive shell on the Pod. Using that command, an attacker can get access to the file system of the Pod and its containers, possibly install a backdoor or search for the account service token of the Pod, which could have more rights than its current account.
The execution of Pod commands is a login by API-Server and displays API calls that target resource pod with the exec subresource. The following logic is to identify a user opening, interactive shell at a Pod: objectRef.resource=”pods” AND objectRef.subresource=”exec” AND verb=”create” AND (request URI contains “%2Fbin%2Fbash” OR “%2Fbin%2Fsh”)
NODE COMPROMISE
However, underlying node hosting the Pods is a better alternative. There is an option to install a volume of the file system into the new Pod when building a Pod. Once an attacker gets access to create pod resources, they will be able to create a brand new Pod with a malicious container image configured to mount root directory of the node.
It would be highly suspicious for a Pod to build another Pod, as mostly Pod creations would come from the k8s controller. The call to the API may come from an account that normally does not have access to create Pods, such as a hacked service account, in that techniques of anomaly detection may also apply.
SECRET ACCESS
The secret may be installed as a number when an application or pod is required to access a secret object. Also, Secrets are not encrypted into repose by default in etcd storage.
Most of those accesses, as with the service account operation, should be routine and planned. It is possible to detect unauthorized secret access by tracking which secrets are usually accessed by account, which could be an intruder abusing the account to view secrets beyond reach.
Azure Security Center and Kubernetes clusters
These activities typically run in insecure containers, such as web applications, which exploit known vulnerabilities. Azure Security Center recently found a new campaign of crypto-mining that specifically targets Kubernetes environments. What separates this attack from other crypto-mining attacks is its scale: In just two hours, a malicious container will be deployed on tens of Kubernetes clusters. The containers were running a image in a public repository: kannix / monero-miner. Also the same actor who deployed the crypto mining containers identified the cluster resources including the secrets of Kubernetes. To be safe, it is necessary to carefully configure the Kubernetes environment to ensure that no container-focus surface doors are left open to attackers.cs, such as a privileged container detected.
Misconfigured workloads in Kubeflow pose a security risk to Kubernetes Clusters
Azure Security Center tracks and resists attacks of 1000+ Kubernetes clusters that run on top of the Azure Kubernetes Service. Although, they have published a blog post related to large-scale campaign against k8s clusters that abused exposed dashboards to deploy cryptocurrency miners. Kubeflow, a Kubernetes ML workflow toolkit, found that this attack occurred on many of Kubernetes clusters.
Kubeflow, a free and open-source machine learning platform, started with Kubernetes as a project to run TensorFlow work. This fact however makes clusters of Kubernetes used for Machine Learning tasks. Kubeflow Dashboards are the exact focus for crypto-mining campaigns. In many different k8s clusters a suspicious image from a public repo was observed during April.
Kubeflow is a container type service. Few of the services include training model frameworks, notebook server Jupyter and Katib, and more. the different tasks in the cluster are running as containers. So if attackers acquire access Kubeflow. They have many doors to run malicious container images in the k8s cluster.
Once attackers acquire access to dashboard, they can upload backdoor container images in the k8s cluster using multiple methods. For example:
- Kubeflow helps users to build a server on the Jupyter notebook. Kubeflow can allow users to pick a notebook server image, including the option to select a custom container image.
- Another tool that attackers can utilize is to install a malicious image from a real Jupyter notebook. Also, Attackers can use Python code to run a new or existing notebook.
Where to test if they affect your Kubernetes clusters?
- Verify that there is no malicious container image/pod into the cluster. The command below will help you test it is: kubectl get pods -A -o jsonpath=”{.items[].spec.containers[].image}”
- If Kubeflow is deployed in the cluster, ensure its dashboard is not exposed to the internet. Search the Istio incoming service form by the below command and make sure it does not have a load-balancer with a publicIP : kubectl get -n istio-system istio-ingress gateway service
Conclusion
Since the past, Azure Security Center has identified several campaigns against K8s clusters which have a common vector of access: an exposed internet connection. This is however the first time we have detected an attack directly targeting Kubeflow environments. When deploying a service such as Kubeflow in a cluster it is essential to be aware of security aspects such as:
- User Authentication and Access Control.
- Monitor the cluster endpoints which are open to the public. Moreover, it ensures that vulnerable devices are not exposing to the Internet in an insecure way.
- Track the Kubernetes runtime environment periodically. That includes monitoring of the containers running, images, and processes they run.
- Make it easy to deploy only trustworthy images and find vulnerabilities in your files. Although using Azure Policy you can restrict the images that are included in the cluster.
Moreover, the Kubernetes threat matrix includes techniques that attackers can use to target the Kubernetes cluster. The intruder used an exposed dashboard to obtain initial access to the cluster. The execution and durability of the cluster were achieved by establishing a container within the cluster that was installed inside the cluster.
Аwesome things here. I am very satisfieⅾ to ѕee your post.
Thanks a lot and I am lookіng ahead to contact you.
Wіll you kindly drop me a e-mail?