{"id":4781,"date":"2023-05-11T07:00:00","date_gmt":"2023-05-11T07:00:00","guid":{"rendered":"http:\/\/www.sumologic.com\/blog\/threat-labs-kubernetes-home-lab"},"modified":"2025-05-08T19:10:55","modified_gmt":"2025-05-09T03:10:55","slug":"threat-labs-kubernetes-home-lab","status":"publish","type":"blog","link":"https:\/\/www.sumologic.com\/blog\/threat-labs-kubernetes-home-lab","title":{"rendered":"Building a Kubernetes purple teaming lab"},"content":{"rendered":"\n
\n
\n
\n
\n
\"Sumo<\/figure>\n<\/div>\n\n\n

Kubernetes, and containerization in general, has a wealth of benefits for many teams operating cloud-native applications. From a threat detection standpoint, however, it is often difficult for newcomers to this space to gain the relevant hands-on experience without trampling over production environments. <\/p>\n\n\n\n

The Sumo Logic team has previously authored articles on Kubernetes DevSecOps vulnerabilities and best practices<\/a> as well as Kubernetes logging and monitoring<\/a>. Now, let\u2019s extend this work and outline how to set up a Kubernetes home lab. <\/p>\n\n\n\n

We will provision and configure this lab, send the relevant telemetry to a free Sumo Logic instance and track our testing activities using the awesome Vectr tool<\/a>. <\/p>\n\n\n\n

What is purple teaming? <\/strong><\/h2>\n\n\n\n

Before diving head first into the tooling, attacks & defenses, we should pause for a moment and outline what purple teaming is and why it’s a powerful tool for developing defenses for complex applications and networks. <\/p>\n\n\n\n

Within the information security community, colors have been ascribed to the attack\/defense spectrum. With those on the attacking end (penetration testers, red teamers) being associated with the red color and those on the defending end (SOC analysts, threat hunters) being associated with the blue color.<\/p>\n\n\n\n

This dichotomy between red and blue is often blurry, as some red teamers may want to perform threat hunting to hone their evasion skills and some SOC analysts may want to understand how an attack tool works in order to craft more comprehensive detection logic. <\/p>\n\n\n\n

It should come as no surprise then, that purple teaming is a mix of both blue and red aspects of the cyber security spectrum. Purple teaming engagements and exercises can come in many flavors and may lean more to one side of the red-blue spectrum than the other, depending on how the exercise is laid out and who the recipient of the engagement is. <\/p>\n\n\n\n

Overall, however, purple teaming generally encompasses the dynamic of collaboratively attacking a system, checking the results of the attack, tracking the related metrics and then iterating; all with the goal of improving system security, response and resiliency. <\/p>\n\n\n\n

Some additional resources on purple teaming can be found here<\/a> and here<\/a>. <\/p>\n\n\n\n

Tooling overview<\/strong><\/h2>\n\n\n\n

Before getting into the various techniques, tactics and procedures (TTPs) as well as technical details, let us step back for a moment and do a quick overview of all the tooling involved and highlight the function that each piece performs. <\/p>\n\n\n\n\n\n\n
\n\n\n\n

Tool<\/strong><\/p>\n\n\n\n<\/td>\n

\n\n\n\n

Purpose<\/strong><\/p>\n\n\n\n\n\n

Ubuntu 20.04 virtual machine<\/a><\/p>\n\n\n\n<\/td>\n

\n\n\n\n

This virtual machine will be running our Kubernetes cluster as well as various other logging tools in addition to Vectr itself.<\/p>\n\n\n\n\n\n

Docker<\/a><\/p>\n\n\n\n<\/td>\n

\n\n\n\n

Docker<\/a> will act as our Kubernetes driver and will host our Vectr instance, all on the same virtual machine.<\/p>\n\n\n\n\n\n

Minikube<\/a><\/p>\n\n\n\n<\/td>\n

\n\n\n\n

Minikube will make it possible for us to run a Kubernetes instance on our virtual machine.<\/p>\n\n\n\n\n\n

Sumo Logic<\/a><\/p>\n\n\n\n<\/td>\n

\n\n\n\n

A free Sumo Logic account will be used in order to monitor our Kubernetes installation.<\/p>\n\n\n\n\n\n

Auditd<\/a> + Laurel<\/a><\/p>\n\n\n\n<\/td>\n

\n\n\n\n

We will be using an auditd configuration file on our virtual machine in order to get host-level telemetry from our Kubernetes cluster. Laurel will be used to transform these auditd logs into JSON format so that they are easier to work with and query.<\/p>\n\n\n\n\n\n

Vectr<\/a><\/p>\n\n\n\n<\/td>\n

\n\n\n\n

Vectr will be used to track our purple teaming activities on our local Kubernetes cluster. <\/p>\n\n\n\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n\n\n\n

Virtual machine setup <\/strong><\/h2>\n\n\n\n

Now that you have all the tools to follow along, let\u2019s dive in and get it all set up. <\/p>\n\n\n\n

Please note that throughout these instructions, a virtual machine with an ARM architecture is used, if the virtual machine you are running is not ARM, then please adjust the relevant installation instructions to match your particular architecture. At time of writing, Vectr does not support ARM architectures. If you wish to run all the tooling outlined in this blog on one virtual machine, then we recommend a x64\/amd64 architecture. <\/p>\n\n\n\n

Docker install<\/strong><\/h3>\n\n\n\n

In order to install Docker, we will be following the official instructions<\/a>. <\/p>\n\n\n\n

We first install the necessary dependencies: <\/p>\n\n\n\n

sudo apt-get update\r\nsudo apt-get install \r\n    ca-certificates \r\n    curl \r\n    gnupg<\/pre>\n\n\n\n
We then add the Docker repo GPG keys: <\/p>\n\n\n\n
sudo install -m 0755 -d \/etc\/apt\/keyrings\r\ncurl -fsSL https:\/\/download.docker.com\/linux\/ubuntu\/gpg | sudo gpg --dearmor -o \/etc\/apt\/keyrings\/docker.gpg<\/em>\r\nsudo chmod a+r \/etc\/apt\/keyrings\/docker.gpg<\/pre>\n\n\n\n
Following this, we set up the relevant Docker repos – note that the below command should be architecture agnostic. <\/p>\n\n\n\n
echo \r\n  \"deb [arch=\"$(dpkg --<\/strong>print<\/strong>-architecture)\" signed-by=\/etc\/apt\/keyrings\/docker.gpg] <a href=\"https:\/\/download.docker.com\/linux\/ubuntu\" class=\"redactor-autoparser-object\">https:\/\/download.docker.com\/li...<\/a> \r\n  \"$(. \/etc\/os-release && echo \"$VERSION_CODENAME\")\" stable\" | \r\n  sudo tee \/etc\/apt\/sources.list.d\/docker.list > \/dev\/null<\/strong> \r\napt-get update<\/strong><\/pre>\n\n\n\n
Finally, we install the relevant Docker packages: <\/p>\n\n\n\n
sudo apt-get install<\/strong> docker-ce docker-ce-cli containerd.io docker-buildx-plugin<\/strong> docker-compose-plugin<\/strong><\/pre>\n\n\n\n
If all went well during the installation, you should be able to run: docker -v and see something similar to the following: <\/p>\n\n\n
\n
\"Kubernetes<\/figure>\n<\/div>\n\n\n
Minikube install<\/strong><\/h3>\n\n\n\n
Next up, we\u2019ll be installing Minikube. We will follow the official instructions<\/a> to perform the installation.<\/p>\n\n\n\n
The instructions are interactive here, and you can click the relevant buttons to match your particular architectures and virtual machine setups. <\/p>\n\n\n
\n
\"Installation\"<\/figure>\n<\/div>\n\n\n
In our case, we will be using the Linux operating system, ARM64 architecture, and will be using the binary installer type, so the command here is:<\/p>\n\n\n\n
curl -LO <a href=\"https:\/\/storage.googleapis.com\/minikube\/releases\/latest\/minikube-linux-arm64\" class=\"redactor-autoparser-object\" target=\"_blank\">https:\/\/storage.googleapis.com\/minikube\/releases\/latest\/minikube-linux-arm64<\/a>\r\nsudo install minikube-linux-arm64 \/usr\/local\/bin\/minikube<\/pre>\n\n\n\n
Prior to starting Minikube, we need to add our current user to the Docker group:<\/p>\n\n\n\n
sudo usermod -aG docker $USER<\/strong> && newgrp<\/strong> docker<\/pre>\n\n\n\n
Now we are ready to start Minikube using: minikube start<\/p>\n\n\n\n
If all goes well, you should see something similar to:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
In order to interact with our Minikube cluster, we also need to install kubectl. <\/p>\n\n\n\n
Minikube can do this for you with the following command: <\/p>\n\n\n\n
minikube kubectl -- get po -A<\/em><\/pre>\n\n\n\n
We can then alias this minikube version of kubectl to just kubectl to make it easier for us: <\/p>\n\n\n\n
alias<\/strong> kubectl<\/strong>=\"minikube kubectl --\"<\/pre>\n\n\n\n
Now we should be able to do a: kubectl get nodes<\/span> in order to see the minikube control plane running.<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Very cool! We now have a virtual machine provisioned with Docker installed as well as a local Kubernetes instance ready for our testing. <\/p>\n\n\n\n
Auditd setup<\/strong><\/h3>\n\n\n\n
As a next step, let\u2019s instrument our Ubuntu host with some host-level telemetry using Auditd and Laurel. <\/p>\n\n\n\n
We first install auditd using the following command: sudo apt-get install auditd<\/span><\/p>\n\n\n\n
If we do a service auditd status<\/span>, we should see output similar to the following: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Next, we need to tell auditd what we want it to log and here we will be using Florian Roth\u2019s<\/a> awesome auditd configuration file<\/a>. <\/p>\n\n\n\n
We can start by making a backup of the existing rules file: <\/p>\n\n\n\n
cp \/etc\/audit\/rules.d \/etc\/audit\/rules.d.bak<\/pre>\n\n\n\n
Then, we can go ahead and change the auditd rules. To do so, use sudo to open up \/etc\/audit\/rules.d\/audit.rules<\/span> with your favorite text editor (no nano versus vim arguments here!) and replace the contents of the audit<\/strong>.rules<\/span> file on your Ubuntu machine with the audit<\/strong>.rules<\/span> linked above. <\/p>\n\n\n\n
Once that is done, restart auditd using sudo service auditd restart<\/p>\n\n\n\n
To ensure that auditd is working properly, you can use: tail -f \/var\/log<\/strong>\/audit\/audit.log<\/strong><\/span> – we can hit ctrl+c to stop the tailing.<\/p>\n\n\n\n
If you take a closer look at the auditd log format, it may look overwhelming at first, as a simple cat command looks like this when logged by auditd: <\/p>\n\n\n\n
type=SYSCALL msg=audit(1681912914.195:1053): arch=c00000b7 syscall=56 success=yes exit=3 a0=ffffffffffffff9c a1=ffffef61a793 a2=0 a3=0 items=1 ppid=3102 pid=4453 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=2 comm=\"cat\" exe=\"\/usr\/bin\/cat\" subj=unconfined key=\"auditlog\"ARCH=aarch64 SYSCALL=openat AUID=\"parallels\" UID=\"root\" GID=\"root\" EUID=\"root\" SUID=\"root\" FSUID=\"root\" EGID=\"root\" SGID=\"root\" FSGID=\"root\"<\/pre>\n\n\n\n
Laurel setup <\/strong><\/h3>\n\n\n\n
In order to make these logs easier to work with, we will be using Laurel<\/a>.<\/p>\n\n\n\n
We start by downloading the relevant Laurel binary: <\/p>\n\n\n\n
wget <a href=\"https:\/\/github.com\/threathunters-io\/laurel\/releases\/download\/v0.5.1\/laurel-v0.5.1-aarch64-musl.tar.gz\" class=\"redactor-autoparser-object\" target=\"_blank\">https:\/\/github.com\/threathunters-io\/laurel\/releases\/download\/v0.5.1\/laurel-v0.5.1-aarch64-musl.tar.gz<\/a><\/pre>\n\n\n\n
We then untar the archive: <\/p>\n\n\n\n
tar<\/strong> xzf<\/strong> laurel-v0.5.1-aarch64-musl.tar.gz<\/strong><\/pre>\n\n\n\n
And copy it to the proper directory: <\/p>\n\n\n\n
sudo install -m755 laurel \/usr\/local\/sbin\/laurel<\/pre>\n\n\n\n
Next up we need to create a user for Laurel: <\/p>\n\n\n\n
sudo useradd --system --home-dir \/var\/log\/laurel --create-home _laurel<\/pre>\n\n\n\n
Next, we need to create a Laurel configuration file, an example is provided here<\/a><\/p>\n\n\n\n
We can do this with the following commands: <\/p>\n\n\n\n
sudo mkdir \/etc\/laurel <\/pre>\n\n\n\n
Followed by:<\/p>\n\n\n\n
sudo wget <a href=\"https:\/\/raw.githubusercontent.com\/threathunters-io\/laurel\/v0.5.1\/etc\/laurel\/config.toml\" class=\"redactor-autoparser-object\" target=\"_blank\">https:\/\/raw.githubusercontent.com\/threathunters-io\/laurel\/v0.5.1\/etc\/laurel\/config.toml<\/a> -O \/etc\/laurel\/config.toml<\/pre>\n\n\n\n
Next, open the config.toml file that we just downloaded and change the value on line 20 to match your Ubuntu user.<\/p>\n\n\n\n
Finally, we need to tell auditd to use Laurel as a plugin, we can do this with the following command: <\/p>\n\n\n\n
sudo wget <a href=\"https:\/\/raw.githubusercontent.com\/threathunters-io\/laurel\/v0.5.1\/etc\/audit\/plugins.d\/laurel.conf\" class=\"redactor-autoparser-object\" target=\"_blank\">https:\/\/raw.githubusercontent.com\/threathunters-io\/laurel\/v0.5.1\/etc\/audit\/plugins.d\/laurel.conf<\/a> -O \/etc\/audit\/plugins.d\/laurel.conf<\/pre>\n\n\n\n
We now need to restart auditd:<\/p>\n\n\n\n
sudo pkill -HUP auditd<\/pre>\n\n\n\n
Before we take a look at our fresh Laurel logs, let\u2019s install the jq utility: sudo apt install<\/strong> jq<\/span><\/p>\n\n\n\n
Now we can browse to \/var\/log\/laurel<\/span> and run: cat audit.log | jq<\/span><\/p>\n\n\n\n
Wow, we ran a lot of commands so far! <\/p>\n\n\n\n
To recap, we\u2019ve set up an Ubuntu virtual machine with Docker and Minikube in order to operationalize a local Kubernetes cluster and to have a Docker base for the installation of Vectr. We\u2019ve also instrumented this Linux host with telemetry in the form of an auditd configuration file and have transformed that telemetry into JSON format using Laurel. <\/p>\n\n\n\n
Next up, let\u2019s install Vectr on this same host so that we can track and monitor our purple teaming journey.<\/p>\n\n\n\n
Vectr setup<\/strong><\/h3>\n\n\n\n
Please note that at time of writing, only x64 platforms are supported for Vectr installations.<\/strong><\/p>\n\n\n\n
Vectr is a fully dockerized application, so we can grab the latest release from the Vectr GitHub repository<\/a>. <\/p>\n\n\n\n
The full instructions can also be found here<\/a><\/p>\n\n\n\n
Once the Vectr release has been downloaded and unzipped to the appropriate directory, you should have a structure that looks like this: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Next up, open up the .env file with your favorite text editor and change the VECTR_HOSTNAME variable to match your virtual machine’s IP address. While here, you can also go ahead and change the MongoDB passwords and JWS\/JWE keys as well.<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Once we save the .env file, we can go ahead and bring up the various Vectr containers by doing a: sudo docker-compose up -d<\/span> within the \/opt<\/strong>\/vectr<\/span> directory. <\/p>\n\n\n\n
docker-compose should have been installed in earlier steps as part of our dependencies, but if it is not, you can run: sudo apt install<\/strong> docker-compose<\/span><\/p>\n\n\n\n
You should now see various containers being pulled and spun up: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Once all the containers are built, you should be able to browse to https:\/\/virtualmachineIPAddress:8081<\/a> and put in the default credentials which can be found within the Vectr install instructions, from here, we can create a database for our testing activities. <\/p>\n\n\n\n
Navigate to the \u201cEnvironment\u201d button in Vectr, and click on \u201cSelect Active Environment\u201d then the + button and create an environment for our testing:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
We should now see a fresh Vectr screen waiting for us: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Recap<\/strong><\/h3>\n\n\n\n
Prior to proceeding, we thought it would be wise to recap what we\u2019ve built thus far and to provide a high level overview of all the moving pieces: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Sumo Logic setup and collection<\/strong><\/h2>\n\n\n\n
Now that we have our Kubernetes set up, and have a host that is generating telemetry, we need to go ahead and send that telemetry to Sumo Logic.<\/p>\n\n\n\n
We can spin up a free trial via this link<\/a> – once you sign up you should receive an email asking you to activate your account.<\/p>\n\n\n\n
Once your account is activated you can click through the relevant wizards and you should be greeted with a blank Sumo Logic page: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Kubernetes monitoring<\/strong><\/h3>\n\n\n\n
In order to set up monitoring of our Kubernetes cluster, navigate to the \u201cApp Catalog\u201d section in the bottom left hand side the menu pane and then click on Kubernetes:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Once in the Kubernetes App Catalog menu, go ahead and click on \u201cAdd Integration\u201d – once you click on this, you should see step by step instructions: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
If helm is not installed on your virtual machine, you can install it with the following commands: <\/p>\n\n\n\n
Helm will allow us to deploy the necessary components for logging and monitoring via a \u201cchart\u201d – for more information about Helm, check out their documentation<\/a>.<\/p>\n\n\n\n
curl -fsSL -o get_helm.sh <a href=\"https:\/\/raw.githubusercontent.com\/helm\/helm\/main\/scripts\/get-helm-3\" class=\"redactor-autoparser-object\" target=\"_blank\">https:\/\/raw.githubusercontent....<\/a>\r\nchmod 700 get_helm.sh\r\n.\/get_helm.sh<\/pre>\n\n\n\n
Once helm is installed, we can follow the instructions on the Sumo Logic App Catalog page – note that the minikube cluster needs to be started prior to performing these steps: minikube start<\/p>\n\n\n\n
Before running the helm upgrade command, we will set the sumologic.setup.monitors.enabled field to true.<\/p>\n\n\n\n
By default, the Helm chart installs three replicas for the various collection pods, however, since our cluster is small and non-production we will be changing this value to just one.<\/p>\n\n\n\n
To do this, we need to copy the contents of this file: <\/p>\n\n\n\n
https:\/\/raw.githubusercontent.com\/SumoLogic\/sumologic-kubernetes-collection\/main\/deploy\/helm\/sumologic\/values.yaml<\/a><\/p>\n\n\n\n
To our Ubuntu host and find the lines with \u201creplicaCount\u201d in them and change the values from 3 to 1<\/p>\n\n\n\n
Then, we will need to add one line to the command that the Kubernetes onboarding wizard gives you:<\/p>\n\n\n\n
helm upgrade --install my-release sumologic\/sumologic \r\n  --namespace=default \r\n  --create-namespace \r\n  --set sumologic.accessId=suuxOoW071wSiB \r\n  --set sumologic.accessKey=<snip> \r\n  --set sumologic.clusterName=<cluster name> \r\n  --set sumologic.collectorName=<collector name> \r\n  --set sumologic.setup.monitors.enabled=false \r\n  -f path\/to\/user_values.yaml<\/pre>\n\n\n\n
Once the wizard completes, you should see something similar to the following: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Note that if the helm repo or wizard fails, you can try to add some more CPU and RAM to your Minikube setup:<\/p>\n\n\n\n
minikube stop\r\nminikube config set memory 4192\r\nminikube config set cpus 4\r\nminikube start \u2014nodes 2<\/pre>\n\n\n\n
After a few minutes the process should complete and the wizard should finish successfully. <\/p>\n\n\n\n
Right now, there is not much happening with our cluster so there isn\u2019t much data to look at. <\/p>\n\n\n\n
Let\u2019s change that and spin up a basic Ubuntu pod, using the following YAML:<\/p>\n\n\n\n
apiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  name: ubuntu\r\n  labels:\r\n    app: ubuntu\r\nspec:\r\n  containers:\r\n  - image: ubuntu\r\n    command:\r\n      - \"sleep\"\r\n      - \"604800\"\r\n    imagePullPolicy: IfNotPresent\r\n    name: ubuntu\r\n  restartPolicy: Always<\/pre>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
If you are wondering what the \u201cbatcat\u201d command is, it is utilizing the awesome \u201cbat\u201d utility which can be found here<\/a><\/p>\n\n\n\n
Within your Sumo Logic screen, you can now click on New \u2192 Log Search: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Once on the log search screen, we can take a look at the logs for our pod creation, we don\u2019t know what we\u2019re looking for just yet, so we\u2019ll just type in \u201cubuntu\u201d in the search bar to get us an idea of what the data looks like: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Nice! Even if you aren\u2019t a Kubernetes or threat hunting expert, this very simple query already gives you a good idea of the types of data that are available to you. <\/p>\n\n\n\n
We can see the container being pulled, created and started all in our Kubernetes telemetry<\/a>.<\/p>\n\n\n\n
Let\u2019s complete our setup with one final step, installing a Sumo Logic collector to grab our Laurel logs off this host. <\/p>\n\n\n\n
Host monitoring<\/strong><\/h3>\n\n\n\n
Let\u2019s start by navigating to the collection menu in Sumo Logic, and clicking on \u201cAdd Collector\u201d

<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Next, click on \u201cInstalled Collector\u201d and choose the collector that matches your system architecture <\/p>\n\n\n\n
On our testing ARM VM, we could use the following command: <\/p>\n\n\n\n
wget <a href=\"https:\/\/collectors\" class=\"redactor-autoparser-object\" target=\"_blank\">https:\/\/collectors<\/a>.ca<\/strong>.sumologic.com<\/strong>\/rest\/download\/linux\/aarch\/64 -O SumoCollector_linux_arm64_19_418-7.sh<\/strong><\/pre>\n\n\n\n
Before running the script, let\u2019s generate a token to be used for the installation. Back within the Sumo Logic UI, navigate to the \u201cToken\u201d section and create one.<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Once you have your token, we can now install the collector using the following command:<\/p>\n\n\n\n
 SumoCollector_linux_arm64_19_418-7.sh -q<\/strong> -Vsumo.token_and_url=<your_token><\/pre>\n\n\n\n
Note that the name of the file may change depending on your architecture.<\/p>\n\n\n\n
Back in the Sumo UI – you should now see the minikube collector below the Kubernetes collectors we set up in earlier steps:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Now we need to tell our collector to ingest the Laurel logs. We can do so by clicking the \u201cAdd\u2026\u201d button next to the collector, then \u201cAdd Source\u201d and then \u201cLocal File\u201d<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
After a few minutes, we should see some Laurel logs trickling in:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
At this point, we have a local Kubernetes cluster setup on a virtual machine. We are sending telemetry from this Kubernetes cluster to a Sumo Logic instance and are also monitoring the host that the cluster is running on via auditd and are using Laurel to transform these auditd logs into JSON format. <\/p>\n\n\n\n
Now we are ready for the fun stuff, testing out some attacks on our Kubernetes cluster! <\/p>\n\n\n\n
The TTPs: attacks and detections for Kubernetes<\/strong><\/h2>\n\n\n\n
T1610 \u2013 deploy container<\/strong><\/h3>\n\n\n\n
To dip our toes into the Kubernetes threat detection world, let us build on our basic example of starting an Ubuntu pod, and look at the MITRE \u201cDeploy Container<\/a>\u201d category.<\/p>\n\n\n\n
If your Ubuntu container is still deployed, go ahead and delete with kubectl delete<\/strong> -f ubuntu.yaml<\/span><\/p>\n\n\n\n
We\u2019ll then go ahead and recreate the pod: kubectl apply -f ubuntu.yaml<\/span><\/p>\n\n\n\n
Now let\u2019s look at the following query in Sumo Logic: <\/p>\n\n\n\n
_collector=\"kubernetes-2023-04-20T12:54:16.324Z\"\r\n| %\"object.reason\" as<\/strong> reason\r\n| %\"object.involvedobject.kind\" as<\/strong> object_kind\r\n| %\"object.involvedobject.name\" as<\/strong> object_name\r\n| where reason = \"Created\"\r\n| values(object_kind) as<\/strong> kinds,values(object_name) as<\/strong> names<\/pre>\n\n\n\n
This will show us what kinds of objects are created within our cluster along with their names. You should see something similar to: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
We see our Ubuntu pod hanging out at the bottom, let\u2019s clean the query up a little bit and filter out the system pods as well as the pods necessary for the Sumo Logic collection to take place:<\/p>\n\n\n\n
_collector=\"kubernetes-2023-04-20T12:54:16.324Z\"\r\n| %\"object.reason\" as<\/strong> reason\r\n| %\"object.involvedobject.kind\" as<\/strong> object_kind\r\n| %\"object.involvedobject.name\" as<\/strong> object_name\r\n| where<\/strong> reason = \"Created\"\r\n| where<\/strong> !(object_name matches \/(coredns|etcd|my<\/strong>-release|kube-|storage-provisioner)\/)\r\n| values(object_kind) as<\/strong> kinds,values(object_name) as<\/strong> names<\/pre>\n\n\n\n
Through some regular expression tweaking on line 6 of our query, we exclude pods that we may not want to see in this particular detection logic, and now we should be left with only our Ubuntu pod showing up in the search results. We highly recommend a regular expression testing site<\/a> of some kind as a resource to aid you in crafting any type of regular expressions. <\/p>\n\n\n\n
That\u2019s pretty cool, but didn\u2019t we spend a bunch of time setting up host logging as well, and can that be used to provide us some additional coverage? Great question; yes it absolutely can!<\/p>\n\n\n\n
We may not be super familiar with auditd and Laurel, but we know that when we created our Ubuntu pod, we used a yaml file called \u201cubuntu.yaml\u201d so let\u2019s start there with a super quick search: <\/p>\n\n\n\n
_collector=\"minikube\" ubuntu.yaml<\/pre>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
After rolling up our proverbial sleeves and getting a handle on the data, we can craft the following query which is annotated: <\/p>\n\n\n\n
_collector=\"minikube\" \r\n| %\"syscall.comm\" as binary_name \/\/renaming fields for ease of use<\/strong>\r\n| %\"proctitle.argv\" as<\/strong> command_line \/\/renaming fields<\/strong> for<\/strong> ease of<\/strong> use<\/strong>\r\n| where<\/strong> binary_name = \"kubectl\" \/\/looking for<\/strong> the kubectl binary used \r\n| where<\/strong> command_line matches \/apply<\/strong>\/ \/\/matching on<\/strong> the \"apply\" verb\r\n| where<\/strong> !(command_line matches \/\/etc\/kubernetes\/) \/\/excluding<\/strong> some<\/strong> system<\/strong> events<\/strong>\r\n| values<\/strong>(command_line)<\/pre>\n\n\n\n
And looking at the results, we see our kubectl<\/a> apply command: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
This is a great example of why host logs are important for cloud native technologies such as Kubernetes – of course this assumes that your cluster is not hosted within a cloud service of some kind. <\/p>\n\n\n\n
In our instance, however, we were able to gain visibility into a container being deployed in the environment from both the host and Kubernetes level – sweet! <\/p>\n\n\n\n
Let\u2019s not forget to track these executions in Vectr. <\/p>\n\n\n\n
We can navigate to the environment we set up earlier and click on the \u201cCreate New ” button on the right hand side within the Vectr UI and create a new Assessment:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Once within the assessment, we can click on \u201cAssessment Actions\u201d and click on \u201cCreate New Campaign\u201d – so the overall structure here is an Environment, which contains an assessment, which contains a campaign, with the campaign containing our test cases. <\/p>\n\n\n\n
From the campaign menu, click on \u201cCampaign Actions \u2192 New Test Case\u201d:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
We can then enter the details of our test case: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
It should be noted that Vectr provides a ton of options for tracking, including tracking time to alert and various sources – here we are just scratching the surface to get some basic metrics of our executions. <\/p>\n\n\n\n
Once we click on \u201cSave\u201d we should see our first test case in Vectr. <\/p>\n\n\n\n
Recall, we did two executions\/variations of the same technique, so we can clone this test case to track our host-based detection as well. <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Once you click the clone button, you will see a window pop up with customizable parameters: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
We can then rename this second test to something like \u201cDeploy Container – Host Based\u201d<\/p>\n\n\n\n
T1609 \u2013 Container administration command<\/strong><\/h3>\n\n\n\n
Let\u2019s look at another example, continuing to use our deployed Ubuntu pod and perform a kubectl exec command in order to get a bash shell into our pod. <\/p>\n\n\n\n
Before running the command, we need to add the following entry into our audit.rules auditd configuration file so that we log the appropriate telemetry: -w \/usr\/local\/bin\/minikube -p x -k minikube<\/span> and then restart auditd: <\/p>\n\n\n\n
sudo pkill -HUP auditd<\/pre>\n\n\n\n
Now we can run our command: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
We take a look at the Kubernetes logs and fail to find any telemetry at this level, so lets pivot to the host level with the following query: <\/p>\n\n\n\n
_collector=\"minikube\" \r\n| %\"syscall.comm\" as<\/strong> binary_name \/\/renaming fields for<\/strong> ease of use\r\n| %\"proctitle.argv\" as<\/strong> command_line \/\/renaming fields for<\/strong> ease of use\r\n| where binary_name = \"minikube\" \/\/looking for<\/strong> minikube here as<\/strong> we aliased kubectl earlier<\/strong> - normally this would just be<\/strong> kubectl\r\n| values(command_line) as<\/strong> command_line,values(binary_name) as<\/strong> binary_name<\/pre>\n\n\n\n
And we get our results: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Now that we know what command line value to look for, let\u2019s tighten up our query a little bit, using some regular expressions. <\/p>\n\n\n\n
_collector=\"minikube\" \r\n| %\"syscall.comm\" as<\/strong> binary_name \/\/renaming fields for<\/strong> ease of use\r\n| %\"proctitle.argv\" as<\/strong> command_line \/\/renaming fields for<\/strong> ease of use\r\n| where binary_name = \"minikube\" \/\/looking for<\/strong> minikube here as<\/strong> we aliased kubectl earlier<\/strong> - normally this would just be<\/strong> kubectl\r\n| where command_line matches \/exec|tty|stdin\/ \/\/looking for<\/strong> a<\/strong> command<\/strong> line that conains exec, tty or stdin \r\n| values(command_line) as<\/strong> command_line,values(binary_name) as<\/strong> binary_name<\/pre>\n\n\n\n
Now we can go ahead and add this execution to our Vectr for tracking purposes. <\/p>\n\n\n\n
T1613 – Container and resource discovery<\/strong><\/h3>\n\n\n\n
Discovery and enumeration type techniques are often difficult to detect as administrators and developers may run these types of commands as part of their normal workflows. Let\u2019s run some basic Kubernetes enumeration commands on our local Minikube cluster:<\/p>\n\n\n\n
kubectl config get-users\r\nkubectl config get-clusters\r\nkubectl auth can-i --list\r\nkubectl get roles\r\nkubectl get secrets\r\nkubectl get serviceaccounts\r\nkubectl get deployments\r\nkubectl get pods -A<\/pre>\n\n\n\n
Now let\u2019s modify a query we used earlier in order to get a sense of how the telemetry looks:<\/p>\n\n\n\n
_collector=\"minikube\" \r\n| %\"syscall.comm\" as<\/strong> binary_name \/\/renaming fields for<\/strong> ease of use\r\n| %\"proctitle.argv\" as<\/strong> command_line \/\/renaming fields for<\/strong> ease of use\r\n| where binary_name = \"minikube\" \/\/looking for<\/strong> minikube here as<\/strong> we aliased kubectl earlier<\/strong> - normally this would just be<\/strong> kubectl\r\n| values(command_line) as<\/strong> command_line,values(binary_name) as<\/strong> binary_name<\/pre>\n\n\n\n
And we get our results: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
We see the command line values with \u201cminikube\u201d followed by \u201ckubectl\u201d as Minikube uses its own version of kubectl – recall that we created an alias that mapped minikube kubectl to just kubectl in earlier steps. In production environments, this command line would show up as just \u201ckubectl\u201d <\/p>\n\n\n\n
In order to find this activity, we can do some string matching on things like \u201cauth can-i\u201d or \u201ckubectl get\u201d – however, we probably do not want to find normal or day-to-day administrative activity. <\/p>\n\n\n\n
Another approach we can take is to slice up our data by time slices and score each of these commands, summing up the score based on the time slice. Our hypothesis here is that threat actors might perform a bunch of enumeration in a short time period, whereas a developer or administrator may not exhibit such behavior. <\/p>\n\n\n\n
Let\u2019s take a look at what this looks like in query format:<\/p>\n\n\n\n
_collector=\"minikube\" \r\n\r\n\/\/ Initialize variables\r\n| 0 as score\r\n| \"\" as messageQualifiers\r\n| \"\" as messageQualifiers1\r\n| \"\" as messageQualifiers2\r\n\r\n\/\/ Setting our time slice\r\n| timeslice 1h\r\n\r\n\/\/ Renaming some fields for ease of use<\/strong>\r\n| %\"syscall.comm\" as<\/strong> binary_name \r\n| %\"proctitle.argv\" as<\/strong> command_line\r\n\r\n\/\/ Only<\/strong> looking at<\/strong> the aliased minikube binary \r\n| where<\/strong> binary_name = \"minikube\" \/\/looking for<\/strong> minikube here as<\/strong> we aliased kubectl earlier - normally this would just be kubectl\r\n\r\n\/\/ Setting our qualifiers, we look for<\/strong> can-i, get<\/strong> or<\/strong> config, we can add<\/strong> more qualiifers here depending on<\/strong> the environment\r\n| if<\/strong>(command_line matches \/(can-i)\/,concat<\/strong>(messageQualifiers, \"Kubectl auth enumeration: \",command_line,\"nBy Binary: \" ,binary_name,\"n# score: 3n\"),\"\") as<\/strong> messageQualifiers\r\n| if<\/strong>(command_line matches \/(get<\/strong>)\/,concat<\/strong>(messageQualifiers1, \"Kubectl cluster enumeration: \",command_line,\"nBy Binary: \" ,binary_name,\"n# score: 3n\"),\"\") as<\/strong> messageQualifiers1\r\n| if<\/strong>(command_line matches \/(config)\/,concat<\/strong>(messageQualifiers2, \"Kubectl config enumeration: \",command_line,\"nBy Binary: \" ,binary_name,\"n# score: 3n\"),\"\") as<\/strong> messageQualifiers2\r\n\r\n\/\/ Putting our qualifiers together \r\n| concat<\/strong>(messageQualifiers,messageQualifiers1,messageQualifiers2) as<\/strong> q \/\/Concact all the qualifiers together\r\n\r\n\/\/ Extracting the score from<\/strong> the qualifiers \r\n| parse<\/strong> regex field<\/strong>=q \"score:s(?<score>-?d+)\" multi \r\n\r\n\/\/Only<\/strong> return<\/strong> results if<\/strong> there is<\/strong> a qualifier of<\/strong> some<\/strong> kind\r\n| where<\/strong> !isEmpty(q) \r\n\r\n\/\/Return<\/strong> our full<\/strong> qualifiers and<\/strong> sum<\/strong> the score by<\/strong> timeslice\r\n| values<\/strong>(q) as<\/strong> qualifiers,sum<\/strong>(score) as<\/strong> score by<\/strong> _timeslice <\/pre>\n\n\n\n
Looking  at the results, we can see our numerous enumeration commands bubbled up with a score of 33, which is much higher than our \u201cnormal\u201d administrative activity which occurred on the previous day with a score of 12.<\/p>\n\n\n\n
Please keep in mind that all the parameters and scoring within these queries can be tweaked depending on your particular set ups and architecture. <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Let\u2019s not forget to add this execution to our Vectr tracking.<\/p>\n\n\n\n
T1496 – Resource hijacking<\/strong><\/h3>\n\n\n\n
Once threat actors compromise a Kubernetes cluster, often a deployment of some kind of crypto currency miner follows. <\/p>\n\n\n\n
This kind of technique is difficult to replicate in a home lab environment, as we probably do not want to be deploying coin miners on our virtual machines. <\/p>\n\n\n\n
However, we can exhaust the resources of our Kubernetes cluster in other ways \u2013 before we dive in it needs to be noted that this technique is not recommended to execute unless you are comfortable with maxing out resources on whatever compute platform your Minikube cluster is running on. <\/p>\n\n\n\n
It goes without saying that this is not recommended for production or even test environments. <\/strong><\/p>\n\n\n\n
Although we all want to avoid stress, we can use the \u201cstress<\/a>\u201d Linux utility in order to stress test the CPU on a Kubernetes pod – we can also create a deployment that goes ahead and creates many replicas of these pods, all running the stress utility in order to generate a high CPU load on our Kubernetes cluster. <\/p>\n\n\n\n
Here is what the YAML looks like:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
If you choose to deploy this on your cluster, you will need to give it some time for all the replicas to spin up. <\/p>\n\n\n\n
After waiting for a few minutes, we can navigate back to Sumo Logic and take a look at some dashboards that were provisioned for us when we set up the Kubernetes monitoring solution: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Once you navigate to the \u201cKubernetes – Cluster\u201d page, you should see the CPU usage chart well into the red: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
As a final step, we can click the \u201cbell\u201d icon on the top of the Sumo Logic menu to see some alerts waiting for us:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
Clicking into the \u201cKubernetes – Node CPU Utilization High\u201d alert will bring us to the following screen where we can see some additional information: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
We can go ahead and add this test case to our Vectr campaign. <\/p>\n\n\n\n
Vectr wrap-up<\/strong><\/h2>\n\n\n\n
At this point, we should have five test cases in our Vectr instance, with the escalation path looking something like this: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
If we navigate to \u201cReporting \u2192 MITRE ATT&CK Coverage\u201d within Vectr: <\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
We should be greeted with a nice MITRE ATT&CK matrix showing us our test cases:<\/p>\n\n\n
\n
\"\"<\/figure>\n<\/div>\n\n\n
From here, we can click the red \u201c&\u201d symbol on the top right and export this layer into a JSON file which can then be loaded into the MITRE ATT&CK Navigator<\/a><\/p>\n\n\n\n
Now you know how to set up a local Kubernetes cluster with host and Kubernetes cluster level visibility, with all the telemetry being fed into a free Sumo Logic instance. We have also used Vectr to track and report on our test cases.<\/p>\n\n\n\n
This setup used freely available tooling all hosted on a local virtual machine, this type of setup may be preferable for many folks who do not want to be on the hook for potentially large cloud bills. <\/p>\n\n\n\n
This type of environment also provides users with the ability to snapshot and recover a virtual machine in order to try out various configurations and threat detection use cases. <\/p>\n\n\n\n
Learn more about how Sumo Logic can help you with your Kubernetes monitoring<\/a>.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div><\/section>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":32,"featured_media":0,"template":"","meta":{"_acf_changed":false,"show_custom_date":false,"custom_date":"","featured":false,"featured_image":0,"learn_more_label":"","image_alt_text":"","learn_more_type":"","show_popup":false,"learn_more_link_file":0,"event_date":false,"event_start_date":"","event_end_date":"","place_holder_image_url":"","post_reading_time":"17","notification_enabled":false,"notification_text":"","notification_logo":"","notification_expiration_time":0,"is_enable_transparent_header":false,"selected_taxonomy_terms":{"blog-category":[127],"blog-tag":[]},"selected_primary_terms":[],"learn_more_link":[],"featured_page_list":[],"notification_enabled_post_list":[],"_gspb_post_css":"","_relevanssi_hide_post":"","_relevanssi_hide_content":"","_relevanssi_pin_for_all":"","_relevanssi_pin_keywords":"","_relevanssi_unpin_keywords":"","_relevanssi_related_keywords":"","_relevanssi_related_include_ids":"","_relevanssi_related_exclude_ids":"","_relevanssi_related_no_append":"","_relevanssi_related_not_related":"","_relevanssi_related_posts":"71501,4668,71369","_relevanssi_noindex_reason":"","inline_featured_image":false,"footnotes":""},"blog-category":[127],"blog-tag":[],"class_list":["post-4781","blog","type-blog","status-publish","hentry","blog-category-secops-security"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog\/4781","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog"}],"about":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/types\/blog"}],"author":[{"embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/users\/32"}],"version-history":[{"count":3,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog\/4781\/revisions"}],"predecessor-version":[{"id":26813,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog\/4781\/revisions\/26813"}],"wp:attachment":[{"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/media?parent=4781"}],"wp:term":[{"taxonomy":"blog-category","embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog-category?post=4781"},{"taxonomy":"blog-tag","embeddable":true,"href":"https:\/\/www.sumologic.com\/wp-json\/wp\/v2\/blog-tag?post=4781"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}