{"id":4997,"date":"2023-01-17T08:00:00","date_gmt":"2023-01-17T08:00:00","guid":{"rendered":"http:\/\/www.sumologic.com\/blog\/kubernetes-logs"},"modified":"2025-06-17T11:24:35","modified_gmt":"2025-06-17T19:24:35","slug":"kubernetes-logs","status":"publish","type":"blog","link":"https:\/\/www.sumologic.com\/blog\/kubernetes-logs","title":{"rendered":"Logging and monitoring Kubernetes"},"content":{"rendered":"\n
\n
\n
\n
\n
\"Kubernetes<\/figure>\n<\/div>\n\n\n

<\/p>\n\n\n\n

Kubernetes is first and foremost an orchestration engine that has well-defined interfaces that allow for a wide variety of plugins and integrations to make it the industry-leading platform in the battle to run the world’s workloads. From machine learning<\/a> to running the applications a restaurant needs<\/a>, you can see that just about everything now uses Kubernetes infrastructure.
<\/p>\n\n\n\n

All these workloads, and the Kubernetes operator itself, produce output that is most often in the form of logs. Kubernetes infrastructure has very difficult to view – and in some cases collect – internal logs and the logs generated by all the individual workloads it is running. Most often, difficulty viewing Kubernetes metrics and logs is due to ephemeral containers. <\/p>\n\n\n\n

This article will cover how Kubernetes monitoring<\/a> is structured with logs, how to use its native functionality, and how to use a third-party logging engine to really enhance what can be done with logs generated from cloud-native environments. Lastly, this article will cover how to visualize deployment metrics in a Kubernetes dashboard designed to tell you everything that happens within a Docker<\/a> container (with logs).<\/p>\n\n\n\n

Basic logging architecture and node-level logging in Kubernetes<\/strong><\/h2>\n\n\n\n

Kubectl<\/a> (pronounced “cube CTL”, “kube control”, “cube cuttle”, …) is a robust command line interface that runs commands against the Kubernetes cluster and controls the cluster manager. Since the command line interface (CLI) is essentially a wrapper around the Kubernetes API, you can do everything directly with the API<\/a> instead of using the CLI, if it suits your purposes.<\/p>\n\n\n\n

The most basic form of logging in Kubernetes is the output generated by individual containers using stdout and stderr. The output for the currently running container instance is available to be accessed via the kubectl logs command.<\/p>\n\n\n\n

The next level up of logging in the Kubernetes node world is called “node level logging”. This is broken down into two components: the actual log files being stored; and the Kubernetes side, which allows the logs to be viewed remotely and removed, under certain circumstances.<\/p>\n\n\n\n

The actual files are created by the container runtime engine – like the Docker container – and contain the output from stdout and stderr. There are files for every running container on the host, and these are what Kubernetes reads when kubectl logs are run. Kubernetes is configured to know where to find these log files and how to read them through the appropriate log driver, which is specific to the container runtime.
<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

Log rotating<\/h3>\n\n\n\n

Kubernetes has some log rotating capabilities, but it is limited to when a pod is evicted or restarted. When a pod is evicted, all logs are removed by kubelet. When a pod is restarted, kubelet keeps the current logs and the most recent version of the logs from before the restart. Any older logs are removed. This is great but does not help keep the logs from long-running pods under control.<\/p>\n\n\n\n

For any live environment with a constant stream of new log entries being generated, the reality of disk space not being infinite becomes very real the first time an application crashes due to no available space. To mitigate this, it is best practice to implement some kind of log rotation on each node that will take into account both the number of pods that potentially will be run on the node and the disk space that is available to support logging. Learn about best practices for log management<\/a> in our guide. <\/p>\n\n\n\n

While Kubernetes itself can not handle scheduled log rotation, there are many tools available that can. One of the more popular tools is log rotate, and like most other tools in the space, it can rotate based on time (like once a day), the size of the file, or a combination of both. Using size as one of the parameters makes it possible to do capacity planning to ensure there is adequate disk space to handle all the number of pods that could potentially run any given node.
<\/p>\n\n\n

\n
\"\"<\/figure>\n<\/div>\n\n\n

System components<\/h3>\n\n\n\n

There are two types of system components within Kubernetes: those that run as part of the OS, and those that run as containers managed by kubelet. As kubelet and the container runtime run as part of the operating system, their logs are consumed using the standard OS logging frameworks. As most modern Linux operating systems use systemd, all the logs are available via journalctl. In non-systemd Linux distributions, these processes create “.log” files in the \/var\/logs\/ directory.<\/p>\n\n\n\n

The second type of system components that run as containers \u2013 like the schedule, API-manager, and cloud-controller-manager \u2013 have their logs managed by the same mechanisms as any other container on any host in that Kubernetes cluster.
<\/p>\n\n\n\n

Cluster-level logging architecture in Kubernetes<\/h2>\n\n\n\n

A Kubernetes monitoring challenge with cluster-level logging is that Kubernetes has no native cluster-level logging. The good news is that there are a few proven methods that can be applied cluster-wide to provide the same effective result of all the logs being collected in a standardized way and sent to a central location.<\/p>\n\n\n\n

The most widely-used methods are:<\/p>\n\n\n\n