Before getting into dashboards for LogInsight, this blog post will go through briefly how to access the different logs stored in the Kubernetes cluster without using tools like Fluentd and a log aggregation service (assuming kubeadm use for bootstrapping the cluster).  This is a great way to really get under the covers and see what’s happening within your Kubernetes cluster!

Accessing control plane component logs —

Using the kubectl command line, access to pod logs are available via the `kubectl logs <pod-name>` command.  This applies to any pod, including the cluster control plane components, which are running as static pods in the kube-system namespace.  *Note for the control plane components the kube-system namespace must be specified 

To access other control plane component logs, simply use their pod names. First, get their pod names by running kubectl get pods -n kube-system and then kubectl logs <pod name here> -n kube-system . Every deployment will have different suffixes for these static pods.

For example, here is accessing the etcd pod logs on my test cluster:

Accessing kubelet logs — 

The kubelet is responsible for interacting with the container engine (Docker in this case) and kubeapi-server, so a lot of good information is stored in these logs. If the nodes of the Kubernetes cluster are running with systemd, then kubelet logs are written to journald and can be accessed via journalctl.  Otherwise, they will be located in the /var/log/ directory, written to a .log file.

Kubeadm deployment using an Ubuntu 18.04 node:

journalctl --unit kubelet

Since kubelet is running as a service under systemd control, the logs are accessible via journalctl as show above. 

Accessing pod/application logs — 

To show that this works with applications as well, there is an Nginx pod running with a NodePort service exposing it.

To access the logs for this pod —

kubectl logs nginx

* Note didn’t need to specify namespace because pod was deployed in the default namespace. 

And we have logs!  The access logs from my browser are visible in the output.  

If you are in a situation where you may have multiple containers within a pod — the syntax to choose which containers’ logs to view is: 

kubectl logs <pod name> <container name>

That does it for this quick post on accessing cluster and application logs. In a previous post, I covered getting up and running with Fluentd running as a DaemonSet agent on every node and forwarding all of these logs to vRealize LogInsight (a log aggregator) for analysis and storage outside of the cluster. Next post will be on LogInsight dashboards and queries using the Interactive Analytics dashboard.

Sources of truth: