Accessing Logs in Kubernetes

Before getting into dashboards for LogInsight, this blog post will go through briefly how to access the different logs stored in the Kubernetes cluster without using tools like Fluentd and a log aggregation service (assuming kubeadm use for bootstrapping the cluster).  This is a great way to really get under the covers and see what’s happening within your Kubernetes cluster!

Accessing control plane component logs —

Using the kubectl command line, access to pod logs are available via the `kubectl logs <pod-name>` command.  This applies to any pod, including the cluster control plane components, which are running as static pods in the kube-system namespace.  *Note for the control plane components the kube-system namespace must be specified 

To access other control plane component logs, simply use their pod names. First, get their pod names by running kubectl get pods -n kube-system and then kubectl logs <pod name here> -n kube-system . Every deployment will have different suffixes for these static pods.

For example, here is accessing the etcd pod logs on my test cluster:

Accessing kubelet logs — 

The kubelet is responsible for interacting with the container engine (Docker in this case) and kubeapi-server, so a lot of good information is stored in these logs. If the nodes of the Kubernetes cluster are running with systemd, then kubelet logs are written to journald and can be accessed via journalctl.  Otherwise, they will be located in the /var/log/ directory, written to a .log file.

Kubeadm deployment using an Ubuntu 18.04 node:

journalctl --unit kubelet

Since kubelet is running as a service under systemd control, the logs are accessible via journalctl as show above. 

Accessing pod/application logs — 

To show that this works with applications as well, there is an Nginx pod running with a NodePort service exposing it.

To access the logs for this pod —

kubectl logs nginx

* Note didn’t need to specify namespace because pod was deployed in the default namespace. 

And we have logs!  The access logs from my browser are visible in the output.  

If you are in a situation where you may have multiple containers within a pod — the syntax to choose which containers’ logs to view is: 

kubectl logs <pod name> <container name>

That does it for this quick post on accessing cluster and application logs. In a previous post, I covered getting up and running with Fluentd running as a DaemonSet agent on every node and forwarding all of these logs to vRealize LogInsight (a log aggregator) for analysis and storage outside of the cluster. Next post will be on LogInsight dashboards and queries using the Interactive Analytics dashboard.

Sources of truth:

Custom Script Monitoring with vRealize Operations 8.0

One of the cool new features that VMware has introduced into vRealize Operations 7.5 is the ability to deploy agents to monitor the operating systems and applications inside your virtual machines. With vRealize Operations 8.0, we have added the extra feature to be able to run custom scripts using the Application Monitoring agent, and then collect the script output as a metric. This provides a lot of flexibility and robustness to our in-guest monitoring feature, since now you can monitor any information that can be pulled by running a script inside your operating system. 

In this blog, I will show off a simple bash script that checks for security patches in an Ubuntu VM, and then passes that metric to vRealize Operations, where we can create an alert to let us know if there are any patches available for our OS. This lets us centralize our Linux patch management into vRealize Operations, and lets us corollate our patching with other metrics collected by vRealize Operations to do things like patch when the system is the least busy, or when our app is least busy as reported by the application monitoring features in vRealize Operations.  

Weekly Update – Week of 2/3/2020

New and Noteworthy:
Following some changes to our team we took a bit of a hiatus from the weekly update component of the site, so apologies for the lack of news over that time. Since we last posted an update, there have been some major changes at VMware, most notably the completion of the acquisition of Pivotal. This acquisition is key to our mission of solving business problems through software, as Pivotal’s software stack combined with our Tanzu portfolio allow us to help developers accelerate deployment of new applications in the hybrid cloud. You can read more about the Pivotal acquisition here.

Additionally, we acquired a company called Nyansa, a producer of a highly-regarded AI-based network analytics platform. Nyansa will be tightly integrated with our VeloCloud offering, and will enable better end-to-end monitoring of the SD-WAN platform as well as the Virtual Cloud Network, enabling us to offer a truly self-healing network. Read more about VMware’s acquisition of Nyansa here.

Finally, our Workspace One offering was chosen as the leader by IDC in three separate End User Computing (EUC) vendor assessments: Unified Endpoint Management, Mobility Management, and IOT/Ruggedized Device Management. This likely isn’t a surprise to anyone who has used the software, as Workspace One is one of the most transformational platforms that I have ever been exposed to. It completely changed the way that I work (for the better) upon joining VMware, and really completes our mission of allowing users to securely access any application, running on any cloud, from any device, whenever they want it. Read more about our wins from IDC here.

Updated KB Articles:
New KB articles published for the week ending 2 February, 2020
New KB articles published for the week ending 26 January, 2020
New KB articles published for the week ending 19 January, 2020
New KB articles published for the week ending 12 January, 2020
New KB articles published for the week ending 5 January, 2020

Upcoming Events:
Dell Technologies Word 2020 – Las Vegas – 05/04/2020 to 05/07/2020 – Register

Upcoming Webinars:
Go Beyond Break/Fix with Skyline – 13 Feb, 2020 – Register
3 Strategies to Advance your Career with Training – 19 Feb, 2020 – Register
Implementing VM Storage Policies with vSAN – 20 Feb, 2020 – Register
Go Beyond Break/Fix with Skyline – 27 Feb, 2020 – Register
Multi-Cloud Load Balancing 101 – 18 March, 2020 – Register
[Full Live Event List]

New Releases:

2020-01-30
VMware Workspace ONE Access 20.01 [Release Notes] [Download]

2020-01-14
VMware vRealize Network Insight 5.1.0 [Release Notes] [Download]
VMware Tools 11.0.5 [Release Notes] [Download]
VMware App Volumes 4.0 [Release Notes] [Download]
VMware vCloud Director 10.0.0.1 for Service Providers [Release Notes] [Download]
VMware Cloud Foundation 3.9.1 Downloads [Release Notes] [Download]
VMware App Volumes 4 [Release Notes] [Download]

2019-12-20
VMware Horizon 7.10.1 Standard (ESB Release) [Release Notes] [Download]
VMware Horizon 7.10.1 Advanced (ESB Release) [Release Notes] [Download]
VMware Horizon 7.10.1 Enterprise (ESB Release) [Release Notes] [Download]
VMware Horizon 7.10.1 Enterprise Add-On [Release Notes] [Download]
VMware Horizon 7.5.4 Standard (ESB Release) [Release Notes] [Download]
VMware Horizon 7.5.4 Advanced (ESB Release) [Release Notes] [Download]
VMware Horizon 7.5.4 Enterprise (ESB Release) [Release Notes] [Download]
VMware Horizon 7.5.4 Enterprise Add-On [Release Notes] [Download]

2019-12-19
VMware vRealize Orchestrator Appliance 8.0.1 [Release Notes] [Download]
VMware vRealize Suite Lifecycle Manager 8.0.1 [Release Notes] [Download]
VMware vRealize Operations 8.0.1 [Release Notes] [Download]
VMware vRealize Automation 8.0.1 [Release Notes] [Download]
VMware vCenter Server 6.5U3f [Release Notes] [Download]
VMware NSX-T Data Center 2.5.1 [Release Notes] [Download]
VMware NSX Intelligence 1.0.1 [Release Notes] [Download]
VMware NSX Cloud 2.5.1 [Download]

2019-12-12
VMware Dynamic Environment Manager 9.10.0 [Release Notes] [Download]
VMware Horizon 7.11.0 Enterprise [Release Notes] [Download]
VMware Horizon Apps Advanced 7.11.0 [Release Notes] [Download]
VMware Horizon 7.11.0 Subscription [Release Notes] [Download]
VMware Horizon Apps 7.11.0 Subscription [Release Notes] [Download]
VMware Unified Access Gateway 3.8 [Release Notes] [Download]
VMware Horizon 7.11.0 Standard [Release Notes] [Download]
VMware Horizon 7.11.0 Advanced [Release Notes] [Download]
VMware Horizon 7.11.0 Enterprise Add-On [Release Notes] [Download]

2019-12-10
VMware Unified Access Gateway 3.7.2 [Release Notes] [Download]

2019-12-05
VMware vCenter Server 6.7U3b [Release Notes] [Download]
VMware vSphere Hypervisor (ESXi) 6.7U3b [Release Notes] [Download]