Leveraging LogInsight for Kubernetes

As part of responsibly running applications, it’s important to have all the supporting Day 2 operations covered. That way, when something goes bump in the night, you’re immediately prepared and able to quickly find the source of the issue. Logging is one critical component of this overall architecture. Many shops are already running mature processes for logging with vRealize LogInsight in supporting their vSphere infrastructures. Wouldn’t it be great to use this existing logging setup for your Kubernetes clusters? You can!

Note: If you’d like help setting up a simple, single node test cluster see this blog.

Setting It Up

Fluentd is an open source project that provides a “unified logging layer.” It is a great project that provides a lot of capabilities, outside of Kubernetes as well. For our purposes, it will be deployed as a DaemonSet within our Kubernetes cluster to provide log collection and shipping to our vRealize LogInsight Appliance.

Luckily for us, the project maintains a set of templates that make it very easy to deploy fluentd as a DaemonSet within a Kubernetes cluster. Remember, a DaemonSet (DS), is a Kubernetes capability that ensures we always have pod of this type running on every node within our cluster. Perfect for the logging use case.

Github for templates: https://github.com/fluent/fluentd-kubernetes-daemonset

For our implementation with LogInsight, we will be using the Kubernetes syslog template.

If you click on that file, you will see the manifest file that shows the configuration that will be deployed into the Kubernetes cluster.

You can see that it will:

  • Create a ServiceAccount and ClusterRole for fluentd
  • Deploy as a DaemonSet
  • Deploy into the kube-system namespace
  • Pull the container image from Fluent’s repository

Within the manifest file, the parameters that we need to change are only the IP address and desired port for our LogInsight Appliance.

Once you change the value: to the LogInsight IP address you can simply use that yaml file to deploy fluentd to the cluster! This will automatically create the DS and start shipping logs to your LogInsight Appliance.

Step by step for the deployment (assumes your have your cluster up and running and kubeconfig set up):

1. git clone https://github.com/fluent/fluentd-kubernetes-daemonset.git

2. Use a text editor to change the syslog template file to have correct value for your LogInsight Appliance

sudo vim fluentd-kubernetes-daemonset/fluentd-daemonset-syslog.yaml

Edit the value field under SYSLOG_HOST to the LogInsight IP and save esc w q !

3. Apply the DS to the Kubernetes cluster kubectl apply -f fluentd-kubernetes-daemonset/fluentd-daemonset-syslog.yaml

Verify the success within the kube-system namespace kubectl get ds -n kube-system

It should be listed along with kube-proxy and whichever CNI you’re leveraging for your Kubernetes cluster, for me that is Antrea.

Testing to make sure it works

In order to test that the logs are shipping and being received, let’s deploy a simple webserver and send it a few requests. I’ve added the label app:nginx so when we create the nodeport service it will select this pod as it’s endpoint to communicate with.

kubectl run nginx --image=nginx --restart=Never --labels=app=nginx

Then create a nodeport service so we can access the default webpage from Nginx. By default, this command uses a selector as app and name of the service, Nginx.

kubectl create svc nodeport nginx --tcp=80

kubectl get svc This will allow us to see the port we need to access the test nginx webserver.

Okay! There should be some http requests we can view from LogInsight which is acting as our syslog server via fluentd DaemonSet running in our cluster!

Logging into LogInsight and selecting Interactive Analytics, with a simple ‘http’ search in the search bar should show our Nginx logs.

There you have it! Logs are now flowing from our Kubernetes into our existing LogInsight appliance and we are able to search for them.

You can match these against the logs being output within the Kubernetes cluster with the kubectl logs nginx command.

It’s not just our app logs that will be shipped, but Kubernetes logs as well. Within LogInsight and the Interactive Analytics window, filter the app name to fluentd and you should see all the logs being sent from the K8s cluster. For example, I had a failed postgres deployment which can be seen in the screenshot below.

That is a lot of material, but the steps are fairly simple and easy thanks to the work done by the fluentd project.

In part 2 of this blog, we will look at creating some dashboards within LogInsight that will help us more easily monitor and analyze the logs coming in from the Kubernetes cluster.

Helpful source docs:

AWS re:Invent 2019 Recap

Amazon AWS introduced almost eighty new services or service enhancements this year at re:Invent. Let’s go over a few of the more important ones.

Serverless

Serverless was one of the main focuses of re:Invent 2019. The big announcement was the launch of ‘provisioned concurrency’ for Lambda. Currently, there is some latency the first time Lambda is invoked because of ‘cold starts’, when containers need to initialize in the background to do processing for your functions. Provisioned concurrency mitigates this by allocating a pool of pre-initialized Lambda containers in the background. This should allow for better latency when a Lambda function is initialized for the first time.

Link to announcement: https://aws.amazon.com/about-aws/whats-new/2019/12/aws-lambda-announces-provisioned-concurrency/

A few other major announcements in the serverless compute space:

Security

IAM Access Analyzer was the biggest security announcement from re:Invent. This new feature continuously monitors your IAM policies for changes and alerts if anything has changed. When an IAM policy violates your security and access standards, it can be remediated faster.

Link to announcement: https://aws.amazon.com/about-aws/whats-new/2019/12/introducing-aws-identity-and-access-management-access-analyzer/

Other major releases and announcements in the security space:

Compute

ARM based compute is the coolest thing that came out of the major compute announcements. Graviton processors, custom ARM based CPUs designed by Amazon, can perform almost as well as x86 CPUs at a fraction of the cost.

Link to announcement: https://aws.amazon.com/about-aws/whats-new/2019/12/announcing-new-amazon-ec2-m6g-c6g-and-r6g-instances-powered-by-next-generation-arm-based-aws-graviton2-processors/

Other major releases and announcements in the AWS compute space:

Networking

The big announcement here seems to be AWS Wavelength, AWS services embedded into the datacenters of telecommunications providers. This will provide very low latency for sensitive applications.

Link to Announcement: https://aws.amazon.com/wavelength/

Other major networking releases and announcements:

Database

Enterprise

AWS Outpost was announced in 2018, but is now generally available. This allows for true hybrid functionality for the cloud with AWS services on-prem and in the public cloud. VMware also offers VMware Cloud on AWS Outpost for customers that want to bring the strengths of AWS and VMware together in their datacenters.

Link to Outposts GA announcement: https://aws.amazon.com/about-aws/whats-new/2019/12/announcing-general-availability-of-aws-outposts/

AWS re:Invent 2019 Keynotes & Further Announcements

If you’re interested in watching any of the keynotes, the re:Invent 2019 YouTube channel has them all here:
https://www.youtube.com/playlist?list=PLhr1KZpdzukcAXSVwQ3L9cWD4QgKPCQ5S

Announcements for the dozens of other new technologies we didn’t cover here can be found on the 2019 re:Invent announcement page:
https://aws.amazon.com/new/reinvent/

That’s about everything we’re going to cover. There was so much more announced this year, but these are what I think they key highlights are for. Thanks for reading!

Weekly Update – Week of 12/09/2019

Updated KB Articles:
New KB articles published for the week ending 1 December, 2019

Upcoming Events:
Gartner IOCS – Las Vegas – 12/09/2019 to 12/12/2019 – Register
Dell Technologies Word 2020 – Las Vegas – 05/04/2020 to 05/07/2020 – Register

Upcoming Webinars:
vSAN View and Dashboard Development in vROps – 12/12/2019 – Register
vCenter Upgrades, What’s in it for You? – 12/17/2019 – Register
[Full Live Event List]

New Releases:
2019-11-22
VMware Horizon Cloud Connector [Download]

KubeCon 2019 – VMware Recap

VMware was very busy this year at KubeCon with the announcement of three new open source projects, a new podcast with rockstar hosts, and presenting lots of sessions.  These new open source projects are in addition to the already very popular and widely adopted Velero, Contour, Sonobuoy and Octant.  The commitment and number of employees at VMware working in the cloud native and open source space is truly impressive.

The first announcement was on Project Antrea, which is an open source CNI for Kubernetes based on Open vSwitch (OVS).  This project aims to deliver a simple and secure Kubernetes networking CNI.  One fantastic feature is it’s plugin for Octant, which is another developer-focused, open source project that gives a very powerful GUI for visibility and management of Kubernetes applications.  With Project Antrea and Octant, you can get even more visibility into your microservices and connectivity.

The next project announced was Project Hamlet, which is a joint effort between VMware, Google Anthos, HashiCorp and Pivotal, is an effort to create an interoperable API for the federation of service meshes.  The end goal being an API that allows for interconnectivity of service meshes across heterogeneous cloud environments.

The third project is Crash Diagnostics for Kubernetes, which is way to automate the investigation of unhealthy or unresponsive Kubernetes clusters.  It does this by automating the collection of diagnostics from all of the nodes within a cluster and bundling that into a TAR file for further analysis.

If that wasn’t enough, a new podcast, ‘The Podlets” was announced: https://blogs.vmware.com/cloudnative/2019/11/20/introducing-podlets-podcast-audio-guide-to-cloud-native-concepts/

The hosts include an impressive list of experts in cloud native and distributed system topics and great all around people.  This will be a great resource for keeping up on the latest news in the fast paced ecosystem.  Direct link to “The Podlets” is at thepodlets.io .  It will be available on the normal podcast distribution platforms, as well as the Cloud Native Applications YouTube Channel here.

VMworld 2019 Europe Recap

Happy Sunday everyone, lets discuss some new announcements from our 2019 VMworld Europe conference.

Recap & Announcements:


– VMware Tanzu Progress and Design Principles: At VMworld US in San Francisco in August, Pat Gelsinger and Joe Beda were on stage to announce VMware Tanzu as a portfolio of products and services for customers that want to build, run and manage their applications on Kubernetes. Now here we are at VMworld Europe in Barcelona having made rapid progress toward our VMware Tanzu vision.

Please follow the link to our official blog for more details:
https://blogs.vmware.com/cloudnative/2019/11/05/vmware-tanzu-progress-design-principles/

Announcing VMware NSX Disributed IDS/IPS: Six years ago, VMware pioneered the concept of micro-segmentation to stop the internal, lateral spread of malware. We then launched the NSX Service-defined Firewall, an internal firewall that’s built into the hypervisor, distributed, and application aware. Shortly thereafter we introduced NSX Intelligence to automate security rule recommendations, streamlining the deployment of micro-segmentation. Now we are announcing that we will be taking internal security to the next level by introducing optional Intrusion Detection and Prevention (IDS/IPS) for our Service-defined Firewall. Built on the same philosophy, the new NSX Distributed IDS/IPS will allow enterprises to fortify applications across private and public clouds.

Please follow the link to our official blog for more details:
https://blogs.vmware.com/networkvirtualization/2019/11/extending-intrinsic-security.html/

– Interested in the Project Pacfic beta? Ever since we announced the technology preview of Project Pacific at VMworld 2019 back in August, customers and partners have been excited to hear more. It’s easy to see why. Leveraging vSphere to deploy and manage containers and Kubernetes infrastructure is a win-win for both vSphere administrators and application developers alike.

Please follow the link to our official blog for more details:
https://blogs.vmware.com/vsphere/2019/11/interested-in-the-project-pacific-beta.html

– VMware Introduces “Workspace ONE for Microsoft Endpoint Manager” to Enable Modern Management for Windows 10: VMware and Microsoft are working together to enable customers’ rapid move to modern management solutions that are built on cloud intelligence; and automate Windows 10 management tasks to help IT spend their precious time in the most impactful ways.

Please follow the link to our official blog for more details:
https://blogs.vmware.com/euc/2019/11/workspace-one-microsoft-endpoint-manager.html

– Announcing vRealize Network Insight 5.1: VMware is announcing the upcoming release of version 5.1 of both vRealize Network Insight and vRealize Network Insight Cloud. This next version of vRealize Network Insight will build on the momentum of the 5.0 release and include additional capabilities to help you discover, optimize and troubleshoot application security and network connectivity, no matter where the application livesdata center, cloud or even the branch.

Please follow the link to our official blog for more details:
https://blogs.vmware.com/management/2019/11/announcing-vrealize-network-insight-5-1.html

– Introducing Project Maestro: Today at VMworld Barcelona, VMware announced as a technology preview its new solution for telco cloud orchestration and automation: Project Maestro. This announcement heralds a critical milestone in VMware’s ongoing advancements in supporting the CSPs telco cloud transformation journey. This new purpose-built solution complements our growing Telco Cloud portfolio, providing capabilities that streamline network service orchestration and automation for any 4G and 5G networks.

Please follow the link to our official blog for more details:
https://blogs.vmware.com/telco/introducing-project-maestro/

——————————————————————————————————————-

VMworld Day 1 General Session Video:
https://www.youtube.com/watch?v=UHlo7aS_8Mc

VMworld Day 2 General Session Video:
https://www.youtube.com/watch?v=u3wWF-vTrzw

vForum 2019 Events

As we put VMworld and its many exciting announcements in our rearview mirror, it’s time to focus on spreading the news for those who were unable to attend, and diving deeper into the technologies that were discussed during the big event. For many of you, your local account team will bring a lot of that messaging to you directly, but another way that VMware does this is through our local and online vForum events.

This year, these events will take place on October 16th. vForum is a great way to engage with technical experts and executives that you may not be able to meet with during your day-to-day dealings, network with other professionals in your geography, and gain additional insight into VMware’s strategy as we move into a container and cloud-centric world. Please read on to learn more about how you can participate both locally and online.

vForum Hartford – Wednesday, October 16, 2019

11:00 AM – 5:30 PM (ET)
Thomas Hooker Brewery
140 Huyshope Avenue
Hartford, CT 06106
REGISTER NOW

Join us for our free local vForum event that will be packed with technical deep dives, peer to peer networking, and fun. Reserve your spot today to join us on October 16th. You will have the opportunity to hear recaps of the key announcements from VMworld and engage 1:1 with VMware technical experts on the newest developments in NSX, vSAN, and Cloud.

Here’s why you should attend:
Watch a livestream keynote with Pat Gelsinger, VMware CEO, followed by an Office of the CTO Expert Panel
Engage with technical experts on deep technical content
Compete for limited edition prizes that include a VMware Lego Set and T-Shirt
Access our latest Hands-on Labs with your own device to compete for a special VMware jacket
Give Back to your community cancer mission while testing your basketball skills

vForum Online – Wednesday, October 16, 2019

9:00 AM – 3:00 PM PDT
12:00 PM – 6:00 PM EDT
Agenda-at a Glance
Register Now!

Disruptive technologies are changing the way organizations are looking at cloud, networking, security, containers and the digital workspace to power their next wave of innovation. Join us at vForum Online, VMware’s largest virtual IT event for expert insight into:
Accelerating your cloud journey with VMware Cloud on AWS, vSphere Platinum, vSAN, Kubernetes and cloud-native apps.
Building the next generation network virtualization and security platform with NSX Data Center, SD-WAN by VeloCloud and App Defense.
Helping your employees work more easily and securely from anywhere, at any time, and on any device with Workspace ONE and Horizon.

Here’s why you should attend:
Exclusive thoughts and observations from theOffice of the CTO Expert Panel and guest customers.
38 technical breakouts on building, running, managing, and securing business-critical applications on any cloud; deploying network and security virtualization; and delivering seamless access to apps and services with a secure, integrated digital workspace.
Live Q&A video chats with more than 130 VMware experts who are ready to answer your toughest questions on cloud migration, networking, security, storage and the digital workspace.
10 instructor-led Hands-On Labs where you can test drive vSphere, vSAN, VMware Cloud on AWS, NSX, and Workspace ONE.

Kubernetes at VMware

What is the Strategy?

So many exciting announcements this year at VMworld have been around the cloud strategy of build, run, and manage. This strategy is outlined perfectly by Paul Fazzone here.

At the heart of these announcements is the integration of Kubernetes(K8s) into all things vSphere. With Project Pacific, Kubernetes will be embedded into vSphere to provide native K8s functionality within ESXi, as well as pure, open-source K8s clusters on-demand for developers.

With Tanzu Mission Control, VMware is enabling companies to manage their K8s clusters from a single location, bringing together operations and developers, and creating a single point of management to apply policies and governance to clusters deployed across a variety of environments on-premise and in public clouds.

Free Open-Source Kubernetes Training

With the increasing importance of Kubernetes to all IT professionals, it is important to provide the resources to enable people to master this new skillset. In that vein, another amazing announcement was the providing of free, vendor agnostic training for open-source Kubernetes by VMware. Available at kubernetes.academy. These courses provide a fantastic overview of containers and Kubernetes led by highly experienced instructors who have been deep in this ecosystem since the beginning.

Sign up today and start up-leveling your skills!

VMworld 2019 – Day 1 Recap

Day 1 Keynote

The day one general session was hosted by Pat Gelsinger, VMware’s CEO. He started with a quick mention of it being his seventh year as CEO of VMware, and then moved onto the theme for the day: “Tech in the Age of Any”. He spoke about the tremendous amount of variety that defines technology today, and how overwhelming all this choice can be if not managed properly.

The big news that everyone expected Pat to talk about was the acquisition of Pivotal and Carbon Black. He officially welcomed the two new companies into the “VMware family” and talked about the synergies that they will bring to the VMware portfolio.

Key Product Announcements:

  • VMware Tanzu, a portfolio of products and services to transform how enterprises build software for Kubernetes.
  • Project Pacific, which will re-architect Kubernetes into vSphere to create one platform to unite developers and operators to one platform.
  • CloudHealth Hybrid, which will take CloudHealth’s capabilities across both public and private clouds to optimize cost, usage, and performance.
  • Secure State, to help manage the security challenges and governance issues across multiple clouds.
  • Azure VMware Solutions is expanding its global reach with three areas currently available, five planned for 2019 and two planned for Q1 2020.
  • VMware Cloud on Dell EMC, a co-engineered Datacenter-as-a-Service solution between VMware and Dell built on VMware Cloud Foundation and Dell vxRail.
  • New versions of vRealize Operations, vRealize Automation, and vRealize Lifecycle Manager.

Key EUC Product Announcements:

  • “Consumer simple and enterprise secure” is VMware’s digital workspace philosophy.
  • Workspace ONE Virtual Assistant powered by Watson AI
  • Workspace ONE “Digital Employee Experience Management” to proactively identify and automatically remediate potential performance issues before they impact the computing experience.
  • New Trust Network integrations are GA for Lookout, Carbon Black and Netskope to incorporate third party risk information into Workspace ONE Intelligence.
  • Details on each of these new technologies can be found on VMware’s EUC Blog.

The day one general session can be watched here.