Kubernetes, Master Service Logging
I noticed that none of the Kubernetes services are writing logs to
/var/log/ on the master nodes. I built my lab with kubeadm and a reasonably basic configuration file, just the settings needed to make it work.
The configuration file contained just enough information to build the lab and make it work. The Kubernetes services
Scheduler have several configuration flags for logging.
In this post, I am going to go through my experience, enabling logging with kubeadm. The goal is to enable the services to write logs to
/var/log/, not integration with a central logging system.
The below code block is the configuration file I used to build my lab cluster with kubeadm:
The Command line reference page provides additional arguments for each service. It appeared that I needed to add the
--log-dir flag as an extra argument for each of the services and then run
sudo kubeadm init phase control-plane all --config=configfile.yaml to generate new manifests for the ApiServer, ControllManager and Scheduler. The reality was not quite so simple, and I found a couple of sticking points to overcome.
Before continuing, I think it’s important to mention that the Kubernetes master services are running as static pods on each of the master nodes. If you’re not familiar with static pods; here’s a post I prepared earlier: Static Pods
The first (and third) hurdle was making the logs available on the master and not just within the filesystem of the container. I could make this work by manually configuring the pod manifests, but, that doesn’t teach the lessons I’m trying to learn. The ‘How to update configurations with kubeadm’, enabling logging to file is a secondary lesson. One attempt was adding
volumeMounts to the configuration file, and the other was using
extraMounts, these didn’t work.
I found that I needed to upgrade my configuration file to one that supported the required options. My lab was built using kubeadm 1.15 (I think), and I have just updated it to 1.16. Meaning I was looking at documentation which didn’t align to the configuration file version I currently had.
Upgrading the configuration file was a quick and painless effort, using the commands below.
The config file now looks a lot different.
Now we are back at the problem of making the log files available on the local master nodes. This problem was solved by adding the below code below to the following sections in the new configuration file:
I then ran the following commands to test that the mounting and permissions worked.
Things are looking promising, kubeadm is using the config file to generate manifests which provide an additional volume mount for log files.
The next part took quite some time to work through; getting the services actually to write the bloody logs to file. Like many things that take a long time, it seemed very easy at first glance. The task appeared to be
log-dir: /var/log/kubernetes to the follow locations in the configuration file;
Kubeadm was used to generate new manifests, but, no-log files were appearing for any of the three services. I validated that the extra argument was in the manifests. An additional argument was required to stop logging to
stderr; I added
logtostderr: false to
This is where two things happened; kubeadm would no longer generate manifests, and I began to dig into the issue further (and cry crocodile tears, like saltwater crocodiles, not those itty bitty freshwater crocs).
Manually adding the argument to the manifest worked a treat and logs began to appear on the master node. But, like the task of mounting volumes, it doesn’t achieve the desired objective.
After quite some time of searching, I stumbled across a response to an arbitrary issue on GitHub; kubeadm expects all extraArgs values to be strings. In YAML, a value of ‘true’ or ‘false’ is considered as type boolean unless wrapped in quotation marks. Kubeadm generated manifests and logs were being written to /var/log/kubernetes on the master node.
As this cluster is a multi-master cluster, the config file needs to be copied to the other master nodes and kubeadm run locally on them to generate new manifests. Below is the complete configuration file.