Kubernetes host logs, journals, and logfiles

Kubernetes host tailer allows you to tail logs like kubelet, audit logs, or the systemd journal from the nodes.

Host-tailer

Create host tailer

To tail logs from the node’s host filesystem, define one or more file tailers in the host-tailer configuration.

kubectl apply -f - <<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: multi-sample
spec:
  # list of File tailers
  fileTailers:
    - name: nginx-access
      path: /var/log/nginx/access.log
    - name: nginx-error
      path: /var/log/nginx/error.log
  # list of Systemd tailers
  systemdTailers:
    - name: my-systemd-tailer
      maxEntries: 100
      systemdFilter: kubelet.service
EOF

Create file tailer

When an application (mostly legacy programs) is not logging in a Kubernetes-native way, Logging operator cannot process its logs. (For example, an old application does not send its logs to stdout, but uses some log files instead.) File-tailer helps to solve this problem: It configures Fluent Bit to tail the given file(s), and sends the logs to the stdout, to implement Kubernetes-native logging.

Host-tailer

However, file-tailer cannot access the pod’s local dir, so the logfiles need to be written on a mounted volume.

Let’s assume the following code represents a legacy application that generates logs into the /legacy-logs/date.log file. While the legacy-logs directory is mounted, it’s accessible from other pods by mounting the same volume.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - image: busybox
    name: test
    volumeMounts:
    - mountPath: /legacy-logs
      name: test-volume
    command: ["/bin/sh", "-c"]
    args:
      - while true; do
          date >> /legacy-logs/date.log;
          sleep 1;
        done
  volumes:
  - name: test-volume
    hostPath:
      path: /legacy-logs
EOF

To tail the logs of the previous example application, you can use the following host-tailer custom resource:

kubectl apply -f - <<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: file-hosttailer-sample
spec:
  fileTailers:
    - name: sample-logfile
      path: /legacy-logs/date.log
      disabled: false
EOF

Logging operator configure the environment and start a file-tailer pod. It’s also able to deal with multi-node clusters, since is starts the host-tailer pod through a daemonset.

Check the created file tailer pod:

kubectl get pod

The output should be similar to:

NAME                                       READY   STATUS    RESTARTS   AGE
file-hosttailer-sample-host-tailer-5tqhv   1/1     Running   0          117s
test-pod                                   1/1     Running   0          5m40s

Checking the logs of the file-tailer's pod. You will see the logfile’s content on stdout. This way Logging operator can process those logs as well.

kubectl logs file-hosttailer-sample-host-tailer-5tqhv

The logs of the sample application should be similar to:

Fluent Bit v1.9.5
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/09/13 12:26:02] [ info] [fluent bit] version=1.9.5, commit=9ec43447b6, pid=1
[2022/09/13 12:26:02] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/09/13 12:26:02] [ info] [cmetrics] version=0.3.4
[2022/09/13 12:26:02] [ info] [sp] stream processor started
[2022/09/13 12:26:02] [ info] [output:file:file.0] worker #0 started
[2022/09/13 12:26:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=418051 watch_fd=1 name=/legacy-logs/date.log
Tue Sep 13 12:22:51 UTC 2022
Tue Sep 13 12:22:52 UTC 2022
Tue Sep 13 12:22:53 UTC 2022
Tue Sep 13 12:22:54 UTC 2022
Tue Sep 13 12:22:55 UTC 2022
Tue Sep 13 12:22:56 UTC 2022

File Tailer configuration options

Variable Name	Type	Required	Default	Description
name	string	Yes	-	Name for the tailer
path	string	No	-	Path to the loggable file
disabled	bool	No	-	Disable tailing the file
containerOverrides	*types.ContainerBase	No	-	Override container fields for the given tailer

Tail systemd journal

This is a special case of file-tailer, since it tails the systemd journal file specifically.

kubectl apply -f - <<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: systemd-tailer-sample
spec:
  # list of Systemd tailers
  systemdTailers:
    - name: my-systemd-tailer
      maxEntries: 100
      systemdFilter: kubelet.service
EOF

Systemd tailer configuration options

Variable Name	Type	Required	Default	Description
name	string	Yes	-	Name for the tailer
path	string	No	-	Override systemd log path
disabled	bool	No	-	Disable component
systemdFilter	string	No	-	Filter to select systemd unit example: kubelet.service
maxEntries	int	No	-	Maximum entries to read when starting to tail logs to avoid high pressure
containerOverrides	*types.ContainerBase	No	-	Override container fields for the given tailer

Example: Configure logging Flow to route logs from a host tailer

The following example uses the flow’s match term to listen the previously created file-hosttailer-sample Hosttailer’s log.

kubectl apply -f - <<EOF
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: hosttailer-flow
  namespace: default
spec:
  filters:
  - tag_normaliser: {}
  # keeps data matching to label, the rest of the data will be discarded by this flow implicitly
  match:
  - select:
      labels: 
        app.kubernetes.io/name: file-hosttailer-sample
      # there might be a need to match on container name too (in case of multiple containers)
      container_names:
        - nginx-access
  outputRefs:
    - sample-output
EOF

Example: Kubernetes host tailer with multiple tailers

kubectl apply -f - <<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: multi-sample
spec:
  # list of File tailers
  fileTailers:
    - name: nginx-access
      path: /var/log/nginx/access.log
    - name: nginx-error
      path: /var/log/nginx/error.log
  # list of Systemd tailers
  systemdTailers:
    - name: my-systemd-tailer
      maxEntries: 100
      systemdFilter: kubelet.service
EOF

Set custom priority

Create your own custom priority class in Kubernetes. Set its value between 0 and 2000000000. Note that:

0 is the default priority
To change the default priority, set the globalDefault key.
2000000000 and above are reserved for the Kubernetes system
PriorityClass is a non-namespaced object.

kubectl apply -f - <<EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: hosttailer-priority
value: 1000000
globalDefault: false
description: "This priority class should be used for hosttailer pods only."
EOF

Now you can use your private priority class name to start hosttailer/eventtailer, for example:

kubectl apply -f -<<EOF
apiVersion: logging-extensions.banzaicloud.io/v1alpha1
kind: HostTailer
metadata:
  name: priority-sample
spec:
  controlNamespace: default
  # Override podSpecBase variables here
  workloadOverrides:
    priorityClassName: hosttailer-priority
  fileTailers:
    - name: nginx-access
      path: /var/log/nginx/access.log
    - name: nginx-error
      path: /var/log/nginx/error.log
EOF

Configuration options

Variable Name	Type	Required	Default	Description
fileTailers	[]FileTailer	No	-	List of file tailers
systemdTailers	[]SystemdTailer	No	-	List of systemd tailers
enableRecreateWorkloadOnImmutableFieldChange	bool	No	-	EnableRecreateWorkloadOnImmutableFieldChange enables the operator to recreate the fluentbit daemonset and the fluentd statefulset (and possibly other resource in the future) in case there is a change in an immutable field that otherwise couldn’t be managed with a simple update.
workloadMetaOverrides	*types.MetaBase	No	-	Override metadata of the created resources
workloadOverrides	*types.PodSpecBase	No	-	Override podSpec fields for the given daemonset

Advanced configuration overrides

MetaBase

Variable Name	Type	Required	Default	Description
annotations	`map[string]string`	No	-
labels	`map[string]string`	No	-

PodSpecBase

Variable Name	Type	Required	Default
tolerations	`[]corev1.Toleration`	No	-
nodeSelector	`map[string]string`	No	-
serviceAccountName	`string`	No	-
affinity	`*corev1.Affinity`	No	-
securityContext	`*corev1.PodSecurityContext`	No	-
volumes	`[]corev1.Volume`	No	-
priorityClassName	`string`	No	-

ContainerBase

Variable Name	Type	Required	Default
resources	`*corev1.ResourceRequirements`	No	-
image	`string`	No	-
pullPolicy	`corev1.PullPolicy`	No	-
command	`[]string`	No	-
volumeMounts	`[]corev1.VolumeMount`	No	-
securityContext	`*corev1.SecurityContext`	No	-

Last modified December 27, 2023: Version number bumps (00b4afd)