Tail logfiles with a webhook

The tailer-webhook is a different approach for the same problem: parsing legacy application’s log file. As an alternative to using a host file tailer service, you can use a file tailer webhook service. While the containers of the host file tailers run in a separated pod, file tailer webhook uses a different approach: if a pod has a specific annotation, the webhook injects a sidecar container for every tailed file into the pod.


The tailer-webhook behaves differently compared to the host-tailer:


  • A simple annotation on the pod initiates the file tailing.
  • There is no need to use mounted volumes, Logging operator will manage the volumes and mounts between your containers.


  • Required to start the Logging operator with webhooks service enabled. This requires additional configuration, especially on certificates since webhook services are allowed over TLS only.
  • Possibly uses more resources, since every tailed file attaches a new sidecar container to the pod.

Enable webhooks in Logging operator

We recommend using cert-manager to manage your certificates. Below is a really simple command that bootstraps generates the required resources for the tailer-webhook.

Issuing certificates using cert-manager

Follow the official installation guide.

Once installed the following commands should allow you to create the required certificate for the webhook.

kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
  name: selfsigned-issuer
  selfSigned: {}
apiVersion: cert-manager.io/v1
kind: Certificate
  name: webhook-tls
  namespace: logging
  isCA: true
  commonName: my-selfsigned-ca
  secretName: webhook-tls
    algorithm: ECDSA
    size: 256
    - sample-webhook.banzaicloud.com
    - logging-webhooks.logging.svc
    - server auth
    name: selfsigned-issuer
    kind: ClusterIssuer
    group: cert-manager.io
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
  name: my-ca-issuer
    secretName: webhook-tls

You will require the following things:

  • a valid client certificate,
  • a CA certificate, and
  • a custom value.yaml file for your helm chart.

The following example refers to a Kubernetes secret named webhook-tls which is a self-signed certificate generated by cert-manager.

Add the following lines to your custom values.yaml or create a new file if needed:

    value: "true"
  - name: webhook-tls
      secretName: webhook-tls
  - name: webhook-tls
    mountPath: /tmp/k8s-webhook-server/serving-certs

This will:

  • Set ENABLE_WEBHOOKS environment variable to true. This is the official way to enable webhooks in Logging operator.
  • Create a volume from the webhook-tls Kubernetes secret.
  • Mount the webhook-tls secret volume to the /tmp/k8s-webhook-server/serving-certs path where Logging operator will search for it.

Now you are ready to install Logging operator with the new custom values:

helm upgrade --install --wait --create-namespace --namespace logging -f operator_values.yaml  logging-operator ./charts/logging-operator

Alternatively, instead of using the values.yaml file, you can run the installation from command line also by passing the values with the set and set-string parameters:

helm upgrade --install --wait --create-namespace --namespace logging --set "env[0].name=ENABLE_WEBHOOKS" --set-string "env[0].value=true" --set "volumes[0].name=webhook-tls" --set "volumes[0].secret.secretName=webhook-tls" --set "volumeMounts[0].name=webhook-tls" --set "volumeMounts[0].mountPath=/tmp/k8s-webhook-server/serving-certs"  logging-operator ./charts/logging-operator

You also need a service which points to the webhook port (9443) of Logging operator, and where the mutatingwebhookconfiuration will point to. Running the following command in shell will create the required service:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
  name: logging-webhooks
  namespace: logging
    - name: logging-webhooks
      port: 443
      targetPort: 9443
      protocol: TCP
    app.kubernetes.io/instance: logging-operator
  type: ClusterIP

Furthermore, you need to tell Kubernetes to send admission requests to our webhook service. To do that, create a mutatingwebhookconfiguration Kubernetes resource, and:

  • Set the configuration to call /tailer-webhook path on your logging-webhooks service when v1.Pod is created.
  • Set failurePolicy to ignore, which means that the original pod will be created on webhook errors.
  • Set sideEffects to none, because we won’t cause any out-of-band changes in Kubernetes.

Unfortunately, mutatingwebhookconfiguration requires the caBundle field to be filled because we used a self-signed certificate, and the certificate cannot be validated through the system trust roots. If your certificate was generated with a system trust root CA, remove the caBundle line, because the certificate will be validated automatically. There are more sophisticated ways to load the CA into this field, but this solution requires no further components.

For example: you can inject the CA with a simple cert-manager cert-manager.io/inject-ca-from: logging/webhook-tls annotation on the mutatingwebhookconfiguration resource.

kubectl apply -f - <<EOF
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
  name: sample-webhook-cfg
  namespace: logging
    app: sample-webhook
    cert-manager.io/inject-ca-from: logging/webhook-tls
  - name: sample-webhook.banzaicloud.com
        name: logging-webhooks
        namespace: logging
        path: "/tailer-webhook"
      - operations: [ "CREATE" ]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        scope: "*"
    failurePolicy: Ignore
    sideEffects: None
    admissionReviewVersions: [v1]

Triggering the webhook


To use the webhook, you must first enable webhooks in the Logging operator.

File tailer webhook is based on a Mutating Admission Webhook. It is called every time when a pod starts.

To trigger the webhook, add the following annotation to the pod metadata:

  • Annotation key: sidecar.logging-extensions.banzaicloud.io/tail

  • Value of the annotation: the filename (including path, and optionally the container) you want to tail, for example:

    annotations: {"sidecar.logging-extensions.banzaicloud.io/tail": "/var/log/date"}
  • To tail multiple files, add only one annotation, and separate the filenames with commas, for example:

        name: test-pod
        annotations: {"sidecar.logging-extensions.banzaicloud.io/tail": "/var/log/date,/var/log/mycustomfile"}
  • If the pod contains multiple containers, see Multi-container pods.

Note: If the pod with the sidecar annotation is in the default namespace, Logging operator handles tailer-webhook annotations clusterwide. To restrict the webhook callbacks to the current namespace, change the scope of the mutatingwebhookconfiguration to namespaced.

File tailer example

The following example creates a pod that is running a shell in infinite loop that appends the date command’s output to a file every second. The annotation sidecar.logging-extensions.banzaicloud.io/tail notifies Logging operator to attach a sidecar container to the pod. The sidecar tails the /var/log/date file and sends its output to the stdout.

apiVersion: v1
kind: Pod
    name: test-pod
    annotations: {"sidecar.logging-extensions.banzaicloud.io/tail": "/var/log/date"}
    - image: debian
      name: sample-container
      command: ["/bin/sh", "-c"]
        - while true; do
            date >> /var/log/date;
            sleep 1;

After you have created the pod with the required annotation, make sure that the test-pod contains two containers by running kubectl get pod

Expected output:

test-pod   2/2     Running   0          29m

Check the container names in the pod to see that the Logging operator has created the sidecar container called legacy-logs-date-log. The sidecar containers’ name is always built from the path and name of the tailed file. Run the following command:

kubectl get pod test-pod -o json | jq '.spec.containers | map(.name)'

Expected output:


Check the logs of the test container. Since it writes the logs into a file, it does not produce any logs on stdout.

kubectl logs test-pod sample-container; echo $?

Expected output:


Check the logs of the legacy-logs-date-log container. This container exposes the logs of the test container on its stdout.

kubectl logs test-pod legacy-logs-date-log

Expected output:

Fluent Bit v1.9.5
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2022/09/15 11:26:11] [ info] [fluent bit] version=1.9.5, commit=9ec43447b6, pid=1
[2022/09/15 11:26:11] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
[2022/09/15 11:26:11] [ info] [cmetrics] version=0.3.4
[2022/09/15 11:26:11] [ info] [sp] stream processor started
[2022/09/15 11:26:11] [ info] [input:tail:tail.0] inotify_fs_add(): inode=938627 watch_fd=1 name=/legacy-logs/date.log
[2022/09/15 11:26:11] [ info] [output:file:file.0] worker #0 started
Thu Sep 15 11:26:11 UTC 2022
Thu Sep 15 11:26:12 UTC 2022

Multi-container pods

In some cases you have multiple containers in your pod and you want to distinguish which file annotation belongs to which container. You can order every file annotations to particular container by prefixing the annotation with a ${ContainerName}: container key. For example:

    name: test-pod
    annotations: {"sidecar.logging-extensions.banzaicloud.io/tail": "sample-container:/var/log/date,sample-container2:/var/log/anotherfile,/var/log/mycustomfile,foobarbaz:/foo/bar/baz"}


  • Annotations without containername prefix: the file gets tailed on the default container (container 0)
  • Annotations with invalid containername: file tailer annotation gets discarded
sample-container:/var/log/datetails file /var/log/date in sample-container
sample-container2:/var/log/anotherfiletails file /var/log/anotherfile in sample-container2
/var/log/mycustomfiletails file /var/log/mycustomfile in default container (sample-container)
foobarbaz:/foo/bar/bazwill be discarded due to non-existing container name