Fluent Bit log collector

Logos

Fluent Bit is an open source and multi-platform Log Processor and Forwarder which allows you to collect data/logs from different sources, unify and send them to multiple destinations.

Logging operator uses Fluent Bit as a log collector agent: Logging operator deploys Fluent Bit to your Kubernetes nodes where it collects and enriches the local logs and transfers them to a log forwarder instance.

Ways to configure Fluent Bit

There are three ways to configure the Fluent Bit daemonset:

  1. Using the spec.fluentbit section of The Logging custom resource. This method is deprecated and will be removed in the next major release.
  2. Using the standalone FluentbitAgent CRD. This method is only available in Logging operator version 4.2 and newer, and the specification of the CRD is compatible with the spec.fluentbit configuration method.
  3. Using the spec.nodeagents section of The Logging custom resource. This method is deprecated and will be removed from the Logging operator. (Note that this configuration isn’t compatible with the FluentbitAgent CRD.)

For the detailed list of available parameters, see FluentbitSpec.

Migrating from spec.fluentbit to FluentbitAgent

The standalone FluentbitAgent CRD is only available in Logging operator version 4.2 and newer. Its specification and logic is identical with the spec.fluentbit configuration method. Using the FluentbitAgent CRD allows you to remove the spec.fluentbit section from the Logging CRD, which has the following benefits.

  • RBAC control over the FluentbitAgent CRD, so you can have separate roles that can manage the Logging resource and the FluentbitAgent resource (that is, the Fluent Bit deployment).
  • It reduces the size of the Logging resource, which can grow big enough to reach the annotation size limit in certain scenarios (e.g. when using kubectl apply).
  • It allows you to use multiple different Fluent Bit configurations within the same cluster. For details, see Multiple Fluent Bit agents in the cluster.

To migrate your spec.fluentbit configuration from the Logging resource to a separate FluentbitAgent CRD, complete the following steps.

  1. Open your Logging resource and find the spec.fluentbit section. For example:

    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Logging
    metadata:
      name: example-logging-resource
    spec:
        controlNamespace: default
        fluentbit:
            inputTail:
              storage.type: filesystem
            positiondb:
              hostPath:
                path: ""
            bufferStorageVolume:
              hostPath:
                path: ""
    
  2. Create a new FluentbitAgent CRD. For the value of metadata.name, use the name of the Logging resource, for example:

    apiVersion: logging.banzaicloud.io/v1beta1
    kind: FluentbitAgent
    metadata:
      # Use the name of the logging resource
      name: example-logging-resource
    
  3. Copy the the spec.fluentbit section from the Logging resource into the spec section of the FluentbitAgent CRD, then fix the indentation.

  4. Specify the paths for the positiondb and the bufferStorageVolume. If you used the default settings in the spec.fluentbit configuration, set empty strings as paths, like in the following example. This is needed to retain the existing buffers of the deployment, otherwise data loss may occur.

    apiVersion: logging.banzaicloud.io/v1beta1
    kind: FluentbitAgent
    metadata:
      # Use the name of the logging resource
      name: example-logging-resource
    spec:
      inputTail:
        storage.type: filesystem
      positiondb:
        hostPath:
          path: ""
      bufferStorageVolume:
        hostPath:
          path: ""
    
  5. Delete the spec.fluentbit section from the Logging resource, then apply the Logging and the FluentbitAgent CRDs.

Examples

The following sections show you some examples on configuring Fluent Bit. For the detailed list of available parameters, see FluentbitSpec.

Note: These examples use the traditional method that configures the Fluent Bit deployment using spec.fluentbit section of The Logging custom resource.

Filters

Kubernetes (filterKubernetes)

Fluent Bit Kubernetes Filter allows you to enrich your log files with Kubernetes metadata. For example:

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: default
spec:
  filterKubernetes:
    Kube_URL: "https://kubernetes.default.svc:443"

For the detailed list of available parameters for this plugin, see FilterKubernetes. More info

Tail input

The tail input plugin allows to monitor one or several text files. It has a similar behavior like tail -f shell command. For example:

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: default-logging-simple
spec:
  inputTail:
    storage.type: filesystem
    Refresh_Interval: "60"
    Rotate_Wait: "5"

For the detailed list of available parameters for this plugin, see InputTail. More Info.

Buffering

Buffering in Fluent Bit places the processed data into a temporal location until is sent to Fluentd. By default, the Logging operator sets storage.path to /buffers and leaves fluent-bit defaults for the other options.

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: default-logging-simple
spec:
  inputTail:
    storage.type: filesystem
  bufferStorage:
    storage.path: /buffers

For the detailed list of available parameters for this plugin, see BufferStorage. More Info.

HostPath volumes for buffers and positions

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: default-logging-simple
spec:
  inputTail:
    storage.type: filesystem
  bufferStorageVolume:
    hostPath:
      path: "" # leave it empty to automatically generate
  positiondb:
    hostPath:
      path: "" # leave it empty to automatically generate

Custom Fluent Bit image

You can deploy custom images by overriding the default images using the following parameters.

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: default-logging-simple
spec:
  image:
    repository: fluent/fluent-bit
    tag: 2.1.8-debug
    pullPolicy: IfNotPresent

Volume Mount

Defines a pod volume mount. For example:

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: default-logging
spec:
  extraVolumeMounts:
  - destination: /data/docker/containers
    readOnly: true
    source: /data/docker/containers

For the detailed list of available parameters for this plugin, see VolumeMount.

Custom Fluent Bit annotations

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: default-logging-simple
spec:
  annotations:
    my-annotations/enable: true

KubernetesStorage

Define Kubernetes storage.

NameTypeDefaultDescription
hostPathHostPathVolumeSource-Represents a host path mapped into a pod. If path is empty, it will automatically be set to /opt/logging-operator/<name of the logging CR>/<name of the volume> 
emptyDirEmptyDirVolumeSource-Represents an empty directory for a pod. 

CPU and memory requirements

To adjust the CPU and memory limits and requests of the pods managed by Logging operator, see CPU and memory requirements.

Probe

A Probe is a diagnostic performed periodically by the kubelet on a Container. To perform a diagnostic, the kubelet calls a Handler implemented by the Container. You can configure a probe for Fluent Bit in the livenessProbe section of the The Logging custom resource. For example:

apiVersion: logging.banzaicloud.io/v1beta1
kind: FluentbitAgent
metadata:
  name: default-logging-simple
spec:
  livenessProbe:
    periodSeconds: 60
    initialDelaySeconds: 600
    exec:
      command:
      - "/bin/sh"
      - "-c"
      - >
        LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300};
        if [ ! -e /buffers ]; then
          exit 1;
        fi;
        touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness;
        if [ -z "$(find /buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
          exit 1;
        fi;        

You can use the following parameters:

NameTypeDefaultDescription
initialDelaySecondsint10Number of seconds after the container has started before liveness probes are initiated.
timeoutSecondsint0Number of seconds after which the probe times out.
periodSecondsint10How often (in seconds) to perform the probe.
successThresholdint0Minimum consecutive successes for the probe to be considered successful after having failed.
failureThresholdint3Minimum consecutive failures for the probe to be considered failed after having succeeded.
execarray{}Exec specifies the action to take. More info
httpGetarray{}HTTPGet specifies the http request to perform. More info
tcpSocketarray{}TCPSocket specifies an action involving a TCP port. More info

Note: To configure readiness probes, see Readiness probe.