Troubleshooting Fluentd

The following sections help you troubleshoot the Fluentd statefulset component of the Logging operator.

Check Fluentd pod status (statefulset)

Verify that the Fluentd statefulset is available using the following command: kubectl get statefulsets

Expected output:

NAME                   READY   AGE
logging-demo-fluentd   1/1     1m

ConfigCheck

The Logging operator has a builtin mechanism that validates the generated fluentd configuration before applying it to fluentd. You should be able to see the configcheck pod and its log output. The result of the check is written into the status field of the corresponding Logging resource.

In case the operator is stuck in an error state caused by a failed configcheck, restore the previous configuration by modifying or removing the invalid resources to the point where the configcheck pod is finally able to complete successfully.

Check Fluentd configuration

Use the following command to display the configuration of Fluentd: kubectl get secret logging-demo-fluentd-app -o jsonpath="{.data['fluentd\.conf']}" | base64 --decode

The output should be similar to the following:

<source>
  @type forward
  @id main_forward
  bind 0.0.0.0
  port 24240
  <transport tls>
    ca_path /fluentd/tls/ca.crt
    cert_path /fluentd/tls/tls.crt
    client_cert_auth true
    private_key_path /fluentd/tls/tls.key
    version TLSv1_2
  </transport>
  <security>
    self_hostname fluentd
    shared_key Kamk2_SukuWenk
  </security>
</source>
<match **>
  @type label_router
  @id main_label_router
  <route>
    @label @427b3e18f3a3bc3f37643c54e9fc960b
    labels app.kubernetes.io/instance:logging-demo,app.kubernetes.io/name:log-generator
    namespace logging
  </route>
</match>
<label @427b3e18f3a3bc3f37643c54e9fc960b>
  <match kubernetes.**>
    @type tag_normaliser
    @id logging-demo-flow_0_tag_normaliser
    format ${namespace_name}.${pod_name}.${container_name}
  </match>
  <filter **>
    @type parser
    @id logging-demo-flow_1_parser
    key_name log
    remove_key_name_field true
    reserve_data true
    <parse>
      @type nginx
    </parse>
  </filter>
  <match **>
    @type s3
    @id logging_logging-demo-flow_logging-demo-output-minio_s3
    aws_key_id WVKblQelkDTSKTn4aaef
    aws_sec_key LAmjIah4MTKTM3XGrDxuD2dTLLmysVHvZrtxpzK6
    force_path_style true
    path logs/${tag}/%Y/%m/%d/
    s3_bucket demo
    s3_endpoint http://logging-demo-minio.logging.svc.cluster.local:9000
    s3_region test_region
    <buffer tag,time>
      @type file
      path /buffers/logging_logging-demo-flow_logging-demo-output-minio_s3.*.buffer
      retry_forever true
      timekey 10s
      timekey_use_utc true
      timekey_wait 0s
    </buffer>
  </match>
</label>

Set Fluentd log Level

Use the following command to change the log level of Fluentd. kubectl edit loggings.logging.banzaicloud.io logging-demo

spec:
  fluentd:
    logLevel: debug

Get Fluentd logs

The following command displays the logs of the Fluentd container.

kubectl logs -f logging-demo-fluentd-0 -c fluentd

Fluentd logs were written to the container filesystem up until Logging operator version 4.3, which has been changed to stdout with 4.4. See FluentOutLogrotate why this was changed and how you can re-enable it if needed.

Tip: If the logs include the error="can't create buffer file ... error message, Fluentd can’t create the buffer file at the specified location. This can mean for example that the disk is full, the filesystem is read-only, or some other permission error. Check the buffer-related settings of your Fluentd configuration.

Set stdout as an output

You can use an stdout filter at any point in the flow to dump the log messages to the stdout of the Fluentd container. For example: kubectl edit loggings.logging.banzaicloud.io logging-demo

apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: exchange
  namespace: logging
spec:
  filters:
    - stdout: {}
  localOutputRefs:
    - exchange
  selectors:
    application: exchange

Check the buffer path in the fluentd container

kubectl exec -it logging-demo-fluentd-0 ls /buffers

Defaulting container name to fluentd.
Use 'kubectl describe pod/logging-demo-fluentd-0 -n logging' to see all of the containers in this pod.
logging_logging-demo-flow_logging-demo-output-minio_s3.b598f7eb0b2b34076b6da13a996ff2671.buffer
logging_logging-demo-flow_logging-demo-output-minio_s3.b598f7eb0b2b34076b6da13a996ff2671.buffer.meta

Getting Support

If you encounter any problems that the documentation does not address, file an issue or talk to us on Discord or Slack.

Before asking for help, prepare the following information to make troubleshooting faster:

• Logging operator version
• kubernetes version
• helm/chart version (if you installed the Logging operator with helm)
• Logging operator logs
• fluentd configuration
• fluentd logs
• fluentbit configuration
• fluentbit logs

Do not forget to remove any sensitive information (for example, passwords and private keys) before sharing.