Troubleshooting Fluent Bit
The following sections help you troubleshoot the Fluent Bit component of the Logging operator.
Check the Fluent Bit daemonset
Verify that the Fluent Bit daemonset is available. Issue the following command: kubectl get daemonsets
The output should include a Fluent Bit daemonset, for example:
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
logging-demo-fluentbit 1 1 1 1 1 <none> 110s
Check the Fluent Bit configuration
You can display the current configuration of the Fluent Bit daemonset using the following command:
kubectl get secret logging-demo-fluentbit -o jsonpath="{.data['fluent-bit\.conf']}" | base64 --decode
The output looks like the following:
[SERVICE]
Flush 1
Daemon Off
Log_Level info
Parsers_File parsers.conf
storage.path /buffers
[INPUT]
Name tail
DB /tail-db/tail-containers-state.db
Mem_Buf_Limit 5MB
Parser docker
Path /var/log/containers/*.log
Refresh_Interval 5
Skip_Long_Lines On
Tag kubernetes.*
[FILTER]
Name kubernetes
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Tag_Prefix kubernetes.var.log.containers
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_URL https://kubernetes.default.svc:443
Match kubernetes.*
Merge_Log On
[OUTPUT]
Name forward
Match *
Host logging-demo-fluentd.logging.svc
Port 24240
tls On
tls.verify Off
tls.ca_file /fluent-bit/tls/ca.crt
tls.crt_file /fluent-bit/tls/tls.crt
tls.key_file /fluent-bit/tls/tls.key
Shared_Key Kamk2_SukuWenk
Retry_Limit False
Debug version of the fluentbit container
All Fluent Bit image tags have a debug version marked with the -debug
suffix. You can install this debug version using the following command:
kubectl edit loggings.logging.banzaicloud.io logging-demo
fluentbit:
image:
pullPolicy: Always
repository: fluent/fluent-bit
tag: 1.3.2-debug
After deploying the debug version, you can kubectl exec into the pod using sh
and look around. For example: kubectl exec -it logging-demo-fluentbit-778zg sh
Check the queued log messages
You can check the buffer directory if Fluent Bit is configured to buffer queued log messages to disk instead of in memory. (You can configure it through the InputTail fluentbit config, by setting the storage.type
field to filesystem
.)
kubectl exec -it logging-demo-fluentbit-9dpzg ls /buffers
Getting Support
If you encounter any problems that the documentation does not address, file an issue or talk to us on Discord or Slack.
Before asking for help, prepare the following information to make troubleshooting faster:
- Logging operator version
- kubernetes version
- helm/chart version (if you installed the Logging operator with helm)
- Logging operator logs
- fluentd configuration
- fluentd logs
- fluentbit configuration
- fluentbit logs
Do not forget to remove any sensitive information (for example, passwords and private keys) before sharing.