Logging operator troubleshooting
The following tips and commands can help you to troubleshoot your Logging operator installation.
First things to do
-
Check that the necessary CRDs are installed. Issue the following command:
kubectl get crd
The output should include the following CRDs:clusterflows.logging.banzaicloud.io 2019-12-05T15:11:48Z clusteroutputs.logging.banzaicloud.io 2019-12-05T15:11:48Z flows.logging.banzaicloud.io 2019-12-05T15:11:48Z loggings.logging.banzaicloud.io 2019-12-05T15:11:48Z outputs.logging.banzaicloud.io 2019-12-05T15:11:48Z
-
Verify that the Logging operator pod is running. Issue the following command:
kubectl get pods |grep logging-operator
The output should include the a running pod, for example:NAME READY STATUS RESTARTS AGE logging-demo-log-generator-6448d45cd9-z7zk8 1/1 Running 0 24m
-
Check the status of your resources. Beginning with Logging Operator 3.8, all custom resources have a
Status
and aProblems
field. In a healthy system, the Problems field of the resources is empty, for example:kubectl get clusteroutput -A
Sample output:
NAMESPACE NAME ACTIVE PROBLEMS default nullout true
The
ACTIVE
column indicates that theClusterOutput
has successfully passed theconfigcheck
and presented it in the current fluentd configuration. When no errors are reported thePROBLEMS
column is empty.Take a look at another example, in which we have an incorrect
ClusterFlow
.kubectl get clusterflow -o wide
Sample output:
NAME ACTIVE PROBLEMS all-log true nullout false 1
You can see that the nullout
Clusterflow
is inactive and there is 1 problem with the configuration. To display the problem, check thestatus
field of the object, for example:kubectl get clusterflow nullout -o=jsonpath='{.status}' | jq
Sample output:
{ "active": false, "problems": [ "dangling global output reference: nullout2" ], "problemsCount": 1 }
After that, check the following sections for further tips.
Getting Support
If you encounter any problems that the documentation does not address, file an issue or talk to us on Discord or Slack.
Commercial support is also available for the Logging operator.
Before asking for help, prepare the following information to make troubleshooting faster:
- Logging operator version
- kubernetes version
- helm/chart version (if you installed the Logging operator with helm)
- Logging operator logs
- fluentd configuration
- fluentd logs
- fluentbit configuration
- fluentbit logs
Do not forget to remove any sensitive information (for example, passwords and private keys) before sharing.