Logging operator troubleshooting

The following tips and commands can help you to troubleshoot your Logging operator installation.

First things to do

1. Check that the necessary CRDs are installed. Issue the following command: kubectl get crd The output should include the following CRDs:

   clusterflows.logging.banzaicloud.io     2019-12-05T15:11:48Z
   clusteroutputs.logging.banzaicloud.io   2019-12-05T15:11:48Z
   flows.logging.banzaicloud.io            2019-12-05T15:11:48Z
   loggings.logging.banzaicloud.io         2019-12-05T15:11:48Z
   outputs.logging.banzaicloud.io          2019-12-05T15:11:48Z
   

2. Verify that the Logging operator pod is running. Issue the following command: kubectl get pods |grep logging-operator The output should include the a running pod, for example:

   NAME                                          READY   STATUS      RESTARTS   AGE
   logging-demo-log-generator-6448d45cd9-z7zk8   1/1     Running     0          24m
   

3. Check the status of your resources. Beginning with Logging Operator 3.8, all custom resources have a Status and a Problems field. In a healthy system, the Problems field of the resources is empty, for example:

   kubectl get clusteroutput -A
   

   Sample output:

   NAMESPACE   NAME      ACTIVE   PROBLEMS
   default     nullout   true
   

   The ACTIVE column indicates that the ClusterOutput has successfully passed the configcheck and presented it in the current fluentd configuration. When no errors are reported the PROBLEMS column is empty.

   Take a look at another example, in which we have an incorrect ClusterFlow.

   kubectl get clusterflow -o wide
   

   Sample output:

   NAME      ACTIVE   PROBLEMS
   all-log   true
   nullout   false    1
   

   You can see that the nullout Clusterflow is inactive and there is 1 problem with the configuration. To display the problem, check the status field of the object, for example:

   kubectl get clusterflow nullout -o=jsonpath='{.status}' | jq
   

   Sample output:

   {
   "active": false,
   "problems": [
       "dangling global output reference: nullout2"
   ],
   "problemsCount": 1
   }
   

After that, check the following sections for further tips.

Getting Support

If you encounter any problems that the documentation does not address, file an issue or talk to us on Discord or Slack.

Before asking for help, prepare the following information to make troubleshooting faster:

• Logging operator version
• kubernetes version
• helm/chart version (if you installed the Logging operator with helm)
• Logging operator logs
• fluentd configuration
• fluentd logs
• fluentbit configuration
• fluentbit logs

Do not forget to remove any sensitive information (for example, passwords and private keys) before sharing.