Configure log routing

You can configure the various features and parameters of the Logging operator using Custom Resource Definitions (CRDs).

The Logging operator manages the log collectors and log forwarders of your logging infrastructure, and the routing rules that specify where you want to send your different log messages.

The log collectors are endpoint agents that collect the logs of your Kubernetes nodes and send them to the log forwarders. Logging operator currently uses Fluent Bit as log collector agents.

The log forwarder instance receives, filters, and transforms the incoming logs, and transfers them to one or more destination outputs. The Logging operator supports Fluentd and syslog-ng as log forwarders. Which log forwarder is best for you depends on your logging requirements. For tips, see Which log forwarder to use.

You can filter and process the incoming log messages using the flow custom resource of the log forwarder to route them to the appropriate output. The outputs are the destinations where you want to send your log messages, for example, Elasticsearch, or an Amazon S3 bucket. You can also define cluster-wide outputs and flows, for example, to use a centralized output that namespaced users can reference but cannot modify. Note that flows and outputs are specific to the type of log forwarder you use (Fluentd or syslog-ng).

You can configure the Logging operator using the following Custom Resource Definitions.

• logging - The logging resource defines the logging infrastructure (the log collectors and forwarders) for your cluster that collects and transports your log messages. It also contains configurations for Fluent Bit, Fluentd, and syslog-ng.
• CRDs for Fluentd:
  • output - Defines a Fluentd Output for a logging flow, where the log messages are sent using Fluentd. This is a namespaced resource. See also clusteroutput. To configure syslog-ng outputs, see SyslogNGOutput.
  • flow - Defines a Fluentd logging flow using filters and outputs. Basically, the flow routes the selected log messages to the specified outputs. This is a namespaced resource. See also clusterflow. To configure syslog-ng flows, see SyslogNGFlow.
  • clusteroutput - Defines a Fluentd output that is available from all flows and clusterflows. The operator evaluates clusteroutputs in the controlNamespace only unless allowClusterResourcesFromAllNamespaces is set to true.
  • clusterflow - Defines a Fluentd logging flow that collects logs from all namespaces by default. The operator evaluates clusterflows in the controlNamespace only unless allowClusterResourcesFromAllNamespaces is set to true. To configure syslog-ng clusterflows, see SyslogNGClusterFlow.
• CRDs for syslog-ng (these resources like their Fluentd counterparts, but are tailored to features available via syslog-ng):
  • SyslogNGOutput - Defines a syslog-ng Output for a logging flow, where the log messages are sent using Fluentd. This is a namespaced resource. See also SyslogNGClusterOutput. To configure Fluentd outputs, see output.
  • SyslogNGFlow - Defines a syslog-ng logging flow using filters and outputs. Basically, the flow routes the selected log messages to the specified outputs. This is a namespaced resource. See also SyslogNGClusterFlow. To configure Fluentd flows, see flow.
  • SyslogNGClusterOutput - Defines a syslog-ng output that is available from all flows and clusterflows. The operator evaluates clusteroutputs in the controlNamespace only unless allowClusterResourcesFromAllNamespaces is set to true.
  • SyslogNGClusterFlow - Defines a syslog-ng logging flow that collects logs from all namespaces by default. The operator evaluates clusterflows in the controlNamespace only unless allowClusterResourcesFromAllNamespaces is set to true. To configure Fluentd clusterflows, see clusterflow.

The following sections show examples on configuring the various components to configure outputs and to filter and route your log messages to these outputs. For a list of available CRDs, see Custom Resource Definitions.