Single app, one destination
This guide shows you how to collect application and container logs in Kubernetes using the Logging operator.
The Logging operator itself doesn’t store any logs. For demonstration purposes, it can deploy a special workload to the cluster to let you observe the logs flowing through the system.
The Logging operator collects all logs from the cluster, selects the specific logs based on pod labels, and sends the selected log messages to the output. For more details about the Logging operator, see the Logging operator overview.
Note: This example aims to be simple enough to understand the basic capabilities of the operator. For a production ready setup, see Logging infrastructure setup and Operation.
In this tutorial, you will:
- Install the Logging operator on a cluster.
- Configure Logging operator to collect logs from a namespace and send it to a sample output.
- Install a sample application (log-generator) to collect its logs.
- Check the collected logs.
Deploy the Logging operator with Helm
To install the Logging operator using Helm, complete the following steps.
Note: You need Helm v3.8 or later to be able to install the chart from an OCI registry.
This command installs the latest stable Logging operator and an extra workload (service and deployment). This workload is called logging-operator-test-receiver
. It listens on an HTTP port, receives JSON messages, and writes them to the standard output (stdout) so that it is trivial to observe.
Expected output:
After the installation is completed, check that the following pods and services are running:
Expected output:
Expected output:
Configure the Logging operator
-
Create a
Logging
resource to deploy syslog-ng or Fluentd as the central log aggregator and forwarder. You can complete this quick start guide with any of them, but they have different features, so they are not equivalent. For details, see Which log forwarder to use.Run one of the following commands.
Note: The control namespace is where the Logging operator deploys the forwarder’s resources, such as the StatefulSet and the configuration secrets. Usually it’s called
logging
.By default, this namespace is used to define the cluster-wide resources:
SyslogNGClusterOutput
,SyslogNGClusterFlow
,ClusterOutput
, andClusterFlow
. For details, see Configure log routing.Expected output:
-
Create a
FluentbitAgent
resource to collect logs from all containers. No special configuration is required for now.Expected output:
-
Check that the resources were created successfully so far. Run the following command:
You should already see a completed configcheck pod that validates the forwarder’s configuration before the actual statefulset starts. There should also be a running fluentbit instance per node, that already starts to send all logs to the forwarder.
-
Create a namespace (for example:
quickstart
) from which you want to collect the logs from.Expected output:
-
Create a flow and an output resource in the same namespace (
quickstart
). The flow resource routes logs from the namespace to a specific output. In this example, the output is calledhttp
. The flow resources are calledSyslogNGFlow
andFlow
, the output resources areSyslogNGOutput
andOutput
for syslog-ng and Fluentd, respectively.Note: In production environment, use a longer
timekey
interval to avoid generating too many objects.Expected output:
-
Check that the resources were created successfully. Run the following command:
You should see that the logging resource has been created and the flow and output are active.
-
Install the log-generator to produce logs labelled with:
app.kubernetes.io/name: log-generator
Expected output:
The log-generator application starts to create HTTP access logs. Logging operator collects these log messages and sends them to the test-receiver pod defined in the output custom resource.
-
Check that the logs are delivered to the test-receiver pod output. First, run the following command to get the name of the test-receiver pod:
The output should be similar to the following:
The log messages include the usual information of the access logs, and also Kubernetes-specific information like the pod name, labels, and so on.
-
(Optional) If you want to retry this guide with the other log forwarder on the same cluster, run the following command to delete the forwarder-specific resources:
Summary
If you have completed this guide, you have made the following changes to your cluster:
-
Installed the Fluent Bit agent on every node of the cluster to collect the logs and the labels from the node.
-
Installed syslog-ng or Fluentd on the cluster, to receive the logs from the Fluent Bit agents, and filter, parse, and transform them as needed, and to route the incoming logs to an output. To learn more about routing and filtering, see Routing your logs with syslog-ng or Routing your logs with Fluentd match directives. - Created the following resources that configure Logging operator and the components it manages:
Logging
to configure the logging infrastructure, like the details of the Fluent Bit and the syslog-ng or Fluentd deployment. To learn more about configuring the logging infrastructure, see Logging infrastructure setup.SyslogNGOutput
orOutput
to define an HTTP output that receives the collected messages. To learn more, see syslog-ng outputs or Output and ClusterOutput.SyslogNGFlow
orFlow
that processes the incoming messages and routes them to the appropriate output. To learn more, see syslog-ng flows or Flow and ClusterFlow.
-
Installed a simple receiver to act as the destination of the logs, and configured the log forwarder to send the logs from the
quickstart
namespace to this destination. -
Installed a log-generator application to generate sample log messages, and verified that the logs of this application arrive to the output.
Getting Support
If you encounter any problems that the documentation does not address, file an issue or talk to us on Discord or on the CNCF Slack.
Before asking for help, prepare the following information to make troubleshooting faster:
- Logging operator version
- kubernetes version
- helm/chart version (if you installed the Logging operator with helm)
- Logging operator logs
- fluentd configuration
- fluentd logs
- fluentbit configuration
- fluentbit logs
Do not forget to remove any sensitive information (for example, passwords and private keys) before sharing.