Single app, one destination

This guide shows you how to collect application and container logs in Kubernetes using the Logging operator.

The Logging operator itself doesn’t store any logs. For demonstration purposes, it can deploy a special workload to the cluster to let you observe the logs flowing through the system.

The Logging operator collects all logs from the cluster, selects the specific logs based on pod labels, and sends the selected log messages to the output. For more details about the Logging operator, see the Logging operator overview.

Note: This example aims to be simple enough to understand the basic capabilities of the operator. For a production ready setup, see Logging infrastructure setup and Operation.

In this tutorial, you will:

  • Install the Logging operator on a cluster.
  • Configure Logging operator to collect logs from a namespace and send it to a sample output.
  • Install a sample application (log-generator) to collect its logs.
  • Check the collected logs.

Deploy the Logging operator with Helm

To install the Logging operator using Helm, complete the following steps.

Note: You need Helm v3.8 or later to be able to install the chart from an OCI registry.

This command installs the latest stable Logging operator and an extra workload (service and deployment). This workload is called logging-operator-test-receiver. It listens on an HTTP port, receives JSON messages, and writes them to the standard output (stdout) so that it is trivial to observe.

helm upgrade --install --wait \
     --create-namespace --namespace logging \
     --set testReceiver.enabled=true \
     logging-operator oci://ghcr.io/kube-logging/helm-charts/logging-operator

Expected output:

Release "logging-operator" does not exist. Installing it now.
Pulled: ghcr.io/kube-logging/helm-charts/logging-operator:4.3.0
Digest: sha256:c2ece861f66a3a2cb9788e7ca39a267898bb5629dc98429daa8f88d7acf76840
NAME: logging-operator
LAST DEPLOYED: Tue Aug 15 15:58:41 2023
NAMESPACE: logging
STATUS: deployed
REVISION: 1
TEST SUITE: None

After the installation is completed, check that the following pods and services are running:

kubectl get deploy -n logging

Expected output:

NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
logging-operator                 1/1     1            1           15m
logging-operator-test-receiver   1/1     1            1           15m
kubectl get svc -n logging

Expected output:

NAME                             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
logging-operator                 ClusterIP   None           <none>        8080/TCP   15m
logging-operator-test-receiver   ClusterIP   10.99.77.113   <none>        8080/TCP   15m

Configure the Logging operator

  1. Create a Logging resource to deploy syslog-ng or Fluentd as the central log aggregator and forwarder. You can complete this quick start guide with any of them, but they have different features, so they are not equivalent. For details, see Which log forwarder to use.

    Run one of the following commands.

    kubectl --namespace logging apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Logging
    metadata:
      name: quickstart
    spec:
      controlNamespace: logging
      syslogNG:
        # `#` is the recommended key delimiter when parsing json in syslog-ng
        jsonKeyDelim: '#'
    EOF
    
    kubectl --namespace logging apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Logging
    metadata:
      name: quickstart
    spec:
      controlNamespace: logging
      fluentd:
        disablePvc: true
    EOF
    

    Note: The control namespace is where the Logging operator deploys the forwarder’s resources, such as the StatefulSet and the configuration secrets. Usually it’s called logging.

    By default, this namespace is used to define the cluster-wide resources: SyslogNGClusterOutput, SyslogNGClusterFlow, ClusterOutput, and ClusterFlow. For details, see Configure log routing.

    Expected output:

    logging.logging.banzaicloud.io/quickstart created
    
  2. Create a FluentbitAgent resource to collect logs from all containers. No special configuration is required for now.

    kubectl --namespace logging apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: FluentbitAgent
    metadata:
        name: quickstart
    spec: {}
    EOF
    

    Expected output:

    fluentbitagent.logging.banzaicloud.io/quickstart created
    
  3. Check that the resources were created successfully so far. Run the following command:

    kubectl get pod --namespace logging --selector app.kubernetes.io/managed-by=quickstart
    

    You should already see a completed configcheck pod that validates the forwarder’s configuration before the actual statefulset starts. There should also be a running fluentbit instance per node, that already starts to send all logs to the forwarder.

    NAME                                        READY   STATUS      RESTARTS   AGE
    quickstart-fluentbit-jvdp5                  1/1     Running     0          3m5s
    quickstart-syslog-ng-0                      2/2     Running     0          3m5s
    quickstart-syslog-ng-configcheck-8197c552   0/1     Completed   0          3m42s
    
    NAME                                      READY   STATUS      RESTARTS   AGE
    quickstart-fluentbit-nk9ms                1/1     Running     0          19s
    quickstart-fluentd-0                      2/2     Running     0          19s
    quickstart-fluentd-configcheck-ac2d4553   0/1     Completed   0          60s
    
  4. Create a namespace (for example: quickstart) from which you want to collect the logs from.

    kubectl create namespace quickstart
    

    Expected output:

    namespace/quickstart created
    
  5. Create a flow and an output resource in the same namespace (quickstart). The flow resource routes logs from the namespace to a specific output. In this example, the output is called http. The flow resources are called SyslogNGFlow and Flow, the output resources are SyslogNGOutput and Output for syslog-ng and Fluentd, respectively.

    kubectl --namespace quickstart apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: SyslogNGFlow
    metadata:
      name: log-generator
    spec:
      match:
        regexp:
          value: "json#kubernetes#labels#app.kubernetes.io/instance"
          pattern: log-generator
          type: string
      localOutputRefs:
        - http
    ---
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: SyslogNGOutput
    metadata:
      name: http
    spec:
      http:
        url: http://logging-operator-test-receiver:8080
        headers:
          - "Content-Type: application/json"
        disk_buffer:
          dir: /buffers
          disk_buf_size: 512000000 # 512 MB
          reliable: true
    EOF
    
    kubectl --namespace quickstart apply -f - <<"EOF"
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Flow
    metadata:
      name: log-generator
    spec:
      match:
        - select:
            labels:
              app.kubernetes.io/name: log-generator
      localOutputRefs:
        - http
    ---
    apiVersion: logging.banzaicloud.io/v1beta1
    kind: Output
    metadata:
      name: http
    spec:
      http:
        endpoint: http://logging-operator-test-receiver:8080
        content_type: application/json
        buffer:
          type: memory
          tags: time
          timekey: 1s
          timekey_wait: 0s
    EOF
    

    Note: In production environment, use a longer timekey interval to avoid generating too many objects.

    Expected output:

    syslogngflow.logging.banzaicloud.io/log-generator created
    syslogngoutput.logging.banzaicloud.io/http created
    
    flow.logging.banzaicloud.io/log-generator created
    output.logging.banzaicloud.io/http created
    
  6. Check that the resources were created successfully. Run the following command:

    kubectl get logging-all --namespace quickstart
    

    You should see that the logging resource has been created and the flow and output are active.

    NAME                                               AGE
    fluentbitagent.logging.banzaicloud.io/quickstart   10m
    
    NAME                                        AGE
    logging.logging.banzaicloud.io/quickstart   10m
    
    NAME                                         ACTIVE   PROBLEMS
    syslogngoutput.logging.banzaicloud.io/http   true
    
    NAME                                                ACTIVE   PROBLEMS
    syslogngflow.logging.banzaicloud.io/log-generator   true
    
    NAME                                        ACTIVE   PROBLEMS
    flow.logging.banzaicloud.io/log-generator   true
    
    NAME                                 ACTIVE   PROBLEMS
    output.logging.banzaicloud.io/http   true
    
    NAME                                        AGE
    logging.logging.banzaicloud.io/quickstart   3m12s
    
    NAME                                               AGE
    fluentbitagent.logging.banzaicloud.io/quickstart   3m2s
    
  7. Install the log-generator to produce logs labelled with: app.kubernetes.io/name: log-generator

    helm upgrade --install --wait --namespace quickstart log-generator oci://ghcr.io/kube-logging/helm-charts/log-generator
    

    Expected output:

    Release "log-generator" does not exist. Installing it now.
    Pulled: ghcr.io/kube-logging/helm-charts/log-generator:0.7.0
    Digest: sha256:0eba2c5c3adfc33deeec1d1612839cd1a0aa86f30022672ee022beab22436e04
    NAME: log-generator
    LAST DEPLOYED: Tue Aug 15 16:21:40 2023
    NAMESPACE: quickstart
    STATUS: deployed
    REVISION: 1
    TEST SUITE: None
    

    The log-generator application starts to create HTTP access logs. Logging operator collects these log messages and sends them to the test-receiver pod defined in the output custom resource.

  8. Check that the logs are delivered to the test-receiver pod output. First, run the following command to get the name of the test-receiver pod:

    kubectl logs --namespace logging -f svc/logging-operator-test-receiver
    

    The output should be similar to the following:

    [0] http.0: [[1692117678.581721054, {}], {"ts"=>"2023-08-15T16:41:18.130862Z", "time"=>"2023-08-15T16:41:18.13086297Z", "stream"=>"stdout", "log"=>"142.251.196.69 - - [15/Aug/2023:16:41:18 +0000] "PUT /index.html HTTP/1.1" 302 24666 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" "-"", "kubernetes"=>{"pod_name"=>"log-generator-56b7dfb79-6v67b", "pod_id"=>"b7e8a5b2-9164-46d1-ba0a-8d142bdfb4cb", "namespace_name"=>"quickstart", "labels"=>{"pod-template-hash"=>"56b7dfb79", "app.kubernetes.io/name"=>"log-generator", "app.kubernetes.io/instance"=>"log-generator"}, "host"=>"minikube", "docker_id"=>"fe60b1c0fdf97f062ed91e3a2074caf3ee3cb4f3d12844f2c6f5d8212419907d", "container_name"=>"log-generator", "container_image"=>"ghcr.io/kube-logging/log-generator:0.7.0", "container_hash"=>"ghcr.io/kube-logging/log-generator@sha256:e26102ef2d28201240fa6825e39efdf90dec0da9fa6b5aea6cf9113c0d3e93aa"}}]
    
    [0] http.0: [[1692118483.267342676, {}], {"log"=>"51.196.131.145 - - [15/Aug/2023:16:54:36 +0000] "PUT / HTTP/1.1" 200 7823 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" "-"", "stream"=>"stdout", "time"=>"2023-08-15T16:54:36.019636047Z", "kubernetes"=>{"pod_name"=>"log-generator-56b7dfb79-rrzsz", "namespace_name"=>"quickstart", "pod_id"=>"902dc881-af36-4054-b377-47e2d751e6cd", "labels"=>{"app.kubernetes.io/instance"=>"log-generator", "app.kubernetes.io/name"=>"log-generator", "pod-template-hash"=>"56b7dfb79"}, "host"=>"minikube", "container_name"=>"log-generator", "docker_id"=>"7615c4c72d8fdd05137dc9845204d7ef681b750b6f2a6d27bd75190b12dc5d8e", "container_hash"=>"ghcr.io/kube-logging/log-generator@sha256:e26102ef2d28201240fa6825e39efdf90dec0da9fa6b5aea6cf9113c0d3e93aa", "container_image"=>"ghcr.io/kube-logging/log-generator:0.7.0"}}]
    

    The log messages include the usual information of the access logs, and also Kubernetes-specific information like the pod name, labels, and so on.

  9. (Optional) If you want to retry this guide with the other log forwarder on the same cluster, run the following command to delete the forwarder-specific resources:

    kubectl delete logging quickstart
    kubectl delete --namespace quickstart syslogngflow log-generator
    kubectl delete --namespace quickstart syslogngoutput http
    
    kubectl delete logging quickstart
    kubectl delete --namespace quickstart flow log-generator
    kubectl delete --namespace quickstart output http
    

Summary

If you have completed this guide, you have made the following changes to your cluster:

  • Installed the Fluent Bit agent on every node of the cluster to collect the logs and the labels from the node.

  • Installed syslog-ng or Fluentd on the cluster, to receive the logs from the Fluent Bit agents, and filter, parse, and transform them as needed, and to route the incoming logs to an output. To learn more about routing and filtering, see Routing your logs with syslog-ng or Routing your logs with Fluentd match directives. - Created the following resources that configure Logging operator and the components it manages:

    • Logging to configure the logging infrastructure, like the details of the Fluent Bit and the syslog-ng or Fluentd deployment. To learn more about configuring the logging infrastructure, see Logging infrastructure setup.
    • SyslogNGOutput or Output to define an HTTP output that receives the collected messages. To learn more, see syslog-ng outputs or Output and ClusterOutput.
    • SyslogNGFlow or Flow that processes the incoming messages and routes them to the appropriate output. To learn more, see syslog-ng flows or Flow and ClusterFlow.
  • Installed a simple receiver to act as the destination of the logs, and configured the log forwarder to send the logs from the quickstart namespace to this destination.

  • Installed a log-generator application to generate sample log messages, and verified that the logs of this application arrive to the output.

Getting Support

If you encounter any problems that the documentation does not address, file an issue or talk to us on Discord or on the CNCF Slack.

Before asking for help, prepare the following information to make troubleshooting faster:

Do not forget to remove any sensitive information (for example, passwords and private keys) before sharing.