Transport all logs into Amazon S3 with Logging operator

This guide describes how to collect all the container logs in Kubernetes using the Logging operator, and how to send them to Amazon S3.

The following figure gives you an overview about how the system works. The Logging operator collects the logs from the application, selects which logs to forward to the output, and sends the selected log messages to the output. For more details about the Logging operator, see the Logging operator overview.

Deploy the Logging operator

Install the Logging operator.

Deploy the Logging operator with Helm

To install the Logging operator using Helm, complete the following steps.

Note: You need Helm v3.8 or later to be able to install the chart from an OCI registry.

1. Install the Logging operator into the logging namespace:

   helm upgrade --install --wait --create-namespace --namespace logging logging-operator oci://ghcr.io/kube-logging/helm-charts/logging-operator
   

   Expected output:

   Release "logging-operator" does not exist. Installing it now.
   Pulled: ghcr.io/kube-logging/helm-charts/logging-operator:4.3.0
   Digest: sha256:c2ece861f66a3a2cb9788e7ca39a267898bb5629dc98429daa8f88d7acf76840
   NAME: logging-operator
   LAST DEPLOYED: Wed Aug  9 11:02:12 2023
   NAMESPACE: logging
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   

   Note: Helm has a known issue in version 3.13.0 that requires users to log in to the registry, even though the repo is public. Upgrade to 3.13.1 or higher to avoid having to log in, see: https://github.com/kube-logging/logging-operator/issues/1522

2. Validate your deployment.

Configure the Logging operator

1. Create AWS secret

   If you have your $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY set you can use the following snippet.

   kubectl -n logging create secret generic logging-s3 --from-literal "awsAccessKeyId=$AWS_ACCESS_KEY_ID" --from-literal "awsSecretAccessKey=$AWS_SECRET_ACCESS_KEY"
   

   Or set up the secret manually.

       kubectl -n logging apply -f - <<"EOF"
       apiVersion: v1
       kind: Secret
       metadata:
         name: logging-s3
       type: Opaque
       data:
         awsAccessKeyId: <base64encoded>
         awsSecretAccessKey: <base64encoded>
       EOF
   

2. Create the logging resource.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Logging
   metadata:
     name: default-logging-simple
   spec:
     fluentd: {}
     fluentbit: {}
     controlNamespace: logging
   EOF
   

   Note: You can use the ClusterOutput and ClusterFlow resources only in the controlNamespace.

3. Create an S3 output definition.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Output
   metadata:
    name: s3-output
    namespace: logging
   spec:
    s3:
      aws_key_id:
        valueFrom:
          secretKeyRef:
            name: logging-s3
            key: awsAccessKeyId
      aws_sec_key:
        valueFrom:
          secretKeyRef:
            name: logging-s3
            key: awsSecretAccessKey
      s3_bucket: logging-amazon-s3
      s3_region: eu-central-1
      path: logs/${tag}/%Y/%m/%d/
      buffer:
        timekey: 10m
        timekey_wait: 30s
        timekey_use_utc: true
   EOF
   

   Note: In production environment, use a longer timekey interval to avoid generating too many objects.

4. Create a flow resource. (Mind the label selector in the match that selects a set of pods that we will install in the next step)

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Flow
   metadata:
     name: s3-flow
   spec:
     filters:
       - tag_normaliser: {}
     match:
       - select:
           labels:
             app.kubernetes.io/name: log-generator
     localOutputRefs:
       - s3-output
   EOF
   

5. Install log-generator to produce logs with the label app.kubernetes.io/name: log-generator

   helm upgrade --install --wait --create-namespace --namespace logging log-generator oci://ghcr.io/kube-logging/helm-charts/log-generator
   

6. Validate your deployment.

Validate the deployment

Check fluentd logs (errors with AWS credentials should be visible here):

kubectl logs -f -n logging default-logging-simple-fluentd-0 -c fluentd

Fluentd logs were written to the container filesystem up until Logging operator version 4.3, which has been changed to stdout with 4.4. See FluentOutLogrotate why this was changed and how you can re-enable it if needed.

Check the output. The logs will be available in the bucket on a path like:

/logs/default.default-logging-simple-fluentbit-lsdp5.fluent-bit/2019/09/11/201909111432_0.gz

If you don’t get the expected result you can find help in the troubleshooting section.