Store Nginx Access Logs in Amazon CloudWatch with Logging Operator

This guide describes how to collect application and container logs in Kubernetes using the Logging operator, and how to send them to CloudWatch.

The following figure gives you an overview about how the system works. The Logging operator collects the logs from the application, selects which logs to forward to the output, and sends the selected log messages to the output. For more details about the Logging operator, see the Logging operator overview.

Deploy the Logging operator and a demo Application

Install the Logging operator and a demo application using Helm.

Deploy the Logging operator with Helm

To install the Logging operator using Helm, complete these steps.

Note: For the Helm-based installation you need Helm v3.2.1 or later.

1. Add the chart repository of the Logging operator using the following commands:

   helm repo add kube-logging https://kube-logging.dev/helm-charts
   helm repo update
   

2. Install the Logging operator.

   helm upgrade --install --wait --create-namespace --namespace logging logging-operator kube-logging/logging-operator
   

3. Create AWS secret

   If you have your $AWS_ACCESS_KEY_ID and $AWS_SECRET_ACCESS_KEY set you can use the following snippet.

       kubectl -n logging create secret generic logging-cloudwatch --from-literal "awsAccessKeyId=$AWS_ACCESS_KEY_ID" --from-literal "awsSecretAccessKey=$AWS_SECRET_ACCESS_KEY"
   

   Or set up the secret manually.

       kubectl -n logging apply -f - <<"EOF"
       apiVersion: v1
       kind: Secret
       metadata:
         name: logging-cloudwatch
       type: Opaque
       data:
         awsAccessKeyId: <base64encoded>
         awsSecretAccessKey: <base64encoded>
       EOF
   

4. Create the logging resource.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Logging
   metadata:
     name: default-logging-simple
   spec:
     fluentd: {}
     fluentbit: {}
     controlNamespace: logging
   EOF
   

   Note: You can use the ClusterOutput and ClusterFlow resources only in the controlNamespace.

5. Create an CloudWatch output definition.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Output
   metadata:
    name: cloudwatch-output
    namespace: logging
   spec:
    cloudwatch:
      aws_key_id:
        valueFrom:
          secretKeyRef:
            name: logging-cloudwatch
            key: awsAccessKeyId
      aws_sec_key:
        valueFrom:
          secretKeyRef:
            name: logging-cloudwatch
            key: awsSecretAccessKey
      log_group_name: operator-log-group
      log_stream_name: operator-log-stream
      region: us-east-1
      auto_create_stream: true
      buffer:
        timekey: 30s
        timekey_wait: 30s
        timekey_use_utc: true
   EOF
   

   Note: In production environment, use a longer timekey interval to avoid generating too many objects.

6. Create a flow resource.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Flow
   metadata:
     name: cloudwatch-flow
   spec:
     filters:
       - tag_normaliser: {}
       - parser:
           remove_key_name_field: true
           reserve_data: true
           parse:
             type: nginx
     match:
       - select:
           labels:
             app.kubernetes.io/name: log-generator
     localOutputRefs:
       - cloudwatch-output
   EOF
   

7. Install log-generator to produce logs with the label app.kubernetes.io/name: log-generator

   helm upgrade --install --wait --create-namespace --namespace logging log-generator kube-logging/log-generator
   

8. Validate your deployment.

Validate the deployment

If you don’t get the expected result you can find help in the troubleshooting section.