Store Nginx Access Logs in Grafana Loki with Logging operator

This guide describes how to collect application and container logs in Kubernetes using the Logging operator, and how to send them to Grafana Loki.

The following figure gives you an overview about how the system works. The Logging operator collects the logs from the application, selects which logs to forward to the output, and sends the selected log messages to the output. For more details about the Logging operator, see the Logging operator overview.

Deploy Loki and Grafana

1. Add the chart repositories of Loki and Grafana using the following commands:

   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update
   

2. Install Loki into the logging namespace:

   helm upgrade --install --create-namespace --namespace logging loki grafana/loki
   

   Expected output:

   Release "loki" does not exist. Installing it now.
   NAME: loki
   LAST DEPLOYED: Wed Aug  9 10:58:32 2023
   NAMESPACE: logging
   STATUS: deployed
   REVISION: 1
   NOTES:
   ***********************************************************************
   Welcome to Grafana Loki
   Chart version: 5.10.0
   Loki version: 2.8.3
   ***********************************************************************
   
   Installed components:
   * grafana-agent-operator
   * gateway
   * read
   * write
   * backend
   

   For details, see the Grafana Loki Documentation

3. Install Grafana into the logging namespace:

    helm upgrade --install --create-namespace --namespace logging grafana grafana/grafana \
    --set "datasources.datasources\\.yaml.apiVersion=1" \
    --set "datasources.datasources\\.yaml.datasources[0].name=Loki" \
    --set "datasources.datasources\\.yaml.datasources[0].type=loki" \
    --set "datasources.datasources\\.yaml.datasources[0].url=http://loki:3100" \
    --set "datasources.datasources\\.yaml.datasources[0].access=proxy"
   

   Expected output:

   Release "grafana" does not exist. Installing it now.
   NAME: grafana
   LAST DEPLOYED: Wed Aug  9 11:00:47 2023
   NAMESPACE: logging
   STATUS: deployed
   REVISION: 1
   NOTES:
   1. Get your 'admin' user password by running:
   
     kubectl get secret --namespace logging grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
   ...
   

Deploy the Logging operator and a demo application

Install the Logging operator and a demo application to provide sample log messages.

Deploy the Logging operator with Helm

To install the Logging operator using Helm, complete the following

Note: You need Helm v3.8 or later to be able to install the chart from an OCI registry.

1. Install the Logging operator into the logging namespace:

   helm upgrade --install --wait --create-namespace --namespace logging logging-operator oci://ghcr.io/kube-logging/helm-charts/logging-operator
   

   Expected output:

   Release "logging-operator" does not exist. Installing it now.
   Pulled: ghcr.io/kube-logging/helm-charts/logging-operator:4.3.0
   Digest: sha256:c2ece861f66a3a2cb9788e7ca39a267898bb5629dc98429daa8f88d7acf76840
   NAME: logging-operator
   LAST DEPLOYED: Wed Aug  9 11:02:12 2023
   NAMESPACE: logging
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   

   Note: Helm has a known issue in version 3.13.0 that requires users to log in to the registry, even though the repo is public. Upgrade to 3.13.1 or higher to avoid having to log in, see: https://github.com/kube-logging/logging-operator/issues/1522

2. Create the logging resource.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Logging
   metadata:
     name: default-logging-simple
   spec:
     fluentd: {}
     fluentbit: {}
     controlNamespace: logging
   EOF
   

   Note: You can use the ClusterOutput and ClusterFlow resources only in the controlNamespace.

3. Create a Loki output definition.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Output
   metadata:
    name: loki-output
   spec:
    loki:
      url: http://loki:3100
      configure_kubernetes_labels: true
      buffer:
        timekey: 1m
        timekey_wait: 30s
        timekey_use_utc: true
   EOF
   

   Note: In production environment, use a longer timekey interval to avoid generating too many objects.

4. Create a flow resource.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Flow
   metadata:
     name: loki-flow
   spec:
     filters:
       - tag_normaliser: {}
       - parser:
           remove_key_name_field: true
           reserve_data: true
           parse:
             type: nginx
     match:
       - select:
           labels:
             app.kubernetes.io/name: log-generator
     localOutputRefs:
       - loki-output
   EOF
   

5. Install log-generator to produce logs with the label app.kubernetes.io/name: log-generator

   helm upgrade --install --wait --create-namespace --namespace logging log-generator oci://ghcr.io/kube-logging/helm-charts/log-generator
   

6. Validate your deployment.

Validate the deployment

Grafana Dashboard

1. Use the following command to retrieve the password of the Grafana admin user:

   kubectl get secret --namespace logging grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
   

2. Enable port forwarding to the Grafana Service.

   kubectl -n logging port-forward svc/grafana 3000:80
   

3. Open the Grafana Dashboard: http://localhost:3000

4. Use the admin username and the password retrieved in Step 1 to log in.

5. Select Menu > Explore, select Data source > Loki, then select Log labels > namespace > logging. A list of logs should appear.

   

If you don’t get the expected result you can find help in the troubleshooting section.