Sumo Logic with Logging operator and syslog-ng

This guide helps you install and configure the Logging operator and syslog-ng to forward logs to your Sumo Logic account.

Prerequisites

We assume that you already have:

• A Sumo Logic account.

• A HTTP Hosted Collector configured in the Sumo Logic service.

  To configure a Hosted Collector, complete the steps in the Configure a Hosted Collector section on the official Sumo Logic website.

• The unique HTTP collector code you receive while configuring your Host Collector for HTTP requests.

Deploy the Logging operator and a demo Application

Install the Logging operator and a demo application to provide sample log messages.

Deploy the Logging operator with Helm

To install the Logging operator using Helm, complete the following steps.

Note: You need Helm v3.8 or later to be able to install the chart from an OCI registry.

1. Install the Logging operator into the logging namespace:

   helm upgrade --install --wait --create-namespace --namespace logging logging-operator oci://ghcr.io/kube-logging/helm-charts/logging-operator
   

   Expected output:

   Release "logging-operator" does not exist. Installing it now.
   Pulled: ghcr.io/kube-logging/helm-charts/logging-operator:4.3.0
   Digest: sha256:c2ece861f66a3a2cb9788e7ca39a267898bb5629dc98429daa8f88d7acf76840
   NAME: logging-operator
   LAST DEPLOYED: Wed Aug  9 11:02:12 2023
   NAMESPACE: logging
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None
   

   Note:

   • Helm has a known issue in version 3.13.0 that requires users to log in to the registry, even though the repo is public.

     Upgrade to 3.13.1 or higher to avoid having to log in, see: https://github.com/kube-logging/logging-operator/issues/1522

   • If you’re installing the Helm chart from Terraform, reference the repository as repository = "oci://ghcr.io/kube-logging/helm-charts/" (without the logging-operator suffix). Otherwise, you’ll get a 403 Forbidden error.

Configure the Logging operator

1. Create the logging resource with a persistent syslog-ng installation.

   kubectl apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Logging
   metadata:
     name: demo
   spec:
     controlNamespace: logging
     fluentbit: {}
     syslogNG:
       statefulSet:
         spec:
           template:
             spec:
               containers:
               - name: syslog-ng
                 volumeMounts:
                 - mountPath: /buffers
                   name: buffer
           volumeClaimTemplates:
           - metadata:
               name: buffer
             spec:
               accessModes:
               - ReadWriteOnce
               resources:
                 requests:
                   storage: 10Gi
   EOF
   

   Note: You can use the ClusterOutput and ClusterFlow resources only in the controlNamespace.

2. Create a Sumo Logic output secret from the URL of your Sumo Logic collection.

   kubectl create secret generic sumo-collector -n logging --from-literal "token=XYZ"
   

3. Create a SyslogNGOutput resource.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: SyslogNGOutput
   metadata:
     name: sumologic-syslog-ng-output
   spec:
     sumologic-http: 
       collector:
         valueFrom:
           secretKeyRef:
             key: token
             name: sumo-collector
       deployment: us2
       batch-lines: 1000
       disk_buffer:
         disk_buf_size: 512000000
         dir: /buffers
         reliable: true
       body: "$(format-json --subkeys json. --exclude json.kubernetes.annotations.* json.kubernetes.annotations=literal($(format-flat-json --subkeys json.kubernetes.annotations.)) --exclude json.kubernetes.labels.* json.kubernetes.labels=literal($(format-flat-json --subkeys json.kubernetes.labels.)))"
       headers:
         - 'X-Sumo-Name: source-name'
         - 'X-Sumo-Category: source-category'
       tls:
         use-system-cert-store: true
   EOF
   

4. Create a SyslogNGFlow resource.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: SyslogNGFlow
   metadata:
     name: log-generator
   spec:
     match:
       and:
       - regexp:
           value: json.kubernetes.labels.app.kubernetes.io/instance
           pattern: log-generator
           type: string
       - regexp:
           value:  json.kubernetes.labels.app.kubernetes.io/name
           pattern: log-generator
           type: string
     filters:
     -  parser:
         regexp: 
           patterns:
           - '^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)"(?:\s+(?<http_x_forwarded_for>[^ ]+))?)?$'
           template: ${json.message}
           prefix: json.
     - rewrite:
       -  set:
           field: json.cluster
           value: xxxxx
       -  unset:
           field: json.message
       -  set:
           field: json.source
           value: /var/log/log-generator
           condition:
             regexp:
               value:  json.kubernetes.container_name
               pattern: log-generator
               type: string
     localOutputRefs:
       - sumologic-syslog-ng-output
   EOF
   

5. Install log-generator to produce logs with the label app.kubernetes.io/name: log-generator

   helm upgrade --install --wait --create-namespace --namespace logging log-generator oci://ghcr.io/kube-logging/helm-charts/log-generator
   

If you don’t get the expected result you can find help in the troubleshooting section.