Splunk operator with Logging operator

This guide describes how to collect application and container logs in Kubernetes using the Logging operator, and how to send them to Splunk.

Logging operator collects the logs from the application, selects which logs to forward to the output, and sends the selected log messages to the output (in this case, to Splunk). For more details about the Logging operator, see the Logging operator overview.

Deploy Splunk

First, deploy Splunk Standalone in your Kubernetes cluster. The following procedure is based on the Splunk on Kubernetes quickstart.

1. Create the logging Namespace.

   kubectl create ns logging
   

2. Install the Splunk operator.

   kubectl apply -n logging -f https://github.com/splunk/splunk-operator/releases/download/2.4.0/splunk-operator-cluster.yaml
   

3. Install the Splunk cluster

   kubectl apply -n logging -f - <<"EOF"
   apiVersion: enterprise.splunk.com/v4
   kind: Standalone
   metadata:
     name: single
     finalizers:
     - enterprise.splunk.com/delete-pvc
   EOF
   

Deploy the Logging operator and a demo Application

Install the Logging operator and a demo application to provide sample log messages.

Deploy the Logging operator with Helm

To install the Logging operator using Helm, see Deploy the Logging operator with Helm.

1. Create the logging resource.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Logging
   metadata:
     name: default-logging-simple
   spec:
     fluentd: {}
     fluentbit: {}
     controlNamespace: logging
   EOF
   

   Note: You can use the ClusterOutput and ClusterFlow resources only in the controlNamespace.

2. Get a Splunk HEC Token.

   HEC_TOKEN=$(kubectl get secret -n logging  splunk-logging-secret -o jsonpath='{.data.hec_token}' | base64 --decode)
   

3. Create a Splunk output secret from the token.

   kubectl  create secret generic splunk-token -n logging --from-literal "SplunkHecToken=${HEC_TOKEN}"
   

4. Define a Splunk output.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Output
   metadata:
    name: splunk-output
   spec:
    splunkHec:
       hec_host: splunk-single-standalone-headless
       insecure_ssl: true
       hec_port: 8088
       hec_token:
           valueFrom:
              secretKeyRef:
                 name:  splunk-token
                 key: SplunkHecToken
       index: main
       format:
         type: json
   EOF
   

5. Create a flow resource.

   kubectl -n logging apply -f - <<"EOF"
   apiVersion: logging.banzaicloud.io/v1beta1
   kind: Flow
   metadata:
     name: splunk-flow
   spec:
     filters:
       - tag_normaliser: {}
       - parser:
           remove_key_name_field: true
           reserve_data: true
           parse:
             type: nginx
     match:
       - select:
           labels:
             app.kubernetes.io/name: log-generator
     localOutputRefs:
       - splunk-output
   EOF
   

6. Install log-generator to produce logs with the label app.kubernetes.io/name: log-generator

   helm upgrade --install --wait --create-namespace --namespace logging log-generator oci://ghcr.io/kube-logging/helm-charts/log-generator
   

7. Validate your deployment.

Validate the deployment

To validate that the deployment was successful, complete the following steps.

1. Use the following command to retrieve the password of the admin user:

   kubectl -n logging get secret splunk-single-standalone-secrets -o jsonpath='{.data.password}' | base64 --decode
   

2. Enable port forwarding to the Splunk Dashboard Service.

   kubectl -n logging port-forward svc/splunk-single-standalone-headless 8000
   

3. Open the Splunk dashboard in your browser: http://localhost:8000. You should see the dashboard and some sample log messages from the demo application.

If you don’t get the expected result you can find help in the troubleshooting section.