Concat

Concat Filter

Overview

Fluentd Filter plugin to concatenate multiline log separated in multiple events.

Configuration

Concat

key (string, optional)

Specify field name in the record to parse. If you leave empty the Container Runtime default will be used.

Default: -

separator (*string, optional)

The separator of lines.

Default: “\n”

n_lines (int, optional)

The number of lines. This is exclusive with multiline_start_regex.

Default: -

multiline_start_regexp (string, optional)

The regexp to match beginning of multiline. This is exclusive with n_lines.

Default: -

multiline_end_regexp (string, optional)

The regexp to match ending of multiline. This is exclusive with n_lines.

Default: -

continuous_line_regexp (string, optional)

The regexp to match continuous lines. This is exclusive with n_lines.

Default: -

stream_identity_key (string, optional)

The key to determine which stream an event belongs to.

Default: -

flush_interval (int, optional)

The number of seconds after which the last received event log will be flushed. If specified 0, wait for next line forever.

Default: -

timeout_label (string, optional)

The label name to handle events caused by timeout.

Default: -

use_first_timestamp (bool, optional)

Use timestamp of first record when buffer is flushed.

Default: False

partial_key (string, optional)

The field name that is the reference to concatenate records

Default: -

partial_value (string, optional)

The value stored in the field specified by partial_key that represent partial log

Default: -

keep_partial_key (bool, optional)

If true, keep partial_key in concatenated records

Default: False

use_partial_metadata (string, optional)

Use partial metadata to concatenate multiple records

Default: -

keep_partial_metadata (string, optional)

If true, keep partial metadata

Default: -

partial_metadata_format (string, optional)

Input format of the partial metadata (fluentd or journald docker log driver)( docker-fluentd, docker-journald, docker-journald-lowercase)

Default: -

use_partial_cri_logtag (bool, optional)

Use cri log tag to concatenate multiple records

Default: -

partial_cri_logtag_key (string, optional)

The key name that is referred to concatenate records on cri log

Default: -

partial_cri_stream_key (string, optional)

The key name that is referred to detect stream name on cri log

Default: stream

Example Concat filter configurations

apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: demo-flow
spec:
  filters:
    - concat:
        partial_key: "partial_message"
        separator: ""
        n_lines: 10
  selectors: {}
  localOutputRefs:
    - demo-output

Fluentd config result:

<filter **>
  @type concat
  @id test_concat
  key message
  n_lines 10
  partial_key partial_message
</filter>