Parser

Parser Filter

Overview

Parses a string field in event records and mutates its event record with the parsed result.

Configuration

ParserConfig

emit_invalid_record_to_error (*bool, optional)

Emit invalid record to @ERROR label. Invalid cases are: key not exist, format is not matched, unexpected error

hash_value_field (string, optional)

Store parsed values as a hash value in a field.

inject_key_prefix (string, optional)

Store parsed values with specified key name prefix.

key_name (string, optional)

Specify field name in the record to parse. If you leave empty the Container Runtime default will be used.

parse (ParseSection, optional)

Parse Section

parsers ([]ParseSection, optional)

Deprecated, use parse instead

remove_key_name_field (bool, optional)

Remove key_name field when parsing is succeeded

replace_invalid_sequence (bool, optional)

If true, invalid string is replaced with safe characters and re-parse it.

reserve_data (bool, optional)

Keep original key-value pair in parsed result.

reserve_time (bool, optional)

Keep original event time in parsed result.

Parse Section

custom_pattern_path (*secret.Secret, optional)

Only available when using type: grok, multiline_grok. File that includes custom grok patterns.

delimiter (string, optional)

Only available when using type: ltsv

Default: “\t”

delimiter_pattern (string, optional)

Only available when using type: ltsv

estimate_current_event (bool, optional)

If true, use Fluent::EventTime.now(current time) as a timestamp when time_key is specified.

expression (string, optional)

Regexp expression to evaluate

format (string, optional)

Only available when using type: multi_format

format_firstline (string, optional)

Only available when using type: multi_format

grok_failure_key (string, optional)

Only available when using type: grok, multiline_grok. The key has grok failure reason.

grok_name_key (string, optional)

Only available when using type: grok, multiline_grok. The key name to store grok section’s name.

grok_pattern (string, optional)

Only available when using type: grok, multiline_grok. The pattern of grok. You cannot specify multiple grok pattern with this.

grok_patterns ([]GrokSection, optional)

Only available when using type: grok, multiline_grok. Grok Section Specify grok pattern series set.

keep_time_key (bool, optional)

If true, keep time field in the record.

keys (string, optional)

Names for fields on each line. (seperated by coma)

label_delimiter (string, optional)

Only available when using type: ltsv

Default: “:”

local_time (bool, optional)

If true, use local time. Otherwise, UTC is used. This is exclusive with utc.

Default: true

multiline ([]string, optional)

The multiline parser plugin parses multiline logs.

multiline_start_regexp (string, optional)

Only available when using type: multiline_grok The regexp to match beginning of multiline.

null_empty_string (bool, optional)

If true, empty string field is replaced with nil

null_value_pattern (string, optional)

Specify null value pattern.

patterns ([]SingleParseSection, optional)

Only available when using type: multi_format Parse Section

time_format (string, optional)

Process value using specified format. This is available only when time_type is string

time_key (string, optional)

Specify time field for event time. If the event doesn’t have this field, current time is used.

time_type (string, optional)

Parse/format value according to this type available values: float, unixtime, string

Default: string

timezone (string, optional)

Use specified timezone. one can parse/format the time value in the specified timezone.

Default: nil

type (string, optional)

Parse type: apache2, apache_error, nginx, syslog, csv, tsv, ltsv, json, multiline, none, logfmt, grok, multiline_grok

types (string, optional)

Types casting the fields to proper types example: field1:type, field2:type

utc (bool, optional)

If true, use UTC. Otherwise, local time is used. This is exclusive with localtime

Default: false

Parse Section (single)

custom_pattern_path (*secret.Secret, optional)

Only available when using format: grok, multiline_grok. File that includes custom grok patterns.

estimate_current_event (bool, optional)

If true, use Fluent::EventTime.now(current time) as a timestamp when time_key is specified.

expression (string, optional)

Regexp expression to evaluate

format (string, optional)

Only available when using type: multi_format

grok_failure_key (string, optional)

Only available when using format: grok, multiline_grok. The key has grok failure reason.

grok_name_key (string, optional)

Only available when using format: grok, multiline_grok. The key name to store grok section’s name.

grok_pattern (string, optional)

Only available when using format: grok, multiline_grok. The pattern of grok. You cannot specify multiple grok pattern with this.

grok_patterns ([]GrokSection, optional)

Only available when using format: grok, multiline_grok. Grok Section Specify grok pattern series set.

keep_time_key (bool, optional)

If true, keep time field in the record.

local_time (bool, optional)

If true, use local time. Otherwise, UTC is used. This is exclusive with utc.

Default: true

multiline_start_regexp (string, optional)

Only available when using format: multiline_grok The regexp to match beginning of multiline.

null_empty_string (bool, optional)

If true, empty string field is replaced with nil

null_value_pattern (string, optional)

Specify null value pattern.

time_format (string, optional)

Process value using specified format. This is available only when time_type is string

time_key (string, optional)

Specify time field for event time. If the event doesn’t have this field, current time is used.

time_type (string, optional)

Parse/format value according to this type available values: float, unixtime, string

Default: string

timezone (string, optional)

Use specified timezone. one can parse/format the time value in the specified timezone.

Default: nil

type (string, optional)

Parse type: apache2, apache_error, nginx, syslog, csv, tsv, ltsv, json, multiline, none, logfmt, grok, multiline_grok

types (string, optional)

Types casting the fields to proper types example: field1:type, field2:type

utc (bool, optional)

If true, use UTC. Otherwise, local time is used. This is exclusive with localtime

Default: false

Grok Section

keep_time_key (bool, optional)

If true, keep time field in the record.

name (string, optional)

The name of grok section.

pattern (string, required)

The pattern of grok.

time_format (string, optional)

Process value using specified format. This is available only when time_type is string.

time_key (string, optional)

Specify time field for event time. If the event doesn’t have this field, current time is used.

Default: time

timezone (string, optional)

Use specified timezone. one can parse/format the time value in the specified timezone.

Example Parser filter configurations

apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
  name: demo-flow
spec:
  filters:
    - parser:
        remove_key_name_field: true
        reserve_data: true
        parse:
          type: multi_format
          patterns:
          - format: nginx
          - format: regexp
            expression: /foo/
          - format: none
  selectors: {}
  localOutputRefs:
    - demo-output

Fluentd config result:

<filter **>
  @type parser
  @id test_parser
  key_name message
  remove_key_name_field true
  reserve_data true
  <parse>
    @type multi_format
    <pattern>
      format nginx
    </pattern>
    <pattern>
      expression /foo/
      format regexp
    </pattern>
    <pattern>
      format none
    </pattern>
  </parse>
</filter>