Parser
Parser Filter
Overview
Parses a string field in event records and mutates its event record with the parsed result.
Configuration
ParserConfig
key_name (string, optional)
Specify field name in the record to parse. If you leave empty the Container Runtime default will be used.
Default: -
reserve_time (bool, optional)
Keep original event time in parsed result.
Default: -
reserve_data (bool, optional)
Keep original key-value pair in parsed result.
Default: -
remove_key_name_field (bool, optional)
Remove key_name field when parsing is succeeded
Default: -
replace_invalid_sequence (bool, optional)
If true, invalid string is replaced with safe characters and re-parse it.
Default: -
inject_key_prefix (string, optional)
Store parsed values with specified key name prefix.
Default: -
hash_value_field (string, optional)
Store parsed values as a hash value in a field.
Default: -
emit_invalid_record_to_error (*bool, optional)
Emit invalid record to @ERROR label. Invalid cases are: key not exist, format is not matched, unexpected error
Default: -
parse (ParseSection, optional)
Default: -
parsers ([]ParseSection, optional)
Deprecated, use parse instead
Default: -
Parse Section
type (string, optional)
Parse type: apache2, apache_error, nginx, syslog, csv, tsv, ltsv, json, multiline, none, logfmt, grok, multiline_grok
Default: -
expression (string, optional)
Regexp expression to evaluate
Default: -
time_key (string, optional)
Specify time field for event time. If the event doesn’t have this field, current time is used.
Default: -
keys (string, optional)
Names for fields on each line. (seperated by coma)
Default: -
null_value_pattern (string, optional)
Specify null value pattern.
Default: -
null_empty_string (bool, optional)
If true, empty string field is replaced with nil
Default: -
estimate_current_event (bool, optional)
If true, use Fluent::EventTime.now(current time) as a timestamp when time_key is specified.
Default: -
keep_time_key (bool, optional)
If true, keep time field in the record.
Default: -
types (string, optional)
Types casting the fields to proper types example: field1:type, field2:type
Default: -
time_format (string, optional)
Process value using specified format. This is available only when time_type is string
Default: -
time_type (string, optional)
Parse/format value according to this type available values: float, unixtime, string
Default: string
local_time (bool, optional)
Ff true, use local time. Otherwise, UTC is used. This is exclusive with utc.
Default: true
utc (bool, optional)
If true, use UTC. Otherwise, local time is used. This is exclusive with localtime
Default: false
timezone (string, optional)
Use specified timezone. one can parse/format the time value in the specified timezone.
Default: nil
format (string, optional)
Only available when using type: multi_format
Default: -
format_firstline (string, optional)
Only available when using type: multi_format
Default: -
delimiter (string, optional)
Only available when using type: ltsv
Default: “\t”
delimiter_pattern (string, optional)
Only available when using type: ltsv
Default: -
label_delimiter (string, optional)
Only available when using type: ltsv
Default: “:”
multiline ([]string, optional)
The multiline parser plugin parses multiline logs.
Default: -
patterns ([]SingleParseSection, optional)
Only available when using type: multi_format Parse Section
Default: -
grok_pattern (string, optional)
Only available when using type: grok, multiline_grok. The pattern of grok. You cannot specify multiple grok pattern with this.
Default: -
custom_pattern_path (*secret.Secret, optional)
Only available when using type: grok, multiline_grok. File that includes custom grok patterns.
Default: -
grok_failure_key (string, optional)
Only available when using type: grok, multiline_grok. The key has grok failure reason.
Default: -
grok_name_key (string, optional)
Only available when using type: grok, multiline_grok. The key name to store grok section’s name.
Default: -
multiline_start_regexp (string, optional)
Only available when using type: multiline_grok The regexp to match beginning of multiline.
Default: -
grok_patterns ([]GrokSection, optional)
Only available when using type: grok, multiline_grok. Grok Section Specify grok pattern series set.
Default: -
Parse Section (single)
type (string, optional)
Parse type: apache2, apache_error, nginx, syslog, csv, tsv, ltsv, json, multiline, none, logfmt, grok, multiline_grok
Default: -
expression (string, optional)
Regexp expression to evaluate
Default: -
time_key (string, optional)
Specify time field for event time. If the event doesn’t have this field, current time is used.
Default: -
null_value_pattern (string, optional)
Specify null value pattern.
Default: -
null_empty_string (bool, optional)
If true, empty string field is replaced with nil
Default: -
estimate_current_event (bool, optional)
If true, use Fluent::EventTime.now(current time) as a timestamp when time_key is specified.
Default: -
keep_time_key (bool, optional)
If true, keep time field in the record.
Default: -
types (string, optional)
Types casting the fields to proper types example: field1:type, field2:type
Default: -
time_format (string, optional)
Process value using specified format. This is available only when time_type is string
Default: -
time_type (string, optional)
Parse/format value according to this type available values: float, unixtime, string
Default: string
local_time (bool, optional)
Ff true, use local time. Otherwise, UTC is used. This is exclusive with utc.
Default: true
utc (bool, optional)
If true, use UTC. Otherwise, local time is used. This is exclusive with localtime
Default: false
timezone (string, optional)
Use specified timezone. one can parse/format the time value in the specified timezone.
Default: nil
format (string, optional)
Only available when using type: multi_format
Default: -
grok_pattern (string, optional)
Only available when using format: grok, multiline_grok. The pattern of grok. You cannot specify multiple grok pattern with this.
Default: -
custom_pattern_path (*secret.Secret, optional)
Only available when using format: grok, multiline_grok. File that includes custom grok patterns.
Default: -
grok_failure_key (string, optional)
Only available when using format: grok, multiline_grok. The key has grok failure reason.
Default: -
grok_name_key (string, optional)
Only available when using format: grok, multiline_grok. The key name to store grok section’s name.
Default: -
multiline_start_regexp (string, optional)
Only available when using format: multiline_grok The regexp to match beginning of multiline.
Default: -
grok_patterns ([]GrokSection, optional)
Only available when using format: grok, multiline_grok. Grok Section Specify grok pattern series set.
Default: -
Grok Section
name (string, optional)
The name of grok section.
Default: -
pattern (string, required)
The pattern of grok.
Default: -
keep_time_key (bool, optional)
If true, keep time field in the record.
Default: -
time_key (string, optional)
Specify time field for event time. If the event doesn’t have this field, current time is used.
Default: time
time_format (string, optional)
Process value using specified format. This is available only when time_type is string.
Default: -
timezone (string, optional)
Use specified timezone. one can parse/format the time value in the specified timezone.
Default: -
Example Parser filter configurations
apiVersion: logging.banzaicloud.io/v1beta1
kind: Flow
metadata:
name: demo-flow
spec:
filters:
- parser:
remove_key_name_field: true
reserve_data: true
parse:
type: multi_format
patterns:
- format: nginx
- format: regexp
expression: /foo/
- format: none
selectors: {}
localOutputRefs:
- demo-output
Fluentd config result:
<filter **>
@type parser
@id test_parser
key_name message
remove_key_name_field true
reserve_data true
<parse>
@type multi_format
<pattern>
format nginx
</pattern>
<pattern>
expression /foo/
format regexp
</pattern>
<pattern>
format none
</pattern>
</parse>
</filter>