Amazon S3

Amazon S3 plugin for Fluentd

Overview

The s3 output plugin buffers event logs in local file and upload it to S3 periodically. This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log ‘2011-01-02 message B’ is reached, and then another log ‘2011-01-03 message B’ is reached in this order, the former one is stored in “20110102.gz” file, and latter one in “20110103.gz” file.

For a detailed example, see S3 Output Deployment.

Example output configurations

spec:
  s3:
    aws_key_id:
      valueFrom:
        secretKeyRef:
          name: logging-s3
          key: awsAccessKeyId
    aws_sec_key:
      valueFrom:
        secretKeyRef:
          name: logging-s3
          key: awsSecretAccessKey
    s3_bucket: logging-amazon-s3
    s3_region: eu-central-1
    path: logs/${tag}/%Y/%m/%d/
    buffer:
      timekey: 10m
      timekey_wait: 30s
      timekey_use_utc: true

Configuration

Output Config

acl (string, optional)

Permission for the object in S3

assume_role_credentials (*S3AssumeRoleCredentials, optional)

Assume Role Credentials

auto_create_bucket (string, optional)

Create S3 bucket if it does not exists

aws_key_id (*secret.Secret, optional) {#output config-aws_key_id}

AWS access key id Secret

aws_iam_retries (string, optional)

The number of attempts to load instance profile credentials from the EC2 metadata service using IAM role

aws_sec_key (*secret.Secret, optional)

AWS secret key. Secret

buffer (*Buffer, optional)

Buffer

check_apikey_on_start (string, optional)

Check AWS key on start

check_bucket (string, optional)

Check bucket if exists or not

check_object (string, optional)

Check object before creation

clustername (string, optional)

Custom cluster name

Default: one-eye

compress (*Compress, optional)

Parquet compressor

compute_checksums (string, optional)

AWS SDK uses MD5 for API request/response by default

enable_transfer_acceleration (string, optional)

If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket

force_path_style (string, optional)

If true, the bucket name is always left in the request URI and never moved to the host as a sub-domain

format (*Format, optional)

Format

grant_full_control (string, optional)

Allows grantee READ, READ_ACP, and WRITE_ACP permissions on the object

grant_read (string, optional)

Allows grantee to read the object data and its metadata

grant_read_acp (string, optional)

Allows grantee to read the object ACL

grant_write_acp (string, optional)

Allows grantee to write the ACL for the applicable object

hex_random_length (string, optional)

The length of %{hex_random} placeholder(4-16)

index_format (string, optional)

sprintf format for %{index}

instance_profile_credentials (*S3InstanceProfileCredentials, optional)

Instance Profile Credentials

oneeye_format (bool, optional)

One-eye format trigger

Default: false

overwrite (string, optional)

Overwrite already existing path

path (string, optional)

Path prefix of the files on S3

proxy_uri (string, optional)

URI of proxy environment

s3_bucket (string, required)

S3 bucket name

s3_endpoint (string, optional)

Custom S3 endpoint (like minio)

s3_metadata (string, optional)

Arbitrary S3 metadata headers to set for the object

s3_object_key_format (string, optional)

The format of S3 object keys (default: %{path}%{time_slice}_%{uuid_hash}_%{index}.%{file_extension})

Default: %{path}%{time_slice}%{uuid_hash}%{index}.%{file_extension}

s3_region (string, optional)

S3 region name

shared_credentials (*S3SharedCredentials, optional)

Shared Credentials

signature_version (string, optional)

Signature version for API Request (s3,v4)

slow_flush_log_threshold (string, optional)

The threshold for chunk flush performance check. Parameter type is float, not time, default: 20.0 (seconds) If chunk flush takes longer time than this threshold, fluentd logs warning message and increases metric fluentd_output_status_slow_flush_count.

sse_customer_algorithm (string, optional)

Specifies the algorithm to use to when encrypting the object

sse_customer_key (string, optional)

Specifies the customer-provided encryption key for Amazon S3 to use in encrypting data

sse_customer_key_md5 (string, optional)

Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321

ssekms_key_id (string, optional) {#output config-ssekms_key_id}

Specifies the AWS KMS key ID to use for object encryption

ssl_verify_peer (string, optional) {#output config-ssl_verify_peer}

If false, the certificate of endpoint will not be verified

storage_class (string, optional)

The type of storage to use for the object, for example STANDARD, REDUCED_REDUNDANCY, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, OUTPOSTS, GLACIER_IR For a complete list of possible values, see the Amazon S3 API reference.

store_as (string, optional)

Archive format on S3

use_bundled_cert (string, optional)

Use aws-sdk-ruby bundled cert

use_server_side_encryption (string, optional)

The Server-side encryption algorithm used when storing this object in S3 (AES256, aws:kms)

warn_for_delay (string, optional)

Given a threshold to treat events as delay, output warning logs if delayed events were put into s3

Assume Role Credentials

assume_role_credentials

duration_seconds (string, optional) {#assume role-credentials-duration_seconds}

The duration, in seconds, of the role session (900-3600)

external_id (string, optional) {#assume role-credentials-external_id}

A unique identifier that is used by third parties when assuming roles in their customers’ accounts.

policy (string, optional) {#assume role-credentials-policy}

An IAM policy in JSON format

role_arn (string, required) {#assume role-credentials-role_arn}

The Amazon Resource Name (ARN) of the role to assume

role_session_name (string, required) {#assume role-credentials-role_session_name}

An identifier for the assumed role session

Instance Profile Credentials

instance_profile_credentials

http_open_timeout (string, optional) {#instance profile-credentials-http_open_timeout}

Number of seconds to wait for the connection to open

http_read_timeout (string, optional) {#instance profile-credentials-http_read_timeout}

Number of seconds to wait for one block to be read

ip_address (string, optional) {#instance profile-credentials-ip_address}

IP address

Default: 169.254.169.254

port (string, optional) {#instance profile-credentials-port}

Port number

Default: 80

retries (string, optional) {#instance profile-credentials-retries}

Number of times to retry when retrieving credentials

Shared Credentials

shared_credentials

path (string, optional)

Path to the shared file.

Default: $HOME/.aws/credentials

profile_name (string, optional)

Profile name. Default to ‘default’ or ENV[‘AWS_PROFILE’]

Parquet compressor

parquet compressor

parquet_compression_codec (string, optional)

Parquet compression codec. (uncompressed, snappy, gzip, lzo, brotli, lz4, zstd)

Default: snappy

parquet_page_size (string, optional)

Parquet file page size.

Default: 8192 bytes

parquet_row_group_size (string, optional)

Parquet file row group size.

Default: 128 MB

record_type (string, optional)

Record data format type. (avro csv jsonl msgpack tsv msgpack json)

Default: msgpack

schema_file (string, optional)

Path to schema file.

schema_type (string, optional)

Schema type. (avro, bigquery)

Default: avro