Amazon S3
Amazon S3 plugin for Fluentd
Overview
s3 output plugin buffers event logs in local file and upload it to S3 periodically. This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log ‘2011-01-02 message B’ is reached, and then another log ‘2011-01-03 message B’ is reached in this order, the former one is stored in “20110102.gz” file, and latter one in “20110103.gz” file.
Example: S3 Output Deployment
Example output configurations
spec:
s3:
aws_key_id:
valueFrom:
secretKeyRef:
name: logging-s3
key: awsAccessKeyId
aws_sec_key:
valueFrom:
secretKeyRef:
name: logging-s3
key: awsSecretAccessKey
s3_bucket: logging-amazon-s3
s3_region: eu-central-1
path: logs/${tag}/%Y/%m/%d/
buffer:
timekey: 10m
timekey_wait: 30s
timekey_use_utc: true
Configuration
Output Config
aws_key_id (*secret.Secret, optional)
AWS access key id Secret
Default: -
aws_sec_key (*secret.Secret, optional)
AWS secret key. Secret
Default: -
check_apikey_on_start (string, optional)
Check AWS key on start
Default: -
grant_read (string, optional)
Allows grantee to read the object data and its metadata
Default: -
overwrite (string, optional)
Overwrite already existing path
Default: -
path (string, optional)
Path prefix of the files on S3
Default: -
grant_write_acp (string, optional)
Allows grantee to write the ACL for the applicable object
Default: -
check_bucket (string, optional)
Check bucket if exists or not
Default: -
sse_customer_key (string, optional)
Specifies the customer-provided encryption key for Amazon S3 to use in encrypting data
Default: -
sse_customer_key_md5 (string, optional)
Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321
Default: -
compute_checksums (string, optional)
AWS SDK uses MD5 for API request/response by default
Default: -
warn_for_delay (string, optional)
Given a threshold to treat events as delay, output warning logs if delayed events were put into s3
Default: -
use_bundled_cert (string, optional)
Use aws-sdk-ruby bundled cert
Default: -
s3_endpoint (string, optional)
Custom S3 endpoint (like minio)
Default: -
ssekms_key_id (string, optional)
Specifies the AWS KMS key ID to use for object encryption
Default: -
s3_metadata (string, optional)
Arbitrary S3 metadata headers to set for the object
Default: -
force_path_style (string, optional)
If true, the bucket name is always left in the request URI and never moved to the host as a sub-domain
Default: -
auto_create_bucket (string, optional)
Create S3 bucket if it does not exists
Default: -
index_format (string, optional)
sprintf format for %{index}
Default: -
signature_version (string, optional)
Signature version for API Request (s3,v4)
Default: -
enable_transfer_acceleration (string, optional)
If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket
Default: -
ssl_verify_peer (string, optional)
If false, the certificate of endpoint will not be verified
Default: -
proxy_uri (string, optional)
URI of proxy environment
Default: -
grant_read_acp (string, optional)
Allows grantee to read the object ACL
Default: -
check_object (string, optional)
Check object before creation
Default: -
sse_customer_algorithm (string, optional)
Specifies the algorithm to use to when encrypting the object
Default: -
use_server_side_encryption (string, optional)
The Server-side encryption algorithm used when storing this object in S3 (AES256, aws:kms)
Default: -
s3_region (string, optional)
S3 region name
Default: -
acl (string, optional)
Permission for the object in S3
Default: -
grant_full_control (string, optional)
Allows grantee READ, READ_ACP, and WRITE_ACP permissions on the object
Default: -
hex_random_length (string, optional)
The length of %{hex_random} placeholder(4-16)
Default: -
s3_object_key_format (string, optional)
The format of S3 object keys (default: %{path}%{time_slice}%{uuid_hash}%{index}.%{file_extension})
Default: %{path}%{time_slice}%{uuid_hash}%{index}.%{file_extension}
s3_bucket (string, required)
S3 bucket name
Default: -
store_as (string, optional)
Archive format on S3
Default: -
storage_class (string, optional)
The type of storage to use for the object, for example STANDARD, REDUCED_REDUNDANCY, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, OUTPOSTS, GLACIER_IR For a complete list of possible values, see the Amazon S3 API reference.
Default: -
aws_iam_retries (string, optional)
The number of attempts to load instance profile credentials from the EC2 metadata service using IAM role
Default: -
buffer (*Buffer, optional)
Default: -
slow_flush_log_threshold (string, optional)
The threshold for chunk flush performance check. Parameter type is float, not time, default: 20.0 (seconds) If chunk flush takes longer time than this threshold, fluentd logs warning message and increases metric fluentd_output_status_slow_flush_count.
Default: -
format (*Format, optional)
Default: -
assume_role_credentials (*S3AssumeRoleCredentials, optional)
Default: -
instance_profile_credentials (*S3InstanceProfileCredentials, optional)
Default: -
shared_credentials (*S3SharedCredentials, optional)
Default: -
compress (*Compress, optional)
Parquet compressor
Default: -
oneeye_format (bool, optional)
One-eye format trigger
Default: false
clustername (string, optional)
Custom cluster name
Default: one-eye
Assume Role Credentials
assume_role_credentials
role_arn (string, required) {#assume role-credentials-role_arn}
The Amazon Resource Name (ARN) of the role to assume
Default: -
role_session_name (string, required) {#assume role-credentials-role_session_name}
An identifier for the assumed role session
Default: -
policy (string, optional) {#assume role-credentials-policy}
An IAM policy in JSON format
Default: -
duration_seconds (string, optional) {#assume role-credentials-duration_seconds}
The duration, in seconds, of the role session (900-3600)
Default: -
external_id (string, optional) {#assume role-credentials-external_id}
A unique identifier that is used by third parties when assuming roles in their customers’ accounts.
Default: -
Instance Profile Credentials
instance_profile_credentials
ip_address (string, optional) {#instance profile-credentials-ip_address}
IP address
Default: 169.254.169.254
port (string, optional) {#instance profile-credentials-port}
Port number
Default: 80
http_open_timeout (string, optional) {#instance profile-credentials-http_open_timeout}
Number of seconds to wait for the connection to open
Default: -
http_read_timeout (string, optional) {#instance profile-credentials-http_read_timeout}
Number of seconds to wait for one block to be read
Default: -
retries (string, optional) {#instance profile-credentials-retries}
Number of times to retry when retrieving credentials
Default: -
Shared Credentials
shared_credentials
profile_name (string, optional)
Profile name. Default to ‘default’ or ENV[‘AWS_PROFILE’]
Default: -
path (string, optional)
Path to the shared file.
Default: $HOME/.aws/credentials
Parquet compressor
parquet compressor
parquet_compression_codec (string, optional)
Parquet compression codec. (uncompressed, snappy, gzip, lzo, brotli, lz4, zstd)
Default: snappy
parquet_page_size (string, optional)
Parquet file page size.
Default: 8192 bytes
parquet_row_group_size (string, optional)
Parquet file row group size.
Default: 128 MB
record_type (string, optional)
Record data format type. (avro csv jsonl msgpack tsv msgpack json)
Default: msgpack
schema_type (string, optional)
Schema type. (avro, bigquery)
Default: avro
schema_file (string, optional)
Path to schema file.
Default: -