Amazon S3
Amazon S3 plugin for Fluentd
Overview
The s3 output plugin buffers event logs in local file and upload it to S3 periodically. This plugin splits files exactly by using the time of event logs (not the time when the logs are received). For example, a log ‘2011-01-02 message B’ is reached, and then another log ‘2011-01-03 message B’ is reached in this order, the former one is stored in “20110102.gz” file, and latter one in “20110103.gz” file.
For a detailed example, see S3 Output Deployment.
Example output configurations
spec:
s3:
aws_key_id:
valueFrom:
secretKeyRef:
name: logging-s3
key: awsAccessKeyId
aws_sec_key:
valueFrom:
secretKeyRef:
name: logging-s3
key: awsSecretAccessKey
s3_bucket: logging-amazon-s3
s3_region: eu-central-1
path: logs/${tag}/%Y/%m/%d/
buffer:
timekey: 10m
timekey_wait: 30s
timekey_use_utc: true
Configuration
Output Config
acl (string, optional)
Permission for the object in S3
assume_role_credentials (*S3AssumeRoleCredentials, optional)
auto_create_bucket (string, optional)
Create S3 bucket if it does not exists
aws_iam_retries (string, optional)
The number of attempts to load instance profile credentials from the EC2 metadata service using IAM role
aws_key_id (*secret.Secret, optional)
AWS access key id Secret
aws_sec_key (*secret.Secret, optional)
AWS secret key. Secret
buffer (*Buffer, optional)
check_apikey_on_start (string, optional)
Check AWS key on start
check_bucket (string, optional)
Check bucket if exists or not
check_object (string, optional)
Check object before creation
clustername (string, optional)
Custom cluster name
Default: one-eye
compress (*Compress, optional)
Parquet compressor
compute_checksums (string, optional)
AWS SDK uses MD5 for API request/response by default
enable_transfer_acceleration (string, optional)
If true, S3 Transfer Acceleration will be enabled for uploads. IMPORTANT: You must first enable this feature on your destination S3 bucket
force_path_style (string, optional)
If true, the bucket name is always left in the request URI and never moved to the host as a sub-domain
format (*Format, optional)
grant_full_control (string, optional)
Allows grantee READ, READ_ACP, and WRITE_ACP permissions on the object
grant_read (string, optional)
Allows grantee to read the object data and its metadata
grant_read_acp (string, optional)
Allows grantee to read the object ACL
grant_write_acp (string, optional)
Allows grantee to write the ACL for the applicable object
hex_random_length (string, optional)
The length of %{hex_random} placeholder(4-16)
index_format (string, optional)
sprintf format for %{index}
instance_profile_credentials (*S3InstanceProfileCredentials, optional)
oneeye_format (bool, optional)
One-eye format trigger
Default: false
overwrite (string, optional)
Overwrite already existing path
path (string, optional)
Path prefix of the files on S3
proxy_uri (string, optional)
URI of proxy environment
s3_bucket (string, required)
S3 bucket name
s3_endpoint (string, optional)
Custom S3 endpoint (like minio)
s3_metadata (string, optional)
Arbitrary S3 metadata headers to set for the object
s3_object_key_format (string, optional)
The format of S3 object keys (default: %{path}%{time_slice}_%{uuid_hash}_%{index}.%{file_extension})
Default: %{path}%{time_slice}%{uuid_hash}%{index}.%{file_extension}
s3_region (string, optional)
S3 region name
shared_credentials (*S3SharedCredentials, optional)
signature_version (string, optional)
Signature version for API Request (s3,v4)
slow_flush_log_threshold (string, optional)
The threshold for chunk flush performance check. Parameter type is float, not time, default: 20.0 (seconds) If chunk flush takes longer time than this threshold, fluentd logs warning message and increases metric fluentd_output_status_slow_flush_count.
sse_customer_algorithm (string, optional)
Specifies the algorithm to use to when encrypting the object
sse_customer_key (string, optional)
Specifies the customer-provided encryption key for Amazon S3 to use in encrypting data
sse_customer_key_md5 (string, optional)
Specifies the 128-bit MD5 digest of the encryption key according to RFC 1321
ssekms_key_id (string, optional) {#output config-ssekms_key_id}
Specifies the AWS KMS key ID to use for object encryption
ssl_verify_peer (string, optional) {#output config-ssl_verify_peer}
If false, the certificate of endpoint will not be verified
storage_class (string, optional)
The type of storage to use for the object, for example STANDARD, REDUCED_REDUNDANCY, STANDARD_IA, ONEZONE_IA, INTELLIGENT_TIERING, GLACIER, DEEP_ARCHIVE, OUTPOSTS, GLACIER_IR For a complete list of possible values, see the Amazon S3 API reference.
store_as (string, optional)
Archive format on S3
use_bundled_cert (string, optional)
Use aws-sdk-ruby bundled cert
use_server_side_encryption (string, optional)
The Server-side encryption algorithm used when storing this object in S3 (AES256, aws:kms)
warn_for_delay (string, optional)
Given a threshold to treat events as delay, output warning logs if delayed events were put into s3
Assume Role Credentials
assume_role_credentials
duration_seconds (string, optional) {#assume role-credentials-duration_seconds}
The duration, in seconds, of the role session (900-3600)
external_id (string, optional) {#assume role-credentials-external_id}
A unique identifier that is used by third parties when assuming roles in their customers’ accounts.
policy (string, optional) {#assume role-credentials-policy}
An IAM policy in JSON format
role_arn (string, required) {#assume role-credentials-role_arn}
The Amazon Resource Name (ARN) of the role to assume
role_session_name (string, required) {#assume role-credentials-role_session_name}
An identifier for the assumed role session
Instance Profile Credentials
instance_profile_credentials
http_open_timeout (string, optional) {#instance profile-credentials-http_open_timeout}
Number of seconds to wait for the connection to open
http_read_timeout (string, optional) {#instance profile-credentials-http_read_timeout}
Number of seconds to wait for one block to be read
ip_address (string, optional) {#instance profile-credentials-ip_address}
IP address
Default: 169.254.169.254
port (string, optional) {#instance profile-credentials-port}
Port number
Default: 80
retries (string, optional) {#instance profile-credentials-retries}
Number of times to retry when retrieving credentials
Shared Credentials
shared_credentials
path (string, optional)
Path to the shared file.
Default: $HOME/.aws/credentials
profile_name (string, optional)
Profile name. Default to ‘default’ or ENV[‘AWS_PROFILE’]
Parquet compressor
parquet compressor
parquet_compression_codec (string, optional)
Parquet compression codec. (uncompressed, snappy, gzip, lzo, brotli, lz4, zstd)
Default: snappy
parquet_page_size (string, optional)
Parquet file page size.
Default: 8192 bytes
parquet_row_group_size (string, optional)
Parquet file row group size.
Default: 128 MB
record_type (string, optional)
Record data format type. (avro csv jsonl msgpack tsv msgpack json)
Default: msgpack
schema_file (string, optional)
Path to schema file.
schema_type (string, optional)
Schema type. (avro, bigquery)
Default: avro