Alibaba Cloud
Aliyun OSS plugin for Fluentd
Overview
Fluent OSS output plugin buffers event logs in local files and uploads them to OSS periodically in background threads.
This plugin splits events by using the timestamp of event logs. For example, a log ‘2019-04-09 message Hello’ is reached, and then another log ‘2019-04-10 message World’ is reached in this order, the former is stored in “20190409.gz” file, and latter in “20190410.gz” file.
Fluent OSS input plugin reads data from OSS periodically.
This plugin uses MNS on the same region of the OSS bucket. We must setup MNS and OSS event notification before using this plugin.
This document shows how to setup MNS and OSS event notification.
This plugin will poll events from MNS queue and extract object keys from these events, and then will read those objects from OSS. More info at https://github.com/aliyun/fluent-plugin-oss
Configuration
Output Config
endpoint (string, required)
OSS endpoint to connect to’
Default: -
bucket (string, required)
Your bucket name
Default: -
access_key_id (*secret.Secret, required)
Your access key id Secret
Default: -
aaccess_key_secret (*secret.Secret, required)
Your access secret key Secret
Default: -
path (string, optional)
Path prefix of the files on OSS
Default: fluent/logs
upload_crc_enable (bool, optional)
Upload crc enabled
Default: true
download_crc_enable (bool, optional)
Download crc enabled
Default: true
open_timeout (int, optional)
Timeout for open connections
Default: 10
read_timeout (int, optional)
Timeout for read response
Default: 120
oss_sdk_log_dir (string, optional)
OSS SDK log directory
Default: /var/log/td-agent
key_format (string, optional)
The format of OSS object keys
Default: %{path}/%{time_slice}%{index}%{thread_id}.%{file_extension}
store_as (string, optional)
Archive format on OSS: gzip, json, text, lzo, lzma2
Default: gzip
auto_create_bucket (bool, optional)
desc ‘Create OSS bucket if it does not exists
Default: false
overwrite (bool, optional)
Overwrite already existing path
Default: false
check_bucket (bool, optional)
Check bucket if exists or not
Default: true
check_object (bool, optional)
Check object before creation
Default: true
hex_random_length (int, optional)
The length of %{hex_random} placeholder(4-16)
Default: 4
index_format (string, optional)
sprintf format for %{index}
Default: %d
warn_for_delay (string, optional)
Given a threshold to treat events as delay, output warning logs if delayed events were put into OSS
Default: -
format (*Format, optional)
Default: -
buffer (*Buffer, optional)
Default: -
slow_flush_log_threshold (string, optional)
The threshold for chunk flush performance check. Parameter type is float, not time, default: 20.0 (seconds) If chunk flush takes longer time than this threshold, fluentd logs warning message and increases metric fluentd_output_status_slow_flush_count.
Default: -