J'essaye de tester le plugin fluentd-s3 mais pour l'instant il ne poste pas mes logs dans le s3 s3. Je cours tout sur ubuntu xenial, ayant installé fluentd avec td-agent. Ce qui suit est le fichier journal td agent situé à /var/log/td-agent/td-agent.logFluentd n'écrit pas les logs dans amazon s3
2016-09-23 09:16:18 -0300 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2016-09-23 09:16:18 -0300 [info]: starting fluentd-0.12.26
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-kinesis' version '1.1.1'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-mongo' version '0.7.13'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-s3' version '0.6.8'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-td' version '0.10.28'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-09-23 09:16:18 -0300 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2016-09-23 09:16:18 -0300 [info]: gem 'fluentd' version '0.12.26'
2016-09-23 09:16:18 -0300 [info]: adding match pattern="nginx.log" type="s3"
2016-09-23 09:16:18 -0300 [info]: adding source type="tail"
2016-09-23 09:16:18 -0300 [info]: using configuration file: <ROOT>
<source>
type tail
format nginx
path /var/log/nginx/access.log
pos_file /var/log/td-agent/nginx-access.pos
tag nginx.log
</source>
<match nginx.log>
@type s3
s3_bucket kd.creatives
aws_key_id xxxxxx
aws_sec_key xxxxxx
s3_region us-west-2
path logs/
buffer_path /var/log/td-agent/s3
time_slice_format %Y%m%d%H%M
utc
format_json true
include_time_key true
buffer_chunk_limit 256m
</match>
</ROOT>
2016-09-23 09:16:18 -0300 [warn]: parameter 'format_json' in <match nginx.log>
@type s3
s3_bucket kd.creatives
aws_key_id xxxxxx
aws_sec_key xxxxxx
s3_region us-west-2
path logs/
buffer_path /var/log/td-agent/s3
time_slice_format %Y%m%d%H%M
utc
format_json true
include_time_key true
buffer_chunk_limit 256m
</match> is not used.
2016-09-23 09:16:20 -0300 [info]: following tail of /var/log/nginx/access.log
2016-09-23 09:16:25 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:25 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:25 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:25 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:26 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:26 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:27 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:27 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET/HTTP/1.1\" 200 612 \"-\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
2016-09-23 09:16:28 -0300 [warn]: pattern not match: "172.17.0.1 - - [23/Sep/2016:12:16:28 +0000] \"GET /favicon.ico HTTP/1.1\" 404 571 \"http://localhost:10241/\" \"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.89 Safari/537.36\" \"-\""
Aussi voici mon td-agent.conf fichier
# Centralized nginx logs into Amazon s3
<source>
type tail
format nginx
path /var/log/nginx/access.log
pos_file /var/log/td-agent/nginx-access.pos
tag nginx.log
</source>
<match nginx.log>
@type s3
s3_bucket xxxxxxx
aws_key_id xxxxxxxxxxxxxxxx
aws_sec_key xxxxxxxxxxxxxxxxxxxxxxxxxx
s3_region us-west-2
path logs/
buffer_path /var/log/td-agent/s3
time_slice_format %Y%m%d%H%M
utc
format_json true
include_time_key true
buffer_chunk_limit 256m
</match>
J'apprécierais vraiment toute réponse sur pourquoi fluentd ne publie pas les journaux dans seau s3.
Merci pour votre réponse. Je commençais juste avec fluentd et maintenant je me suis habitué au format log regex (même que ruby). – francotestori