La sortie de Logstash ne reconnaît pas les colonnes dans Kibana

Question

J'essaie d'obtenir mon fichier .CSV dans Kibana pour la visualisation. J'ai l'impression que je suis sur le point de le faire fonctionner mais je n'arrive pas à comprendre comment obtenir une sortie correcte.

En Kibana je vois mon fichier .csv comme:

message: News,test@email.com,10.10.10.10 

Il ressemble à mon CSV ouput est appelé dans le message 1 sur le terrain. J'aimerais obtenir 3 champs différents: Nom, Email, IP. J'ai essayé beaucoup de fichiers csv et de codes différents, mais pas encore de succès.

CSV FICHIER:

Name,Email,IP 
Auto,auto@newsuk,10.0.0.196 
News,test@email.com,10.10.10.10 
nieuwsbrieven,nieuwsbrieven@nl,10.10.10.10 

fichier CONF:

input { 
file { 
path => "C:\Users\JOEY2\Desktop\Deelproblemen\Applicatie\Output\test.csv" 
start_position => beginning 
sincedb_path => "/dev/null" 
}} 

filter { 
csv { 
separator => "," 
columns => ["Date","Open","High"] 
} 
} 
output { 
elasticsearch { 
hosts => ["http://localhost:9200"] 
index => "csv_index" 
} 
stdout {} 
} 

filebeat.yml filebeat.prospectors:

- input_type: log 
paths: 
- C:\Users\JOEY2\Desktop\Deelproblemen\Applicatie\Output\test.csv 

output.elasticsearch: 
hosts: ["localhost:9200"] 
template.name: "testttt" 
template.overwrite: true 

output.logstash: 
hosts: ["localhost:5044"] 

Logstash sortie CMD:

[2017-10-12T13:53:52,682][INFO ][logstash.pipeline  ] Pipeline main started 
[2017-10-12T13:53:52,690][INFO ][org.logstash.beats.Server] Starting server on port: 5044 
[2017-10-12T13:53:53,003][INFO ][logstash.agent   ] Successfully started Logstash API endpoint {:port=>9600} 
{ 
    "@timestamp" => 2017-10-12T11:53:53.659Z, 
     "offset" => 15, 
     "@version" => "1", 
    "input_type" => "log", 
      "beat" => { 
      "name" => "DESKTOP-VEQHHVT", 
     "hostname" => "DESKTOP-VEQHHVT", 
     "version" => "5.6.2" 
    }, 
      "host" => "DESKTOP-VEQHHVT", 
     "source" => "C:\\Users\\JOEY2\\Desktop\\Deelproblemen\\Applicatie\\Output\\test.csv", 
     "message" => "Name,Email,IP", 
      "type" => "log", 
      "tags" => [ 
     [0] "beats_input_codec_plain_applied" 
    ] 
} 
{ 
    "@timestamp" => 2017-10-12T11:53:53.659Z, 
     "offset" => 44, 
     "@version" => "1", 
    "input_type" => "log", 
      "beat" => { 
      "name" => "DESKTOP-VEQHHVT", 
     "hostname" => "DESKTOP-VEQHHVT", 
     "version" => "5.6.2" 
    }, 
      "host" => "DESKTOP-VEQHHVT", 
     "source" => "C:\\Users\\JOEY2\\Desktop\\Deelproblemen\\Applicatie\\Output\\test.csv", 
     "message" => "Auto,auto@newsuk,10.0.0.196", 
      "type" => "log", 
      "tags" => [ 
     [0] "beats_input_codec_plain_applied" 
    ] 
} 
{ 
    "@timestamp" => 2017-10-12T11:53:53.659Z, 
     "offset" => 77, 
     "@version" => "1", 
      "beat" => { 
      "name" => "DESKTOP-VEQHHVT", 
     "hostname" => "DESKTOP-VEQHHVT", 
     "version" => "5.6.2" 
    }, 
    "input_type" => "log", 
      "host" => "DESKTOP-VEQHHVT", 
     "source" => "C:\\Users\\JOEY2\\Desktop\\Deelproblemen\\Applicatie\\Output\\test.csv", 
     "message" => "News,test@email.com,10.10.10.10", 
      "type" => "log", 
      "tags" => [ 
     [0] "beats_input_codec_plain_applied" 
    ] 

Al mes colonnes CSV/lignes entrent dans le message variable.

sortie de commande Curl: (boucle -s localhost: 9200/_cat/indices v?)

yellow open filebeat-2017.10.12 ux6-ByOERj-2XEBojkxhXg 5 1   3   0  13.3kb   13.3kb 
    enter code here 

Terminal ELAC SORTIE:

[2017-10-12T13:53:11,763][INFO ][o.e.n.Node    ] [] initializing ... 
[2017-10-12T13:53:11,919][INFO ][o.e.e.NodeEnvironment ] [Zs6ZAuy] using [1] data paths, mounts [[(C:)]], net usable_space [1.9tb], net total_space [1.9tb], spins? [unknown], types [NTFS] 
[2017-10-12T13:53:11,920][INFO ][o.e.e.NodeEnvironment ] [Zs6ZAuy] heap size [1.9gb], compressed ordinary object pointers [true] 
[2017-10-12T13:53:12,126][INFO ][o.e.n.Node    ] node name [Zs6ZAuy] derived from node ID [Zs6ZAuyyR2auGVnPoD9gRw]; set [node.name] to override 
[2017-10-12T13:53:12,128][INFO ][o.e.n.Node    ] version[5.6.2], pid[3384], build[57e20f3/2017-09-23T13:16:45.703Z], OS[Windows 10/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_144/25.144-b01] 
[2017-10-12T13:53:12,128][INFO ][o.e.n.Node    ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=C:\ELK-Stack\elasticsearch\elasticsearch-5.6.2] 
[2017-10-12T13:53:13,550][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [aggs-matrix-stats] 
[2017-10-12T13:53:13,616][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [ingest-common] 
[2017-10-12T13:53:13,722][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [lang-expression] 
[2017-10-12T13:53:13,798][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [lang-groovy] 
[2017-10-12T13:53:13,886][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [lang-mustache] 
[2017-10-12T13:53:13,988][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [lang-painless] 
[2017-10-12T13:53:14,059][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [parent-join] 
[2017-10-12T13:53:14,154][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [percolator] 
[2017-10-12T13:53:14,223][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [reindex] 
[2017-10-12T13:53:14,289][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [transport-netty3] 
[2017-10-12T13:53:14,360][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] loaded module [transport-netty4] 
[2017-10-12T13:53:14,448][INFO ][o.e.p.PluginsService  ] [Zs6ZAuy] no plugins loaded 
[2017-10-12T13:53:18,328][INFO ][o.e.d.DiscoveryModule ] [Zs6ZAuy] using discovery type [zen] 
[2017-10-12T13:53:19,204][INFO ][o.e.n.Node    ] initialized 
[2017-10-12T13:53:19,204][INFO ][o.e.n.Node    ] [Zs6ZAuy] starting ... 
[2017-10-12T13:53:20,071][INFO ][o.e.t.TransportService ] [Zs6ZAuy] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300} 
[2017-10-12T13:53:23,130][INFO ][o.e.c.s.ClusterService ] [Zs6ZAuy] new_master {Zs6ZAuy}{Zs6ZAuyyR2auGVnPoD9gRw}{jBwTE7rUS4i_Ugh6k6DAMg}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) 
[2017-10-12T13:53:23,883][INFO ][o.e.g.GatewayService  ] [Zs6ZAuy] recovered [5] indices into cluster_state 
[2017-10-12T13:53:25,962][INFO ][o.e.c.r.a.AllocationService] [Zs6ZAuy] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]). 
[2017-10-12T13:53:25,981][INFO ][o.e.h.n.Netty4HttpServerTransport] [Zs6ZAuy] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[::1]:9200} 
[2017-10-12T13:53:25,986][INFO ][o.e.n.Node    ] [Zs6ZAuy] started 
[2017-10-12T13:53:59,245][INFO ][o.e.c.m.MetaDataCreateIndexService] [Zs6ZAuy] [filebeat-2017.10.12] creating index, cause [auto(bulk api)], templates [filebeat, testttt], shards [5]/[1], mappings [_default_] 
[2017-10-12T13:53:59,721][INFO ][o.e.c.m.MetaDataMappingService] [Zs6ZAuy] [filebeat-2017.10.12/ux6-ByOERj-2XEBojkxhXg] create_mapping [doc] 

sortie Filebeat:

C:\ELK-Stack\filebeat>filebeat -e -c filebeat.yml -d "publish" 
2017/10/12 11:53:53.632142 beat.go:297: INFO Home path: [C:\ELK-Stack\filebeat] Config path: [C:\ELK-Stack\filebeat] Data path: [C:\ELK-Stack\filebeat\data] Logs path: [C:\ELK-Stack\filebeat\logs] 
2017/10/12 11:53:53.632142 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.2 
2017/10/12 11:53:53.634143 publish.go:228: WARN Support for loading more than one output is deprecated and will not be supported in version 6.0. 
2017/10/12 11:53:53.635144 output.go:258: INFO Loading template enabled. Reading template file: C:\ELK-Stack\filebeat\filebeat.template.json 
2017/10/12 11:53:53.636144 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: C:\ELK-Stack\filebeat\filebeat.template-es2x.json 
2017/10/12 11:53:53.637143 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: C:\ELK-Stack\filebeat\filebeat.template-es6x.json 
2017/10/12 11:53:53.638144 client.go:128: INFO Elasticsearch url: http://localhost:9200 
2017/10/12 11:53:53.639143 outputs.go:108: INFO Activated elasticsearch as output plugin. 
2017/10/12 11:53:53.639143 logstash.go:90: INFO Max Retries set to: 3 
2017/10/12 11:53:53.640143 outputs.go:108: INFO Activated logstash as output plugin. 
2017/10/12 11:53:53.640143 publish.go:243: DBG Create output worker 
2017/10/12 11:53:53.641143 publish.go:243: DBG Create output worker 
2017/10/12 11:53:53.641143 publish.go:285: DBG No output is defined to store the topology. The server fields might not be filled. 
2017/10/12 11:53:53.642144 publish.go:300: INFO Publisher name: DESKTOP-VEQHHVT 
2017/10/12 11:53:53.634143 metrics.go:23: INFO Metrics logging every 30s 
2017/10/12 11:53:53.646143 async.go:63: INFO Flush Interval set to: 1s 
2017/10/12 11:53:53.647142 async.go:64: INFO Max Bulk Size set to: 50 
2017/10/12 11:53:53.647142 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=50) 
2017/10/12 11:53:53.648144 async.go:63: INFO Flush Interval set to: 1s 
2017/10/12 11:53:53.648144 async.go:64: INFO Max Bulk Size set to: 2048 
2017/10/12 11:53:53.649144 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=2048) 
2017/10/12 11:53:53.649144 beat.go:233: INFO filebeat start running. 
2017/10/12 11:53:53.650144 registrar.go:68: INFO No registry file found under: C:\ELK-Stack\filebeat\data\registry. Creating a new registry file. 
2017/10/12 11:53:53.652144 registrar.go:106: INFO Loading registrar data from C:\ELK-Stack\filebeat\data\registry 
2017/10/12 11:53:53.654145 registrar.go:123: INFO States Loaded from registrar: 0 
2017/10/12 11:53:53.655145 crawler.go:38: INFO Loading Prospectors: 1 
2017/10/12 11:53:53.655145 prospector_log.go:65: INFO Prospector with previous states loaded: 0 
2017/10/12 11:53:53.656144 prospector.go:124: INFO Starting prospector of type: log; id: 11034545279404679229 
2017/10/12 11:53:53.656144 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1 
2017/10/12 11:53:53.655145 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s 
2017/10/12 11:53:53.655145 registrar.go:236: INFO Starting Registrar 
2017/10/12 11:53:53.657144 log.go:91: INFO Harvester started for file: C:\Users\JOEY2\Desktop\Deelproblemen\Applicatie\Output\test.csv 
2017/10/12 11:53:53.655145 sync.go:41: INFO Start sending events to output 
2017/10/12 11:53:58.682432 client.go:214: DBG Publish: { 
    "@timestamp": "2017-10-12T11:53:53.659Z", 
    "beat": { 
    "hostname": "DESKTOP-VEQHHVT", 
    "name": "DESKTOP-VEQHHVT", 
    "version": "5.6.2" 
    }, 
    "input_type": "log", 
    "message": "Name,Email,IP", 
    "offset": 15, 
    "source": "C:\\Users\\JOEY2\\Desktop\\Deelproblemen\\Applicatie\\Output\\test.csv", 
    "type": "log" 
} 
2017/10/12 11:53:58.685434 client.go:214: DBG Publish: { 
    "@timestamp": "2017-10-12T11:53:53.659Z", 
    "beat": { 
    "hostname": "DESKTOP-VEQHHVT", 
    "name": "DESKTOP-VEQHHVT", 
    "version": "5.6.2" 
    }, 
    "input_type": "log", 
    "message": "Auto,auto@newsuk,10.0.0.196", 
    "offset": 44, 
    "source": "C:\\Users\\JOEY2\\Desktop\\Deelproblemen\\Applicatie\\Output\\test.csv", 
    "type": "log" 
} 
2017/10/12 11:53:58.685434 client.go:214: DBG Publish: { 
    "@timestamp": "2017-10-12T11:53:53.659Z", 
    "beat": { 
    "hostname": "DESKTOP-VEQHHVT", 
    "name": "DESKTOP-VEQHHVT", 
    "version": "5.6.2" 
    }, 
    "input_type": "log", 
    "message": "News,test@email.com,10.10.10.10", 
    "offset": 77, 
    "source": "C:\\Users\\JOEY2\\Desktop\\Deelproblemen\\Applicatie\\Output\\test.csv", 
    "type": "log" 
} 
2017/10/12 11:53:58.686434 output.go:109: DBG output worker: publish 3 events 
2017/10/12 11:53:58.686434 output.go:109: DBG output worker: publish 3 events 
2017/10/12 11:53:58.738437 client.go:667: INFO Connected to Elasticsearch version 5.6.2 
2017/10/12 11:53:58.748436 output.go:317: INFO Trying to load template for client: http://localhost:9200 
2017/10/12 11:53:58.890446 output.go:324: INFO Existing template will be overwritten, as overwrite is enabled. 
2017/10/12 11:53:59.154461 client.go:592: INFO Elasticsearch template with name 'testttt' loaded 
2017/10/12 11:54:00.020510 sync.go:70: DBG Events sent: 4 

Kibana sortie:

@timestamp:October 12th 2017, 13:53:53.659 beat.hostname:DESKTOP-VEQHHVT beat.name:DESKTOP-VEQHHVT beat.version:5.6.2 input_type:log message:Auto,auto@newsuk,10.0.0.196 offset:44 source:C:\Users\JOEY2\Desktop\Deelproblemen\Applicatie\Output\test.csv type:log _id:AV8QbyIcTtSiVplm9CwA _type:doc _index:filebeat-2017.10.12 _score:1 
@timestamp:October 12th 2017, 13:53:53.659 beat.hostname:DESKTOP-VEQHHVT beat.name:DESKTOP-VEQHHVT beat.version:5.6.2 input_type:log message:News,test@email.com,10.10.10.10 offset:77 source:C:\Users\JOEY2\Desktop\Deelproblemen\Applicatie\Output\test.csv type:log _id:AV8QbyIcTtSiVplm9CwB _type:doc _index:filebeat-2017.10.12 _score:1 
@timestamp:October 12th 2017, 13:53:53.659 beat.hostname:DESKTOP-VEQHHVT beat.name:DESKTOP-VEQHHVT beat.version:5.6.2 input_type:log message:Name,Email,IP offset:15 source:C:\Users\JOEY2\Desktop\Deelproblemen\Applicatie\Output\test.csv type:log _id:AV8QbyIcTtSiVplm9Cv_ _type:doc _index:filebeat-2017.10.12 _score:1 

Answer 1

Vous donnez des noms de colonne incorrects dans les filtres csv et le nom de colonne doit être donné sans guillemets ("). J'ai essayé cela et ça marche pour moi. Vérifiez si cela fonctionne pour vous. Mon fichier de configuration logstash:

input { 
    file { 
     path => "/home/quality/Desktop/work/csv.csv" 
     start_position => "beginning" 
     sincedb_path => "/dev/null" 
    } 

} 

filter { 
    csv { 
     separator => "," 
     columns => [Name,Email,IP] 
    } 
} 

output { 
    elasticsearch { 
     hosts => "localhost" 
     index => "csv" 
     document_type => "csv" 
    } 

    stdout { codec => rubydebug} 
}