Is it possible to configure multiple output for a filebeat?

Question

In one of our applications we parse the application logs using logstash and indexing them into elasticsearch. Our simple architecture is logfiles ---> filebeat--->logstash-----> elasticsearch.

As we enabled multiple log files example (apachelogs, passengerlogs, application logs etc,,), logstash is not able to parse the volume of data and hence there are logs missing at elasticsearch. Is there any way to handle huge volume of data at logstash or can we have multiple logstash server to receive logs from filebeat based on the log type? for example: application logs send output logstash-1 and apachelogs to logstash-2.

Thanks in advance.

Answer 1

It is not currently possible to define the same output type multiple time in Filebeat.

But there is a few options to achieve what you want:

• You can use the loadbalance option in filebeat to distribute your events to multiple Logstash. https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html#loadbalance, by default beats will pick a random host and stick to it.

• Use a queue, like kafka and make logstash uses the kafka input, this will allow you add more LS as you need.