FileBeat not sending data to ElasticSearch Kibana

Question

I'm unable to receive data in the Kibana dashboard from the Filebeat agent. I'm using self-managed ELK with AWS EC2 server. Below is my filebeat.yml

filebeat.inputs:
  - type: log
  enabled: true
  paths:
    - /home/ubuntu/logs/consumer-app/*.log

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

setup.template.settings:
  index.number_of_shards: 1

output.elasticsearch:
  hosts: ["http://PUBLIC_IP:9200"]
setup.kibana:
  host: "http://PUBLIC_IP:5601"

elasticsearch.yaml

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: PUBLIC_IP
http.port: 9200
discovery.type: single-node

kibana.yml

server.port: 5601
server.host: PUBLIC_IP
elasticsearch.hosts: ["http://PUBLIC_IP:9200"]

When I try to hit sudo filebeat setup command. Im getting the below error.

Overwriting ILM policy is disabled. Set setup.ilm.overwrite: true for enabling. Index setup finished. Loading dashboards (Kibana must be running and reachable)

Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to http://localhost:5601/api/status fails: fail to execute the HTTP GET request: Get "http://localhost:5601/api/status": dial tcp 127.0.0.1:5601: connect: connection refused. Response: .

Since I'm new to ELK and filebeat, any help is much appreciated.

Answer 1

i believe the filbeat is trying to communicate to kibana but unfortunately kibana is not running.

In filebeat.yml add this line

setup.ilm.overwrite: true

Ensure that elasticsearch and kibana is running then execute the setup command of filebeat. Keep posted, Thanks!!!