'Elastic Cloud APM Server - Queue is full

I have many Java microservices running in a Kubernetes Cluster. All of them are APM agents sending data to an APM server in our Elastic Cloud Cluster.

Everything was working fine but suddenly every microservice received the error below showed in the logs.

enter image description here

I tried to restart the cluster, increase the hardware power and I tried to follow the hints but no success.

Obs: The disk is almost empty and the memory usage is ok. Everything is in 7.5.2 version

apmelastic-cloud


Solution 1:[1]

I deleted all the indexes related to APM and everything worked after some minutes.

Solution 2:[2]

for better performance u can fine tune these fields in apm-server.yml file

  1. internal queue size increase queue.mem.events=output.elasticsearch.worker * output.elasticsearch.bulk_max_size default is 4096
  2. output.elasticsearch.worker (increase) default is 1
  3. output.elasticsearch.bulk_max_size (increase) default is 50 very less

Example : for my use case i have used following stats for 2 apm-server nodes and 3 es nodes (1 master 2 data nodes )

queue.mem.events=40000
output.elasticsearch.worker=4
output.elasticsearch.bulk_max_size=10000

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Edgar Peixoto
Solution 2 Paulo

Related Questions