'Loki Ruler not sending alerts to alert Manager
I have loki + grafana + prometheus running in EKS using official helm charts.
I want to send alerts based on error log messages and for that i'm trying to use the loki ruler.
following is the ruler config
ruler:
storage:
type: local
local:
directory: /rules
rule_path: /tmp
alertmanager_url: http://prometheus-alertmanager.prometheus.svc.cluster.local
ring:
kvstore:
store: inmemory
enable_api: true
enable_alertmanager_v2: true
and i have following alert rules/groups
alerting_groups:
- name: testrule
interval: 60s
rules:
- alert: SnowplowError
expr: sum(count_over_time({namespace="snowplow",app="stream-collector"}|="INFO" [30m]))>0
for: 1m
labels:
source: loki
target: easystack
annotations:
summary: High request latency
message: '{{ $labels.clustername }} alert: {{ $labels.message }}.'
the deployement goes successful while i do not get any email alerts. the logs from the pod show that query is being executed.
level=info ts=2022-05-04T18:07:18.823133872Z caller=metrics.go:92 org_id=fake latency=fast query="(sum(count_over_time({namespace=\"snowplow\", app=\"stream-collector\"} |= \"INFO\"[30m])) > 0)" query_type=metric range_type=instant length=0s step=0s duration=47.290222ms status=200 limit=0 returned_lines=0 throughput=0B total_bytes=0B
and i'm not getting any alerts or any error. it just running with no error and no alerts. i'm suspecting that the limit 0 in query might be the culprit but i dont know how to change that limit.
would be really glad if someone could help. been stuck here for a week now.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|