'Prometheus alert multiple routes and receiver doesn't send one rules

I have this alert configuration and expect this behavior.

If destination: bloom and severity: info send to slack-alert-info - it's work

Error there. If destination: bloom and severity: warning|critical send to slack-alert-multi - this works with error. Sverity: warning sending as expected to both Slack's channels but critical sending only to default channel.

May someone help me understand my error, please?

Amtool gives no error.

amtool config routes test --config.file=/opt/prometheus/etc/alertmanager.yml --tree --verify.receivers=slack-alert-multi severity=warning destination=bloom
Matching routes:
.
└── default-route
    └── {destination=~"^(?:bloom)$",severity=~"^(?:warning|critical)$"}  receiver: slack-alert-multi

slack-alert-multi


amtool config routes test --config.file=/opt/prometheus/etc/alertmanager.yml --tree --verify.receivers=slack-alert-multi severity=critical destination=bloom
Matching routes:
.
└── default-route
    └── {destination=~"^(?:bloom)$",severity=~"^(?:warning|critical)$"}  receiver: slack-alert-multi

slack-alert-multi

Alert configuration

...
    labels:
      alerttype: infrastructure
      severity: warning
      destination: bloom
...

---
global:
  resolve_timeout: 30m

route:
  group_by: [ 'alertname', 'cluster', 'severity' ]
  group_wait: 30s
  group_interval: 30s
  repeat_interval: 300s
  receiver: 'slack'

  routes:
  - receiver: 'slack-alert-multi'
    match_re:
      destination: bloom
      severity: warning|critical
  - receiver: 'slack-alert-info'
    match_re:
      destination: bloom
      severity: info

receivers:
- name: 'slack-alert-multi'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T0/B0/V2'
    channel: '#alert-upload'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'
  - api_url: 'https://hooks.slack.com/services/T0/B0/J1'
    channel: '#alert-exports'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'

# Default receiver
- name: 'slack'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T0/B0/2x'
    channel: '#aws-notification'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'

- name: 'slack-alert-info'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/T0/B0/EA'
    channel: '#alert-info'
    send_resolved: true
    icon_url: 'https://avatars3.githubusercontent.com/u/3380462'
    title: '{{ template "custom_title" . }}'
    text: '{{ template "custom_slack_message" . }}'

templates:
- '/opt/alertmanager_notifications.tmpl'
prometheus-alertmanager


Solution 1:[1]

Try add

continue: true

into

  - receiver: 'slack-alert-info'
    match_re:
      destination: bloom
      severity: info
      continue: true

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Andrey True

Related Questions