How to configure prometheus with alertmanager?

Question

docker-compose.yml: This is the docker-compose to run the prometheus, node-exporter and alert-manager service. All the services are running great. Even the health status in target menu of prometheus shows ok.

version: '2'

services:

    prometheus:
        image: prom/prometheus
        privileged: true
        volumes:
            - ./prometheus.yml:/etc/prometheus/prometheus.yml
            - ./alertmanger/alert.rules:/alert.rules
        command:
            - '--config.file=/etc/prometheus/prometheus.yml'
        ports:
            - '9090:9090'

    node-exporter:
        image: prom/node-exporter
        ports:
            - '9100:9100'

    alertmanager:
        image: prom/alertmanager
        privileged: true
        volumes:
             - ./alertmanager/alertmanager.yml:/alertmanager.yml
        command:
            - '--config.file=/alertmanager.yml'
        ports:
            - '9093:9093'

prometheus.yml

This is the prometheus config file with targets and alerts target sets. The alertmanager target url is working fine.

global:
  scrape_interval: 5s
  external_labels:
    monitor: 'my-monitor'

# this is where I have simple alert rules
rule_files:
  - ./alertmanager/alert.rules

scrape_configs:
    - job_name: 'prometheus'
      static_configs: 
        - targets: ['localhost:9090']

    - job_name: 'node-exporter'
      static_configs:
        - targets: ['node-exporter:9100']

alerting:
  alertmanagers:
    - static_configs:
      - targets: ['some-ip:9093']

alert.rules: Just a simple alert rules to show alert when service is down

ALERT service_down
  IF up == 0

alertmanager.yml

This is to send the message on slack when alerting occurs.

global:
  slack_api_url: 'https://api.slack.com/apps/A90S3Q753'

    route:
      receiver: 'slack'

    receivers:
      - name: 'slack'
        slack_configs:
          - send_resolved: true
            username: 'tara gurung'
            channel: '#general'
            api_url: 'https://hooks.slack.com/services/T52GRFN3F/B90NMV1U2/QKj1pZu3ZVY0QONyI5sfsdf'

Problems: All the containers are working fine I am not able to figure out the exact problem.What am I really missing. Checking the alerts in prometheus shows.

Alerts No alerting rules defined

Answer 1

Your ./alertmanager/alert.rules file is not included in your docker config, so it is not available in the container. You need to add it to the prometheus service:

prometheus:
    image: prom/prometheus
    privileged: true
    volumes:
        - ./prometheus.yml:/etc/prometheus/prometheus.yml
        - ./alertmanager/alert.rules:/alertmanager/alert.rules
    command:
        - '--config.file=/etc/prometheus/prometheus.yml'

    ports:
        - '9090:9090'

And probably give an absolute path inside prometheus.yml:

rule_files:
- "/alertmanager/alert.rules"

You also need to make sure you alerting rules are valid. Please see the prometheus docs for details and examples. You alert.rules file should look something like this:

groups:
- name: example
  rules:

  # Alert for any instance that is unreachable for >5 minutes.
  - alert: InstanceDown
    expr: up == 0
    for: 5m

Once you have multiple files, it may be better to add the entire directory as a volume rather than individual files.

Answer 2

If you need answers to this question see the explanation on this link How to make alert rules visible on Prometheus User Interface?

Your alert rules inside the prometheus.yml should look like this

rule_files:
 - "/etc/prometheus/alert.rules.yml"

You need to stop the alertmanager and prometheus containers and run this

docker run -d --name prometheus_ops -p 9191:9090 -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml -v $(pwd)/alert.rules.yml:/etc/prometheus/alert.rules.yml prom/prometheus

Verify if you can see the alert.rule config path : Prometheus container ID and go to cd /etc/prometheus

docker exec -it fa99f733f69b sh