Prometheus/Grafana Alert Manager 🔥

The Integration of Grafana / Prometheus AlertManager allows alerts triggered by AlertManager to appear on Komodor's timelines.

Installation

The Alert Manager integration involves 3 steps:

1. Enabling the integration in Komodor.
2. Creating a webhook on the alert manager / Grafana.
3. Adding labels to the alert.

1. Enabling the integration in Komodor

To enable the Komodor Prometheus Alert Manager integration go to the Komodor integrations page, select Prometheus/Grafana Alert Manager, and install the integration.

2. (Option A) The following steps are for manual configuration on the alert manager

Creating webhook

1. Open your alertmanager.yml configuration file
2. Add a receiver to your receivers list, name the receiver komodor , and attach a sink to webhook_configs. In the url field, put the URL that was provided to you during the integration setup. Also set the field send_resolved to true.

receivers:
  - name: komodor
    webhook_configs:
      - url: "<URL_FROM_KOMODOR>"
        send_resolved: true

1. Next, in alertmanager.yml, configure a route so that your alert is routed to komodor

route:
  receiver: komodor

If you already have configured routes you can config multiple as follows:

routes:
  - match:
      severity: critical
    receiver: pagerduty
  - match:
    receiver: komodor

A note about the continued configuration of AlertManager routing rules. If it is set to false, AlertManager sends the alert to the first matching route and stops. continue default value is false.

A full YAML configuration example:

global:
  group_interval: 5m
  repeat_interval: 12h
routes:
  - match:
    receiver: komodor
receivers:
  - name: komodor
    webhook_configs:
      - url: "<URL_FROM_KOMODOR>"
        send_resolved: true

2. (Option B) Configure alerts from Grafana

1. Go to Alerting -> Contact points.

1. Click New contact point.

1. Enter in name komodor, in Contact point type select Webhook and in Url insert the URL from the UI.

1. Click Alerting -> Notification policies.

1. In the notification policies, configure the komodor contact endpoint as you wish to configure alerts to Komodor.

3. Adding labels to the alert.

To relate the alert to the relevant workload and make the alerts visible on workload timelines - adding labels to alerts is required. Each alert without a label will be added to the system without mapping to a specific service.

1. Please note to specify 2 labels on the alert to connect them to the Kubernetes workload:

service: <workload-name>
cluster: <cluster-name>

1. [Optional] Defining custom description to the alert on Komodor (specify it in the annotation):

description: <content>