Zum Inhalt springen

Prometheus Alertmanager

Richte einen Alertmanager-Webhook-Receiver auf das Relay. Ein ausgelöster Alert startet eine Live-Aktivität und sendet eine Push-Benachrichtigung; sobald der Alert behoben ist, wird die Karte grün und endet.

ℹ Info

This runs on the hosted Relay, so there is nothing to deploy. Alertmanager's webhook is one of the universal webhook presets. A self-hosted relay has to turn the universal webhook on first.

Get Your Integration Key

Verwende deinen Standard-Integrationsschlüssel aus der PushWard-App unter Einstellungen → Integrationsschlüssel oder erstelle einen bereichsbeschränkten Schlüssel für diese Integration.

The key needs notifications at send and activities at manage (permissions). Cards also need an active PushWard subscription. With a lower activities level or no subscription you still get every alert and its resolve, as plain notifications with no card.

Add the receiver

alertmanager.yml
route:
  receiver: pushward
  group_by: [alertname, instance]
  repeat_interval: 1h

receivers:
  - name: pushward
    webhook_configs:
      - url: https://relay.pushward.app/?source=alertmanager
        http_config:
          authorization:
            credentials: hlk_YOUR_INTEGRATION_KEY

authorization defaults to the Bearer type, which is what the relay expects. To keep the key out of this file, use credentials_file in place of credentials.

Leave send_resolved at its default of true. Without it the card never hears that the alert cleared. ?source=alertmanager is optional: it puts that name under the alert on the card and gives these notifications their own thread.

What you see

When an alert fires

An alert Live Activity opens on the Lock Screen and in the Dynamic Island, and one push goes out with it. On Android the card is a Live Update in the notification shade.

  • The title is the alertname label.
  • The line under it is the summary annotation, cut to 100 characters on the card. An alert with no summary just says firing.
  • The card links to the alert's generatorURL. Prometheus puts the whole query in that link and the relay drops any link over 256 characters, so a long query means no link. A link to your own network opens only on a device that can reach it.
  • The timer starts at the group's first webhook, so after group_wait, not when the rule began firing.

While it keeps firing

Alertmanager sends the group again when it changes and every repeat_interval. Those webhooks update the card silently, with no second push.

A card that hears nothing for 4 hours ends on its own, and repeat_interval also defaults to 4 hours, so on the default a long incident closes its card and the next repeat opens a new one with a new push. That is why the example sets 1h.

When it resolves

Once nothing in the group is firing, the card turns green, reads "Resolved" and ends a few seconds later. A passive notification goes out too, with no sound or banner: Resolved · Memory on web-2 above 90% for 10 minutes.

Severity

The alert's severity label sets the card's color and the push's interruption level.

severityCardPush level
critical, page, error, highRedtime-sensitive
warning, warn, mediumOrangeactive
info, low, noneBluepassive
Missing, or a value the relay does not knowOrangeactive

One card per group

The relay keys the card on Alertmanager's groupKey and reads only the first alert in the payload, so a group of three alerts shows one of them. A route with no group_by puts every alert in a single group and you get a single card. Group by alertname at least, and add instance if each host should get its own card.

Query parameters

The universal webhook's query parameters go on the same URL. channels=notification skips the card. level replaces the table above for every notification, the resolved one included. ack=1 repeats the push until someone acknowledges it, and the alert resolving stops the repeats.

Repeat every 2 minutes until acknowledged
https://relay.pushward.app/?source=alertmanager&ack=1&ack_repeat=120