Aller au contenu

Alert delivery authority — P1.4

Status

P1.4B2A Discord delivery foundation  CLOSED / PASS
P1.4B2B systemd alert bridge         CLOSED / PASS
P1.4 GLOBAL                          CLOSED / PASS
Certified main SHA                   c1b2f6a356f23c1ace064a2a7e4f71c33ffb584e

P1.4B2A established Alertmanager + Discord delivery. P1.4B2B extended the same routing authority to host systemd backup/maintenance failures. The B2B contract below supersedes the B2A host-port statement while preserving the B2A egress/secret model.

Scope

P1.4B2A introduces Alertmanager as the single routing authority for Prometheus alerts and Discord as the external notification channel. The Discord webhook secret stays outside Git and outside the platform .env.

The runtime contract is:

  • Prometheus remains on the internal observability network.
  • Alertmanager joins both observability and the dedicated alert-egress network.
  • No other service may join alert-egress.
  • P1.4B2A originally published no host port. Under the current P1.4B2B authority, Alertmanager publishes only 127.0.0.1:9093 so host systemd units can reach its API; public exposure such as 0.0.0.0:9093 remains forbidden.
  • Alertmanager state is persisted on /mnt/data/cmonchoix-infra/observability/alertmanager.
  • The Discord webhook URL is read from /etc/cmonchoix/alerting/discord-webhook-url on the host.
  • The Discord webhook value must never be committed, printed in evidence, or copied into .env.

Runtime secret

Create the parent directory and empty secret file without placing the URL in shell history:

sudo install -d -m 0750 -o root -g root /etc/cmonchoix/alerting
sudo install -m 0640 -o root -g 65534 /dev/null /etc/cmonchoix/alerting/discord-webhook-url
sudoedit /etc/cmonchoix/alerting/discord-webhook-url

The file must contain exactly one Discord HTTPS webhook URL followed by a newline. Create the webhook in the target Discord channel and paste only that URL into this root-managed file.

Required metadata:

owner uid = 0
group gid = 65534
mode      = 0640

The group is required because the Alertmanager container runs explicitly as UID/GID 65534:65534.

Persistent state

Before first start:

sudo install -d \
  -m 0750 \
  -o 65534 \
  -g 65534 \
  /mnt/data/cmonchoix-infra/observability/alertmanager

The Docker volume cmonchoix-platform_alertmanager_data_v1 is bind-backed by that directory. Never prune or delete it during ordinary deployment or rollback.

Alert routing

The repository-managed monitoring/alertmanager.yml groups alerts by:

  • alertname
  • scope
  • severity

Critical alerts repeat hourly. Warning alerts repeat every six hours. Resolved alerts are delivered as well.

The receiver uses Alertmanager's native discord_configs integration with webhook_url_file; no Discord destination URL is stored in repository configuration.

Egress isolation

The existing observability network remains internal. Alertmanager alone is also connected to alert-egress, which exists only to permit external notification delivery.

Do not attach Prometheus, Grafana, Loki, Promtail, cAdvisor, node-exporter, or business services to alert-egress.

Live certification

A live installation is not certified until all of these are proven:

  1. exact Git release authority and immutable Alertmanager image;
  2. secret metadata without printing its contents;
  3. Alertmanager data directory is on /mnt/data;
  4. Alertmanager is ready and any host exposure is loopback-only on 127.0.0.1:9093;
  5. Prometheus reports exactly one active Alertmanager;
  6. all 14 P1.4B1 rules remain healthy;
  7. a synthetic test alert reaches the configured Discord channel;
  8. a resolved notification reaches the same Discord channel;
  9. business container identity is unchanged;
  10. Docker volume and network inventories change only by the expected Alertmanager volume and alert-egress network.

P1.4B2B — systemd backup / maintenance bridge

The current authority permits host-originated alert publication only through:

http://127.0.0.1:9093/api/v2/alerts

The systemd services ccx-offsite-backup.service and ccx-offsite-maintenance.service use OnFailure to invoke ccx-backup-alert-notify@.service. The helper scripts/backup-alert-notify.sh sends firing/resolved state to Alertmanager only; it has no Discord webhook knowledge and reads no Discord secret.

This preserves one routing authority:

Prometheus alerts ─┐
                   ├─> Alertmanager ─> Discord
systemd failures ──┘

The final live certification observed one synthetic firing and one resolved Discord notification, with notification delta +2, failed counter 0, and no remaining synthetic active alert.

Rollback

Rollback must:

  • restore Prometheus to the previous release authority so it no longer sends to Alertmanager;
  • remove only the Alertmanager container and new runtime network membership;
  • retain the Alertmanager data directory/volume;
  • retain the Discord webhook secret file;
  • never prune Docker volumes;
  • never mutate the historical live checkout.

P1.4B2B is now certified live. Any future change to the systemd bridge or to the Alertmanager delivery boundary requires revalidation of this certification.

See P1.4 certification for the final evidence scope and reopening criteria.