Aller au contenu

P1.4 — Observability and Alerting Certification

Status: CERTIFIED / HISTORICAL EVIDENCE FOR CURRENT P1.4 CLOSURE

Scope

This certificate records the final closure of P1.4 infrastructure observability and alert delivery. It is a certification record, not a replacement for the CURRENT runtime description in docs/operations/runtime-current-state.md.

Certified main revision:

c1b2f6a356f23c1ace064a2a7e4f71c33ffb584e

Final status:

P1.3 Disaster Recovery                  CLOSED / PASS
P1.4A Observability audit               CLOSED / PASS
P1.4B1 Alertability                     CLOSED / PASS
P1.4B2A Discord Alertmanager            CLOSED / PASS
P1.4B2B systemd backup alert bridge     CLOSED / PASS
P1.4 GLOBAL                             CLOSED / PASS

Release authorities

P1.4B2B deliberately did not recreate the already-certified Prometheus runtime.

Prometheus authority SHA
88ae34f6563cf10f75726501d6266405bf28201b

Alertmanager / systemd B2B authority SHA
c1b2f6a356f23c1ace064a2a7e4f71c33ffb584e

Observed release-authority sources during final certification:

Prometheus:
/mnt/data/cmonchoix-release-authority/88ae34f6563cf10f75726501d6266405bf28201b/monitoring/prometheus.yml

Alertmanager:
/mnt/data/cmonchoix-release-authority/c1b2f6a356f23c1ace064a2a7e4f71c33ffb584e/monitoring/alertmanager.yml

The split authority is intentional and must not be collapsed by recreating Prometheus merely to make provenance labels uniform.

Prometheus

Final observed posture:

  • healthy and ready;
  • 14/14 rules healthy;
  • prometheus, node, cadvisor and alertmanager targets up;
  • Alertmanager router exactly http://alertmanager:9093/api/v2/alerts.

Observed Prometheus container ID:

181ac875b492be819a261fb838306837dd09d92e88dc28c281e5cccffec5c63d

Container IDs are evidence only and are not stable configuration identifiers.

Alertmanager

Current P1.4B2B authority:

image:
ghcr.io/prometheus/alertmanager:v0.34.0@sha256:690c7b525f4367aa91f73e2f91c632206d32e97c6384bdbf2fb7a861b420340d

host API:
127.0.0.1:9093 only

persistent data:
/mnt/data/cmonchoix-infra/observability/alertmanager

persistent data metadata:
uid=65534
gid=65534
mode=0750

Observed Alertmanager container ID:

e5c04a0da4a173960d1887217b1e6f99c9b057b3ac75d32c07f58a265759e563

No public exposure such as 0.0.0.0:9093 is part of the certified contract.

Discord secret boundary

The Discord webhook secret remains outside Git:

/etc/cmonchoix/alerting/discord-webhook-url

Required metadata:

uid=0
gid=65534
mode=0640

Observed SHA256 of the secret file during certification:

05cf46b09a37232ae172d703974a9d97553fc11a7b2fb21e8fca35ccf68564f1

The digest is evidence of file identity. The webhook value itself must never be committed or printed in evidence.

systemd B2B authority

The final manifest contains exactly 13 artifacts:

  • 8 backup helpers;
  • 5 systemd units.

Final comparison:

TOTAL=13
B2B_MATCH=13
B2A_MATCH=0
OTHER=0
ABSENT=0

Helpers:

  1. backup-create.sh
  2. backup-critical-state-restic.sh
  3. backup-offsite-daily.sh
  4. backup-offsite-maintenance.sh
  5. backup-offsite-restic.sh
  6. backup-restore-check.sh
  7. backup-verify.sh
  8. backup-alert-notify.sh

Units:

  1. ccx-backup-alert-notify@.service
  2. ccx-offsite-backup.service
  3. ccx-offsite-backup.timer
  4. ccx-offsite-maintenance.service
  5. ccx-offsite-maintenance.timer

Timer state observed:

ccx-offsite-backup.timer       enabled + active
ccx-offsite-maintenance.timer  enabled + active

Failure routing:

ccx-offsite-backup.service
  -> OnFailure=ccx-backup-alert-notify@ccx-offsite-backup.service.service

ccx-offsite-maintenance.service
  -> OnFailure=ccx-backup-alert-notify@ccx-offsite-maintenance.service.service

systemd-analyze verify passed during certification.

External delivery proof

The final synthetic alert lifecycle proved both directions:

Discord firing received       PASS
Discord resolved received     PASS
notification delta            +2
failed counter                0
synthetic alert active after  no

systemd failure notifications and Prometheus alerts converge on the same Alertmanager routing authority. The systemd helper does not know the Discord webhook and does not implement a second external notification path.

Final evidence manifest

VPS evidence manifest:

/home/ubuntu/.local/state/cmonchoix/p1-4b2b-final-certify/20260905T150609Z/evidence.sha256

Manifest SHA256:

b09c31f9972e6dfa39bf74e7b05305ed7305ebce6cb0bd0e8031f11425161cde

This repository certificate records the manifest identity and certification conclusions. It does not import VPS evidence or secrets into Git.

Certification markers

P1_4B2B_TARGET_SHA=c1b2f6a356f23c1ace064a2a7e4f71c33ffb584e
P1_4B2B_ALERTMANAGER_AUTHORITY=B2B
P1_4B2B_PROMETHEUS_AUTHORITY=B2A_UNCHANGED
P1_4B2B_SYSTEMD_ARTIFACTS=13_OF_13_B2B
P1_4B2B_DISCORD_FIRING_RESOLVED=PASS
P1_4B2B_LIVE=CLOSED_PASS
P1_4=CLOSED_PASS
P1_4B2B_FINAL_STATUS=PASS

Scope boundary

P1.4 certifies infrastructure observability, Alertmanager routing, external Discord delivery, and the systemd backup/maintenance alert bridge.

It does not certify:

  • quantitative VPS capacity;
  • business SLO attainment under representative load;
  • pipeline throughput limits;
  • DB latency under benchmark load;
  • frontend p95/p99 under concurrent sync;
  • queue/projection lag capacity limits.

Those belong to the following Capacity & SLO certification work.

Reopening criteria

P1.4 must be revalidated when a change modifies any certified boundary, including:

  • Prometheus alert routing;
  • the 14-rule corpus or its mandatory targets;
  • Alertmanager image/configuration;
  • Alertmanager network or host exposure;
  • Discord receiver configuration or secret-file contract;
  • Alertmanager persistent storage contract;
  • backup/maintenance OnFailure wiring;
  • backup-alert-notify.sh;
  • the 13-artifact systemd authority;
  • any runtime path used by firing/resolved delivery.

A future revision must not inherit CERTIFIED merely because this historical certificate exists.

References