The event path#
- A monitoring source sends an authenticated event to the integration gateway in the API.
- PostgreSQL commits the raw payload and outbox event together before the API returns accepted.
- An outbox publisher sends committed work to NATS JetStream; a consumer normalizes, routes and deduplicates it.
- Incident state and escalation jobs are persisted in PostgreSQL.
- Workers claim due escalation jobs and create notification work; provider attempts and results are tracked separately.
- Responders view the same authoritative state through the Web/API and acknowledge or resolve the incident.
What each dependency owns#
| Component | Responsibility |
|---|---|
| PostgreSQL | Authoritative configuration, raw events, incidents, jobs, notification history and audit. |
| NATS JetStream | Durable transport for asynchronous committed work; not a replacement for the database. |
| Redis | Available in the on-prem profile; no current core correctness dependency. |
| Web/API | Authenticated interaction, ingress validation and authoritative state changes. |
| Workers | Processing, escalation, notification dispatch and reconciliation. |
Concurrency and ordering#
State transitions lock the incident and validate the requested version. Jobs and notification attempts use leases and durable state to support restart/recovery. Source timestamps prevent older alert state from replacing newer observations. Delivery is at least once with database idempotency; this is not a blanket exactly-once delivery promise.
When infrastructure fails#
- Database unavailable: state changes cannot safely commit; readiness fails and clients should retry appropriately.
- NATS unavailable: accepted outbox records remain in PostgreSQL. Broker supervision is separate from SQL workers; reconciliation can recover aged unprocessed raw events.
- Worker restart: pending work remains durable or leases expire for recovery.
- Provider outage: affected notification attempts fail/retry visibly; incident access and response remain available if core dependencies are healthy.
- Whole stream lost: published outbox records are not automatically all replayed. Restore/replay must be coordinated and tested.