From a monitoring signal to a coordinated response#

AckNexa is an incident management and on-call platform. It receives monitoring events, normalizes and deduplicates alerts, routes them to owned services, finds the current responder, and runs escalation policies. Responders acknowledge and resolve incidents through the Web application or the iOS and Android apps, with a shared history of what happened.

Around the response, AckNexa can keep customers informed on status pages, send updates to stakeholders, open Jira or Halo tickets, route hotline calls to whoever is on call, and record a postmortem with action items.

The core runs on customer-controlled infrastructure. You do not need a vendor-hosted AckNexa control plane, a public cloud account, or an external notification API to ingest incidents, coordinate response, and use the Web UI.

Choose your path#

The product model#

What this public site is#

This site is product documentation, not a hosted AckNexa tenant, customer portal, or live status page. AckNexa status pages are served by your own installation. This site does not require a login and does not connect to an installation. Examples use fictional data. Your deployment administrator supplies the actual application address and access.

To reach the AckNexa team, write to [email protected].

Before you begin#

• Ask your platform operator for the application URL and first-run setup token. The token must be supplied by the operator; there is no public self-registration or universal production password.
• First-run setup creates only the first organization of a deployment. The operator creates further organizations with the bootstrap command in the backend image; see Users and access.
• Confirm PostgreSQL migrations and the API, workers, and Web application are running. See Deployment.
• Choose an organization name and slug, an IANA timezone such as Europe/London, and an initial administrator.
• Identify at least one responder and a notification channel that works in your network. Internal SMTP is an option for isolated installations.

Create the administrator#

1. Open the Web application. An empty installation shows first-run setup only when the operator has enabled it with a setup token.
2. Enter the setup token, organization details, administrator email/display name, timezone, and a strong password. Passwords must be 12–72 bytes.
3. Create the organization, then sign in using its slug, email, and password. The organization administrator sees configuration navigation.
4. Have the operator retire the setup token after initialization. It is not an ongoing user invitation or tenant registration mechanism.

Build the response chain#

1. In Users, create active responders and set their roles, timezone, contact details, and notification preferences.
2. In Teams, create the owning team and assign its members.
3. In On Call, create a primary schedule with a rotation that covers the current time. Check the selected responder.
4. In Escalation Policies, create a policy with at least one valid target, delay, and available channel.
5. In Services, create an active service, assign its team and policy, and add a runbook URL if available.
6. Configure a Notification Provider or ask the operator to supply a deployment-level provider. Send a provider test to the intended responder.
7. Create an Integration for the service. Copy its webhook secret immediately and store it securely; it cannot be read back as plaintext.

Prove the whole workflow#

1. Use the integration's test action with a representative payload, or send a real authenticated webhook from your monitoring source.
2. Follow the test's processing state. Inspect the created incident, matched routing information, on-call responder, escalation history, and notification result.
3. Have the actual responder acknowledge the incident. Provider acceptance is not the same as a human acknowledgement.
4. Complete the guided onboarding flow after its prerequisites are satisfied, then resolve the test incident with a clear reason.
5. Repeat using your real monitoring source and selected notification channels before enabling production alert traffic.

If setup or admin navigation is missing#

Setup is not available after an organization exists. Admin navigation requires organization_admin; a team_admin is not an organization administrator. After a role change, sign out and back in. If the UI is old, rebuild and recreate the relevant Web/API images: restarting a container does not rebuild it. See Troubleshooting.

Everyday response#

Configuration#

Administration#

Refresh, permissions, and stale state#

The backend is authoritative. An acknowledgement is successful only when the API confirms it. Concurrent actions may return a conflict; refresh the incident and inspect its new version before retrying. Hidden controls do not grant access, and a visible URL cannot bypass backend authorization.

Triage with context#

1. Open the incident from the incident list or notification link. Confirm the service, severity, affected resource, environment, and source timestamps.
2. Review the owning team, current on-call responder, runbook, related alerts, and matched routing rule. A repeated alert is not necessarily a new incident.
3. Inspect the timeline and notification history before acting. Another responder may already have acknowledged or resolved it.

Incident states#

Acknowledge#

Select Acknowledge when you are taking responsibility. AckNexa records the actor and time and cancels pending escalation jobs and unsent notifications in the same transaction. A notification request already in flight can still complete; a late message does not necessarily mean the ACK failed.

Escalate and resolve#

Use the permitted escalation or assignment controls when more help is needed. Check that the destination is an active responder or valid policy target. Changes are visible in the incident history.

Resolve only when the response can be closed, and provide a meaningful resolution reason. A valid source recovery can also resolve the lifecycle. Older, out-of-order source data must not replace newer state. Manual resolution does not make a still-firing monitor healthy.

Notes, updates and tickets#

• Add notes to record what you checked. Notes stay internal unless you tick Send to stakeholders, which posts the note as an update to everyone following the service or team. See Status pages and stakeholders.
• Publish a customer-facing update from the incident's Status page update panel when the service is listed on a status page.
• Open a Jira or Halo ticket with Create ticket when your organization has a ticketing connection. See Ticketing.
• After resolution, start the review from the incident. See Postmortems.

Severity and ownership#

AckNexa normalizes severity to P1 through P5, with P1 highest priority. Agree on a severity mapping for your organization and integrations. Business criticality describes the service; it is not a substitute for incident severity. Acknowledge and resolve rights depend on your role, not on merely receiving a notification.

A good handoff#

• Link to the incident instead of copying credentials or private payloads into chat.
• Explain symptoms, checks already performed, the relevant runbook, and the next expected action.
• Confirm that the receiving responder has actually acknowledged. Provider acceptance or a ringing phone is not an acknowledgement.
• Keep sensitive information out of alert summaries when it is not needed for response.

Three different records#

Repeated and out-of-order events#

A stable integration idempotency key prevents transport retries from creating a second raw event. Fingerprinting separately identifies repeated observations of the same condition. These are different layers of duplicate protection.

Source timestamps determine precedence; equal-timestamp recovery wins over firing. A persisted receipt timestamp is used when the source omits time information. Preserve stable identity fields and consistent timestamps across firing and recovery events.

Reading processing status#

• Accepted: the raw payload and outbox record are committed, but processing may still be pending.
• Processed: normalization and incident handling completed for that raw event.
• Quarantined: processing could not safely complete; inspect Delivery Health and the source payload.
• Recovery: the source reports the condition is no longer firing; verify its identity and timestamps match the intended lifecycle.

Protect the event history#

Do not delete raw events or rewrite database state to fix a live incident. Diagnose the source/parser/configuration issue and use the audited retry path where appropriate. Changes to integration identity fields with existing history may require a new integration. See Routing & deduplication.

Build a schedule#

1. An organization administrator opens On Call and creates an active schedule for the owning team.
2. Set an IANA timezone and select primary or secondary tier.
3. Add a rotation layer with start time, optional end time, rotation interval, and ordered active participants.
4. Use distinct layer priorities when multiple layers overlap. Higher-priority active layers take precedence.
5. Save and inspect the current on-call result. Then connect the schedule to an escalation policy and verify a test incident.

Rotation intervals and local time#

Intervals are durations in seconds, not a promise of a fixed local-clock handoff across daylight-saving changes. Explicitly check the intended handoff times, timezone, end date, and participant order. Choose intervals that fit your team's operating model and test DST boundaries if applicable.

Temporary overrides#

1. Choose the schedule and create an override for a specific active user, time window, and reason.
2. Review the displayed boundaries in the intended timezone. Check that the override covers the full shift.
3. Verify the effective on-call responder during the override. Remove an obsolete override through the supported action rather than changing historical event data.

Avoid coverage gaps#

• Keep the team, schedule, and participants active.
• Rotation and override participants are paged only while they are an acting member (responder, team admin or organization admin) of the schedule's team. Organization viewers are never paged. If a schedule shows nobody on call, check team membership first.
• Ensure at least one rotation is effective now and at future handoff boundaries.
• Review primary and secondary coverage independently.
• Do not rely on sampled UI previews to prove every minute of future coverage. Continuous coverage certification is not yet an exhaustive timeline feature.
• Re-test after deactivating a user or changing a schedule referenced by policies.

Create a policy#

1. Open Escalation Policies as an organization administrator and create an active policy.
2. Add ordered steps with a delay, target type, and one or more channels.
3. Choose a user, team, schedule, primary on-call, secondary on-call, or team-admin target as supported by the form.
4. Configure repeats deliberately. Then assign the policy to an active service.
5. Send a test event and check that the effective responder, timing, and channels match your intent.

Choose realistic targets#

A policy can be syntactically valid but operationally useless if the referenced schedule has no coverage, the target is inactive, or the responder lacks a usable destination. Channel preferences and provider configuration are separate from selecting a channel in a policy.

Durable execution#

Escalation jobs live in PostgreSQL and are claimed by workers. Worker restarts do not erase pending jobs. Acknowledgement or resolution cancels pending escalation work. Notification attempts are tracked separately, with bounded retries and visible failure outcomes.

Pause paging with a maintenance window#

Schedule a maintenance window from the Overview while a deployment or planned work runs. A window covers one service, or every service when an organization administrator creates it; team admins can schedule windows for services their team owns. A window lasts at most 14 days and must start within 90 days.

Incidents created during the window are recorded but not paged: they start acknowledged and snoozed until the window ends, and anything still open when it closes is paged then. Ending a window early releases its held incidents at once. Status pages that list an affected service show it as under maintenance while the window runs.

Delivery guarantees and limits#

Provider calls cannot participate in a database transaction. Notification attempts are at least once: a crash after a provider accepted a message but before success was recorded can cause another attempt. Stable identifiers help, but exactly-once handset delivery is not guaranteed. Provider acceptance is never proof of human response.

Web access is a core workflow#

Responders can sign in, review incidents, acknowledge, and resolve through the customer-hosted Web application. Use the deployment's HTTPS origin and trusted certificate chain. Browser-based response does not require APNs, FCM, or a public cloud service.

iOS and Android apps#

AckNexa has native iOS and Android apps. Both list incidents with live updates, acknowledge and resolve through the API, show your on-call shifts and cover requests, your team and your notification history, and open the incident when you tap a page. An action is shown as successful only after the API confirms it. Session tokens are stored per server: in the iOS Keychain, and encrypted with the Android Keystore.

On Android, P1 pages use a separate alarm channel that can show a full-screen notification when the phone allows it; other pages use a high-importance channel.

How the apps are distributed#

For release 1.0.0-rc.1 neither app is published in the App Store or Google Play. The operator builds them from source: the iOS app with Xcode and their own Apple developer account (TestFlight can distribute it to a pilot group), and the Android app with Android Studio or Gradle, their own Firebase project and their own signing key. Signing, device trust and real-device testing are deployment responsibilities.

How pages reach a phone#

Through the relay, the worker sends only an incident reference: the notification and incident IDs, the incident number and severity, the device token, and the server's address and name. No alert title, service name or body text leaves the customer network; the app fetches the details from the server itself. Each relay credential is pinned to the deployment's own server address or addresses, and the relay refuses a push that names any other origin, so one installation cannot label pushes as another. An organization uses either the relay or its own APNs key, not both. The relay model is intended for store-distributed apps, which are not published for this release.

Several servers on one phone#

Each app keeps one signed-in account per server address. A push names the server it came from, and tapping it switches to the matching account; a push from a server the phone is not signed in to says so and pre-fills the sign-in screen.

Inside a private network#

• Devices need a route to the private API, directly or through the customer's approved VPN.
• Private DNS and the private CA must work on the device, not just on servers.
• APNs and FCM require connectivity to Apple and Google. An isolated device cannot receive pushes merely because the API is reachable.
• A lost network connection during ACK is not a successful response. Reconnect and refresh to verify the server state.

Keep a fallback channel#

What a status page shows#

A status page is a curated, customer-facing view of selected services. Only the services an administrator lists appear, under the public names they choose, and an incident appears only once a responder publishes a status update from it. The internal incident title and number never appear on the page.

The Web application serves each page at /status/SLUG without a login. A page is public, or private with a secret link (?key=…) that is shown once when you save it and can be rotated. To use a separate hostname such as status.example.com, point it at the Web application and rewrite / to /status/SLUG in your reverse proxy.

Create a status page#

1. Open Status Pages under Configure as an organization administrator and create a page with a name and URL slug.
2. Tick the services to list and give each the name the public should see, for example Checkout rather than an internal service name.
3. Choose public or private. For a private page, copy the secret link when it is shown.
4. Share the URL and check it from the network your audience uses.

Publish an update during an incident#

Open the incident, expand Status page update, pick a state (Investigating, Identified, Monitoring or Resolved) and write a sentence for customers. Optionally set a public title of up to 120 characters; it names the incident on the page and in subscriber emails, and later updates keep it unless you change it. Without one, the service's public name is used.

• The update appears on every status page that lists the incident's service.
• The service shows as Major outage for P1 and P2 incidents, and Degraded performance for lower severities.
• The incident timeline records the update, and every subscriber of those pages is emailed.
• If an incident with published updates is resolved without a final update, the worker publishes "This incident has been resolved." Resolved incidents stay on the page for 14 days.
• Maintenance windows that cover a listed service appear under Scheduled maintenance and show the service as under maintenance while they run.

Publishing needs the responder, team admin or organization admin role, the same as acknowledging.

Email subscribers#

Each page has an email subscribe form. A subscriber receives one email per published update with the page link and a one-click unsubscribe link. Subscribing twice to the same page changes nothing, and the API accepts at most 200 new subscriptions per page per hour. Emails are plain text and deliberately minimal.

What status pages need#

Stakeholders#

Responders get paged; stakeholders get told. A stakeholder follows a service or a team and receives an informational email, and a push for app users, when incidents open, are acknowledged, get an update or resolve. Stakeholders are never part of an escalation policy: organization viewers and plain email addresses outside AckNexa both work.

1. Edit a team or service and open Stakeholders. Team admins manage the lists for their own teams.
2. Add a person in AckNexa or type an email address, and pick the moments they hear about: opened, acknowledged, updates posted, resolved.
3. For app users, choose whether to push as well.

A stakeholder who is an AckNexa user can open the incidents they follow in the Web application and the apps, read-only unless their team role allows acting. Deliveries are retried up to six times, and the Stakeholders panel shows how many went out in the last 30 days per person.

Post a stakeholder update#

In the incident drawer, tick Send to stakeholders before posting a note. The note becomes an update on the timeline and is emailed to everyone following the service or its team with updates enabled. Ordinary notes stay internal. Outbound webhooks carry the same update event, so another system can mirror it.

One review per incident#

A postmortem records what happened, why, how it was detected and resolved, and the action items that stop it recurring. Postmortems, under Respond, lists what is due, in review and published. Everyone who can see an incident can read its review; responders, team admins and organization admins can write one. Resolved incidents offer Start postmortem or Open postmortem.

Automatic drafts and reminders#

The “Postmortem expected from” setting on the Organization page under Settings sets the severity threshold. The default is P2, so every resolved P1 and P2 incident is listed as waiting for a review until one exists; none turns the reminder off.

• Every minute, the worker creates a draft for incidents resolved in the last 24 hours that meet the threshold and have no review yet.
• The owner is the incident's assignee, otherwise whoever acknowledged it, otherwise nobody. Service accounts are never owners.
• The due date is the resolution time plus Postmortem due after (days): default 5, from 1 to 60.
• When a review is past due, the owner is emailed through the organization's email provider, or the owning team's team admins when there is no owner. The reminder repeats every 3 days until the review is published.

Write the review#

The editor opens with a draft prefilled from the incident: a title, when and how it was detected, time to acknowledge, time to resolve, and the full timeline beside the form with notifications filtered out. The sections are Summary, Impact, Root cause, Detection, Resolution, What went well, What went badly and Lessons learned. Each action item has a text, an optional owner, a due date and a done box.

Review statuses#

Each status change is recorded on the incident timeline and in the audit log. Export Markdown downloads the review with its timeline for a wiki or a ticket.

Organization roles#

Create and maintain a user#

1. In Users, create the person with their email, display name, timezone, role, and active state.
2. Set an initial password if password login will be used, and share it through an approved secure process. There is no email invitation workflow in this release; with single sign-on, directory sign-in or SCIM, the identity provider can create accounts instead.
3. Add team memberships and configure available contact details and notification preferences.
4. Test sign-in and delivery as the intended user. After changing a role, sign out and back in to refresh the navigation.

Deactivation and least privilege#

Use the lowest sufficient role. Before deactivating a responder, update schedules and escalation targets to preserve coverage. Inactive users cannot continue normal authentication. Organization administrators should review team membership, active users, and provider access regularly.

Sign-in methods#

The methods can be mixed per organization, and password sign-in keeps working while SSO or LDAP is enabled. Sign-in uses the organization slug. With Create accounts automatically, a first directory sign-in creates the user with the configured default role; otherwise an administrator adds the account first. Group mappings tie directory groups to teams, team roles and organization roles, and are applied at each sign-in. An SSO account is pinned to its issuer and subject on first sign-in, and later mismatches are refused.

Changes in the directory reach AckNexa at the next sign-in, and a person disabled in the directory keeps an existing session until it expires. Deactivate the user in AckNexa as well, or use SCIM, when a departure is immediate.

SCIM provisioning#

SCIM 2.0 lets Okta, Microsoft Entra ID or Keycloak create, update and deactivate AckNexa users and keep group memberships current, so a departure stops paging at once instead of at the next sign-in. SCIM provisions accounts and groups; people still sign in through SSO, and SCIM-created users have no password.

1. Open Provisioning (SCIM) under Identity, enable provisioning and choose the role for new SCIM users: team admin, responder or viewer (default responder).
2. Generate the SCIM token and copy it; it is shown once and only its hash is stored. Rotate replaces it immediately and Revoke removes it.
3. Configure the identity provider with the base URL PUBLIC_API_URL/scim/v2 and the bearer token, mapping the user's email address to userName.
4. Assign test users and groups, then check the last change and last error shown on the Identity page.

Deactivation never deletes a user: it marks the account inactive, revokes every session, disables the person's mobile device tokens and stops them being paged, in one transaction. AckNexa refuses to deactivate the last active organization administrator. SCIM groups use the same group mappings as SSO and LDAP, so use one source of truth for groups. Bulk operations, sorting, ETags and password changes are not supported.

Break-glass accounts#

Keep at least one break-glass account: a local organization administrator with a password, marked Break-glass account in Users. Only organization administrators can set the flag, from a signed-in session, and only on an active organization administrator with a local password.

• SCIM can read the account but every write to it is refused with 403 and audited, including the deactivation an identity provider sends on unassignment.
• Group mappings never change its role or team memberships.
• An SSO or LDAP sign-in with its email is refused and audited; it signs in with its password only.
• While the flag is set, it cannot be demoted or deactivated from the Users page, and API keys cannot edit it.
• Identity warns when SCIM is on without an active break-glass account, and when one has not signed in for 90 days. Sign in with it now and then so you know the password still works.

Organization boundary#

Tenant-scoped API queries and database relationships use the organization boundary. A valid account in one organization does not grant access to another organization's resources. The current read model is not a promise of per-team concealment of every incident inside a shared organization.

Several organizations on one deployment#

One deployment can host several organizations; users, teams, services, integrations, incidents and device registrations are all scoped to their organization. First-run setup creates only the first organization. The operator creates each further organization, with its first administrator, using the bootstrap command in the backend image; it also creates a default team, escalation policy, service, schedule and four integrations, and prints their webhook secrets once. There is no API or Web screen for adding an organization, and no organization delete.

Create a team#

1. Open Teams as an organization administrator. Supply a clear name, stable slug, description, and active state.
2. Add users with appropriate team memberships. A team's administration role is separate from organization-wide administration.
3. Check that people referenced by schedules and policies remain active.

Create a service#

1. Open Services and define the service name, slug, environment, description, and business criticality.
2. Assign its owning team and an active escalation policy.
3. Add a runbook URL and useful metadata. Prefer customer-controlled runbooks for isolated environments.
4. Configure severity mappings when source conventions differ from your P1–P5 response model.
5. Connect an integration, then verify that an incoming alert appears under the intended service.

Configuration is connected#

Changing a team or service can affect the integrations, routing rules, schedules, and policies that reference it. Inspect active dependencies before deactivation. Do not remove historical database records to make a configuration form accept a change.

Stakeholders, chat and alert sources#

Edit a team or service to add stakeholders, a chat channel and, for services, filtered alert sources and severity mappings. A service's chat channel overrides its team's. Team admins manage stakeholders and chat channels for their own teams.

Useful service context#

• A runbook should explain how to diagnose the monitored condition, what actions are safe, and when to escalate.
• Keep metadata descriptive rather than storing secrets in it.
• Use environment names consistently in monitoring payloads, rules, and service records.
• Business criticality describes importance; normalized alert severity determines response priority.

Supported inbound sources#

Email intake and SNMP traps are not available yet.

Create and secure an integration#

1. Open Integrations and choose the source kind, owning service, name, enabled state, and authentication mode.
2. Copy the webhook path and newly issued secret. The stored secret is hashed, so plaintext recovery is not available later.
3. Configure the sender's exact deployment API URL. Keep secrets out of URLs, logs, screenshots, and shared shell history.
4. Use token/HMAC mode or HMAC-only mode as required by policy.
5. Send a representative test, inspect processing status and routing, then verify acknowledgement.

Generic payload#

{
  "alert_name": "DatabaseUnavailable",
  "resource": "db-01",
  "environment": "production",
  "severity": "critical",
  "status": "firing",
  "summary": "Database health check failed",
  "started_at": "2026-09-21T09:00:00Z",
  "labels": { "team": "platform" }
}

Use your actual current source timestamp; the example is illustrative. Send recovery with the same stable identity and appropriate source lifecycle timestamps. Test the exact payload shape for your integration kind.

Authenticated request#

curl --request POST "https://api.example.internal/integrations/webhook/INTEGRATION_ID" \
  --header "Content-Type: application/json" \
  --header "X-AckNexa-Key: YOUR_WEBHOOK_SECRET" \
  --header "Idempotency-Key: YOUR_STABLE_EVENT_ID" \
  --data-binary @event.json

Signature v2 (recommended)#

Sign TIMESTAMP + '.' + IDEMPOTENCY_KEY + '.' + RAW_BODY with HMAC-SHA256 using the webhook secret itself as the key, where TIMESTAMP is the X-AckNexa-Timestamp header exactly as sent and IDEMPOTENCY_KEY is the Idempotency-Key header (empty if absent). Send X-AckNexa-Signature: v2=HEX_SIGNATURE. The server stores the secret encrypted with the deployment configuration key, so database or backup access alone cannot forge requests, and replaying a request under a new idempotency key fails. Choose 'Require signature v2' on the integration to refuse tokens and v1 signatures; rotate older secrets once first.

HMAC mode (v1, legacy)#

Sign RFC3339_TIMESTAMP + '.' + RAW_BODY with HMAC-SHA256 using SHA256(webhook_secret) as the binary HMAC key. Send X-AckNexa-Timestamp and X-AckNexa-Signature: sha256=HEX_SIGNATURE. The timestamp must be within five minutes. Sign the exact bytes transmitted; JSON reformatting after signing changes the signature. When a signature is supplied, it must validate even if a token is also supplied.

Rotation and retirement#

Rotate the integration secret through the supported action, update the sender, and verify a successful event. Do not assume a grace period for the previous secret. Archive an integration when retiring it; retain its history. Identity-affecting changes on an integration with history can require creating a new integration.

Heartbeat monitors#

A heartbeat turns silence into a page: a backup that never ran, a cron job that hung, a pipeline that stopped.

1. Open Heartbeats under Configure and create a heartbeat with its service, the job's interval, a grace period and the severity to raise.
2. Copy the ping URL, which is shown once, and call it at the end of the job. GET, POST and HEAD all count.
3. If no ping arrives within the interval plus grace, an incident opens on the service and its escalation policy pages the team. The next ping resolves it.

curl -fsS -m 10 "https://api.example.internal/heartbeat/YOUR_HEARTBEAT_TOKEN" >/dev/null

Heartbeat incidents use the normal alert pipeline, so routing rules, maintenance windows, deduplication and auto-resolution apply. Pausing a monitor stops the clock; rotating the token invalidates the old URL immediately.

Outbound webhooks#

Webhooks, under Configure, sends selected incident events (created, acknowledged, escalated, assigned, note, update, snoozed, resolved) to your own HTTPS endpoint, for the whole organization, one team or one service. The host must be in the deployment's NOTIFICATION_WEBHOOK_HOSTS allowlist, and private addresses also need NOTIFICATION_PRIVATE_CIDRS. Each request carries X-AckNexa-Signature, an HMAC-SHA256 over the timestamp and body with the secret shown once at creation; verify it and reject old timestamps. Deliveries retry with backoff, and a retry after a timeout can repeat a delivery your endpoint already processed.

How routing is selected#

An integration supplies the default service. Enabled routing rules are evaluated in position order, with stable tie-breaking. The first matching rule selects a service; if no rule matches, the integration default remains. Rules may be scoped to an integration or apply more broadly within the organization.

Rule conditions#

Conditions are exact string matches, not arbitrary expressions. All supplied conditions must match. Supported condition fields include source, alert_name, environment, resource, raw source severity, labels.KEY, and annotations.KEY. The severity condition uses the source's severity value; normalized P1–P5 mapping is a separate step.

{
  "environment": "production",
  "labels.team": "platform"
}

Put specific rules before broad fallback rules. Use the integration test and incident routing context to verify the matched rule and destination.

Stable identity across recovery#

Routing binds source alert identity to the selected service. Editing a rule does not automatically move an already-bound condition and strand its recovery. Test a new alert identity when validating a changed rule, and plan migrations of existing integrations deliberately.

Two kinds of deduplication#

Severity mapping#

Source normalization establishes a severity. Service mappings can override it, and integration mappings take precedence over service mappings for matching source values. The organization default applies when source severity is absent. Validate both routing conditions and final P1–P5 severity with realistic payloads.

Common mistakes#

• Putting a changing summary, random request ID, or ingestion timestamp in the fingerprint.
• Using different resource/environment identity on firing and recovery.
• Expecting exact-match conditions to behave like regexes.
• Testing a changed rule only with an existing pinned alert identity.
• Treating a duplicate-accepted response as evidence that the notification reached a human.

Three layers to check#

1. Provider: the organization or deployment must have a usable adapter and credentials.
2. Responder: the recipient must be active and have valid destinations and preferences.
3. Policy: the escalation step must select the intended target and channel.

Connectivity and availability#

See Web and mobile access for the APNs, FCM and push-relay options.

Configure and test#

Organization administrators can save supported provider settings and run an audited provider test. Saved secret fields are redacted. Leaving a secret field blank during an edit preserves the existing secret; it is not a delete operation. Organization-level configuration overrides deployment defaults for that organization.

Disabled, unreadable, or invalid organization configuration fails closed rather than silently falling back to a different account. Correct the override before relying on a deployment-level adapter.

Team chat channels#

A team, or a single service that overrides its team, can post incident events to a Slack, Mattermost or Microsoft Teams channel: opened, paged or escalated, acknowledged and resolved. Messages carry Open and Acknowledge links. The Acknowledge link opens AckNexa, signs the responder in if needed and acknowledges as that person, so the audit trail stays honest without a bot token or inbound chat app.

Edit the team or service, open Chat channel, pick the kind, paste the incoming-webhook URL, choose the events and send a test. Slack's hooks.slack.com is always allowed; a Mattermost or Teams host must be listed in NOTIFICATION_WEBHOOK_HOSTS. Delivery is asynchronous with retries, and posting failures never delay paging. Webhook URLs are encrypted at rest when ACKNEXA_CONFIG_KEY is set.

Internal SMTP#

Use the customer's SMTP server with mandatory STARTTLS and a trusted certificate. For tenant-configured private destinations, the operator must allow the exact DNS hostname and its private CIDR. Loopback, link-local, and metadata endpoints remain blocked. Private webhook relays use the same operator-controlled private-network guard. See Security & identity.

Delivery evidence#

A provider-accepted result means the provider accepted the request. It does not prove inbox arrival, handset presentation, a completed call, or human acknowledgement. Inspect individual attempts and the incident timeline, and test with real recipients. Unknown outcomes can occur after a lease expires or a response is lost; duplicate delivery remains possible.

Phone and SIP destinations#

Use a destination supported by the selected adapter. A SIP address sent through Twilio still depends on Twilio and does not become an internal PBX call. For private telephony, use the customer voice-gateway contract.

How a call flows#

1. The caller hears the route's greeting and, if the route has a menu, picks a service with a digit. No digit within 8 seconds selects the default service.
2. AckNexa opens an incident on the chosen service with the route's severity. Its escalation policy starts as for any alert, so other channels page in parallel with the phone ringing.
3. AckNexa rings, in order, whoever is on call for the service's owning team by schedule tier, then the team's admins. Only people with a phone number on their profile are rung, each for the route's ring timeout (default 25 seconds).
4. Before the call is bridged, the responder hears a short whisper and must press a key, so a personal voicemail box cannot take the call.
5. Answered: the incident is acknowledged by that responder. Nobody answered: the caller can leave a voicemail if the route allows it, the recording link is added to the incident, and the incident stays open and escalating.

Recent calls, on the Call Routing page under Configure, lists every call, who was tried and the outcome. Routes and the call log are for organization administrators.

Choose how the number connects#

Connect your own PBX#

Create the route with provider My own PBX. AckNexa shows a bearer token (cr_…) once; rotate it from the route card. For each call, the PBX first asks who should ring:

POST /v1/voice/route/ROUTE_ID
Authorization: Bearer YOUR_ROUTE_TOKEN
Content-Type: application/json

{"from": "+13035550199", "digit": "1", "provider_call_id": "pbx-call-id"}

The response names the service and incident, the ring timeout, whether voicemail is on, and the ordered list of people with their phone numbers. The incident already exists at that point. When the call ends, the PBX posts the result to /v1/voice/route/ROUTE_ID/result with status connected, missed or voicemail; connected acknowledges the incident for the user who answered, while missed and voicemail leave it open and note the outcome. A PBX that plays the menu itself can read the route configuration with GET on the same path.

Cisco CUCM gateway#

The gateway has two parts: the acknexa-voice-asterisk image, an Asterisk with one SIP trunk to CUCM that accepts SIP only from the CUCM addresses you list and has no phones, registrations or management interfaces; and the voicegateway command in the backend image, which asks AckNexa who to ring, tells Asterisk what to do and reports the result. Deploy it with the deploy/onprem/voice-gateway.compose.yml overlay or the Helm voiceGateway values. Plain CUCM is enough; no UCCX, CUBE scripting or IVR licence is involved.

• CUCM 12.5, 14 or 15, with administrator access.
• A host or Kubernetes node with a fixed IP that CUCM can reach on SIP 5060 (UDP and TCP), plus the RTP range (UDP 10000-10099 by default) from phones, PSTN gateways and media resources.
• PSTN access from CUCM. Responders are called on the numbers on their AckNexa profile, in international format.
• G.711 between the gateway and the phones or PSTN gateways; the gateway does not transcode G.729.
• A call route with provider My own PBX, its ID and its token.

Run voicegateway check after deployment to confirm the route. If AckNexa cannot be reached, the caller hears that the on-call team could not be reached and can be put through to a fallback number (VOICEGW_FALLBACK_NUMBER). Calls in progress are dropped when the gateway restarts.

Call routing is not voice paging#

Call routing answers inbound hotline calls. Voice notifications from escalation policies are separate: they use Twilio or the internal voice gateway contract, which still needs a customer-supplied bridge to CUCM, SIP or the PBX.

What it does#

AckNexa opens a ticket for an incident in Jira Cloud, Jira Server / Data Center, or Halo (HaloITSM / HaloPSA), links it from the incident, mirrors acknowledgements, notes, stakeholder and status page updates, reassignment and the resolution as comments, and can move the ticket to its resolved state. There is one ticket per incident per tool, never one per alert.

Connect a tool#

Open Ticketing under Configure as an organization administrator and add a connection. Credentials are write-only and stored encrypted with ACKNEXA_CONFIG_KEY. Test connection signs in and loads the projects, issue types, ticket types, priorities and statuses the account can see, so rules are picked from lists. The account needs permission to create issues or tickets, add comments and, for the resolve action, transition them.

Rules per team#

Add one rule per team; a rule on a single service overrides its team's. A rule chooses the connection, the Jira project and issue type or the Halo ticket type, optional Jira components and labels or Halo team, agent, client and category, a priority per severity, the transition or status to apply on resolve, and whether to mirror comments.

It also chooses when to open the ticket: only when a responder asks, when the incident opens, or when it is acknowledged, from a minimum severity. Teams without a rule get no automatic tickets, but anyone who can acknowledge an incident can still open one with Create ticket, using the connection's defaults.

Ticket closure flowing back#

By default, closing a ticket changes nothing in AckNexa. Tick Sync ticket closure on the connection to get a webhook URL, shown once, and configure that URL in Jira (an Issue updated webhook) or Halo (a ticket status-change webhook). Then tick Resolve the incident when the ticket is closed on the rule (default off).

When a linked ticket reaches a closed state, the incident timeline always records it, and if the rule opted in, the system resolves the incident with the ticket as the reason, cancelling escalation and pending notifications. The webhook is idempotent and does not loop with the outbound sync. It only affects incidents that have a ticket on that connection.

Security and network#

• The webhook token is a random value stored only as a hash; a wrong token, a disabled webhook and an unknown connection all get the same 401. Senders that can set headers can send the token in X-AckNexa-Token or a bearer header instead of the path; the query string is never accepted.
• Because the path form is part of the URL, keep reverse-proxy access logs for /v1/ticketing/inbound/ off or redacted.
• AckNexa needs outbound HTTPS from the API and worker to the tool. TICKETING_HOSTS limits the hosts a connection may use, and TICKETING_ALLOW_HTTP=true permits http:// for an internal Jira on a trusted network.
• The tool needs to reach the API only if Sync ticket closure is on.
• Failed creation and comments are retried with backoff; credential and configuration errors stop at once and are reported on the incident timeline.

Audit records#

The Audit area shows organization-scoped administrative and response actions, including the actor, action, target, request context, and time where recorded. Provider configuration is redacted rather than exposed as plaintext. Normal runtime access cannot edit audit rows; the database also has an immutability trigger.

Export the audit log#

Export, on the Audit page, downloads a date range as JSON Lines or CSV, at most 366 days per download; export longer periods in parts. Rows arrive in time order and are streamed, so a failed read aborts the transfer instead of producing a silently truncated file. CSV cells that a spreadsheet could evaluate as a formula are prefixed with a quote; prefer JSON Lines for machines. Exports are themselves audited.

Passwords, tokens, secrets, credentials and private keys never appear in an export or a stream: values under keys whose names contain password, secret, token, credential, private_key or webhook_url are replaced with [redacted].

Stream to a SIEM#

Each organization can have one stream, under Audit stream on the Audit page. It delivers every record written after the stream is saved; use Export for earlier history.

Delivery is at least once: the cursor moves only after the collector accepts a batch, failures are retried with backoff of up to 10 minutes, and one failing batch holds back the ones after it. De-duplicate on the event id. The stream shows the last error and the number of events waiting, and a test action sends one synthetic event without moving the cursor. Private collector addresses also need NOTIFICATION_PRIVATE_CIDRS.

Delivery Health#

Organization administrators use Delivery Health to inspect pending/unpublished events, processing failures, overdue or failed escalation work, exhausted notifications, and quarantined raw events. This view helps diagnose the tenant's processing path; it is not a replacement for independent infrastructure monitoring.

Retry a quarantined event#

1. Inspect the reported failure and relevant source payload using authorized operational access.
2. Correct the integration, parser, or configuration problem. Do not change the historical payload just to make the record disappear.
3. Use the supported dead-letter retry action. It writes an audit record and queues the original event.
4. Check processing and incident history. If the underlying issue remains, the event may be quarantined again.

Interpret failures carefully#

• No notification record: inspect routing, active policy targets, schedule coverage, and worker progress.
• Failed notification: check destination, provider configuration, credentials, TLS and network reachability.
• Unknown attempt: the provider may have accepted the request; do not assume no delivery occurred.
• Delayed backlog: inspect database capacity, worker health and broker state before changing application data.

Retention#

Workers prune operational data hourly: expired sessions and login state, outbox rows older than RETENTION_OUTBOX_DAYS (default 7), payloads of processed raw events older than RETENTION_RAW_PAYLOAD_DAYS (default 30; 0 keeps them), and finished delivery-queue rows older than RETENTION_DELIVERY_DAYS (default 30). Pending deliveries are never pruned. Raw event rows, incidents, timelines, notifications and audit events are kept; archive or purge those under your own records policy.

Reports offers basic incident and paging charts; it is not a full analytics suite, and the audit export is not a turnkey compliance product. UI lists are bounded; preserve authoritative history through supported backups.

Deployment models#

Core components#

The API contains the integration gateway. Workers contain the escalation and notification engines. PostgreSQL is authoritative; NATS JetStream transports committed outbox events. Redis is included in the on-prem Compose profile but is not required for incident correctness. The Web UI talks to a server-side API proxy.

Prepare the environment#

1. Obtain the approved source/release artifacts through your distribution channel. No public repository or license terms are asserted by this guide. Release images (backend, Web and the voice-gateway Asterisk image) are multi-architecture (linux/amd64 and linux/arm64), signed, with SBOM attestations; every binary reports its release with --version.
2. Provision private DNS, trusted certificates, stable time synchronization, storage, and backup destinations.
3. Build or mirror approved backend, Web, PostgreSQL, NATS, Redis, ingress and selected observability images. Use immutable digests where appropriate.
4. Prepare secrets locally, including the database and broker credentials, encryption key, and first-run setup token.
5. Run migrations before serving application traffic, then verify the first organization workflow. Both deployment profiles run the paging upgrade check before migrating.

Small-install Compose#

cp deploy/onprem/.env.example /secure/path/acknexa.env
bash scripts/onprem_init.sh /secure/path/acknexa-config

# Configure approved images, paths, private DNS and TLS files first.
docker compose --env-file /secure/path/acknexa.env \
  -f deploy/onprem/compose.yml --profile observability up -d --wait

Run these commands from the product repository. Initialization refuses to overwrite an existing configuration directory. Supply ca.crt, tls.crt, and tls.key from customer PKI, update ingress hostnames, and set the exact HTTPS Web origin. The profile preloads images with pull_policy: never and does not seed demo data.

Production Helm#

helm upgrade --install acknexa ./deploy/helm/acknexa \
  --namespace acknexa \
  -f deploy/helm/acknexa/values-onprem.yaml \
  -f /secure/path/site-values.yaml \
  --atomic --wait --timeout 15m

Pre-create the namespace, private-registry credentials, customer CA ConfigMap, state-service Secrets, application encryption key, and ingress TLS Secrets. Supply actual image digests and ingress/resource/topology values. PostgreSQL uses verified TLS and separate runtime/migration identities. NATS uses authenticated TLS seeds and a replicated file-backed stream.

The chart deploys stateless application components, not the customer's entire HA platform. Database operators, NATS clusters, ingress controllers and independent monitoring are customer-managed. A private VIP, hardware appliance or internal load balancer can replace a cloud load balancer.

Private-network and disconnected operation#

Mirror images and build dependencies before disconnecting. Runtime isolation does not automatically provide an offline build cache. Private-registry CA trust must be installed on container nodes; the application's CA mount cannot fix image pulls. Build the Web image with the private API URL so its integration instructions display the correct address.

The on-prem Compose backend network is internal by default. Reach customer LAN services through an approved network attachment or a customer-only firewall allowlist. The Helm overlay supplies restricted egress; use an enforcing CNI and add explicit rules for dependencies outside the namespace. Do not assume NetworkPolicy alone covers every host or ingress path.

Configuration overlays#

• Internal OIDC: deploy/onprem/oidc.compose.yml.
• SMTP authentication: deploy/onprem/smtp-auth.compose.yml.
• Customer voice gateway for paging calls: deploy/onprem/voice.compose.yml.
• CUCM gateway for live call routing: deploy/onprem/voice-gateway.compose.yml, or the Helm voiceGateway values. Include the overlay in every later Compose command, upgrades too.
• Android push through FCM: deploy/onprem/fcm.compose.yml.
• Helm adapters are opt-in via oidc.enabled and notifications.*.enabled (including notifications.fcm and notifications.pushRelay; the push relay cannot be combined with APNs or FCM). Disabled adapters do not require their cloud credentials.
• Set PROVIDER_CALLBACK_BASE_URL (Helm config.providerCallbackBaseURL) when Twilio, call routing or ticketing inbound webhooks are used.

Trust model#

AckNexa enforces role checks and organization scope in the API, with tenant-aware database relationships as a second boundary. Sessions and API credentials are represented by hashes at rest; passwords use bcrypt. The Web proxy uses an HttpOnly session cookie and checks the configured browser origin for mutations.

TLS and private CA#

• Terminate browser/integration traffic at the customer's TLS ingress with certificates covering the exact private DNS names.
• Mount the customer trust bundle for Go with SSL_CERT_FILE and for the Web runtime with NODE_EXTRA_CA_CERTS.
• Production PostgreSQL URLs require sslmode=verify-full and the appropriate sslrootcert path.
• Use authenticated TLS NATS seed URLs and certificates matching those hostnames.
• Keep verification enabled. Do not use TLS verification bypasses as a private-CA workaround. Restart workloads after relevant trust/secret changes.

Secrets management#

Use customer-controlled secret files, Kubernetes Secrets, or an internal secret manager. External Secrets is optional and need not connect to a public service. Protect Kubernetes Secrets with appropriate RBAC and encryption at rest; protect Compose secret files and their parent directory on the host.

Supported sensitive settings accept a corresponding _FILE form, including the settings above and selected APNs/Twilio secrets. Set either the value or the file, not both. Files are loaded at startup; rotation requires a restart. Keep ACKNEXA_CONFIG_KEY in an independently protected recovery copy.

Single sign-on and directory sign-in#

Organization administrators configure SSO (OpenID Connect, for example Keycloak, Microsoft Entra ID or Okta) and LDAP / Active Directory sign-in under Identity, with optional automatic account creation and group mappings. Unverified emails are refused. LDAP uses ldaps:// or StartTLS, never a clear-text password, and accepts a pasted private CA certificate. Client secrets and bind passwords are sealed with ACKNEXA_CONFIG_KEY and never returned by the API. Organization-level SSO needs PUBLIC_API_URL and PUBLIC_BASE_URL so callback URLs are known.

Operators can instead configure one issuer for the whole deployment with the OIDC_* environment variables. That client must put the organization slug in the acknexa_organization claim (or the claim named by OIDC_ORGANIZATION_CLAIM), because one client serves every organization. Control who can change those claims in the IdP.

SSO is optional, not a dependency on a public identity service. Keep a local break-glass administrator. Identity rebinding requires an approved operator procedure; the Web UI does not yet manage it.

Notification egress#

Tenant-configured SMTP and HTTPS webhooks require operator-approved DNS hosts. Private destinations additionally require explicit NOTIFICATION_PRIVATE_CIDRS. DNS answers are checked and the validated address is dialed; loopback, link-local/metadata destinations and redirects remain restricted. Environment-configured operator endpoints are a separate trusted boundary and need firewall policy. Ticketing connections are limited to TICKETING_HOSTS, and audit syslog streams to AUDIT_SYSLOG_HOSTS.

Shared responsibility#

• Operate hardened container hosts, ingress, database/broker authentication, network policies and encrypted backups.
• Keep sensitive tokens and customer payloads out of logs, tickets and public documentation.
• Review access when people change teams or leave the organization.
• Do not expose metrics or database ports as unauthenticated public services.
• Report a suspected vulnerability in AckNexa to [email protected], never in a public issue. Include the affected version or image digest, a description and impact, and reproduction steps, but no customer data. No PGP key is published yet; if the details need encryption, say so in a short first email. For anything else, write to [email protected].

The event path#

1. A monitoring source sends an authenticated event to the integration gateway in the API.
2. PostgreSQL commits the raw payload and outbox event together before the API returns accepted.
3. An outbox publisher sends committed work to NATS JetStream; a consumer normalizes, routes and deduplicates it.
4. Incident state and escalation jobs are persisted in PostgreSQL.
5. Workers claim due escalation jobs and create notification work; provider attempts and results are tracked separately.
6. Responders view the same authoritative state through the Web/API and acknowledge or resolve the incident.

What each dependency owns#

Concurrency and ordering#

State transitions lock the incident and validate the requested version. Jobs and notification attempts use leases and durable state to support restart/recovery. Source timestamps prevent older alert state from replacing newer observations. Delivery is at least once with database idempotency; this is not a blanket exactly-once delivery promise.

When infrastructure fails#

• Database unavailable: state changes cannot safely commit; readiness fails and clients should retry appropriately.
• NATS unavailable: accepted outbox records remain in PostgreSQL. Broker supervision is separate from SQL workers; reconciliation can recover aged unprocessed raw events.
• Worker restart: pending work remains durable or leases expire for recovery.
• Provider outage: affected notification attempts fail/retry visibly; incident access and response remain available if core dependencies are healthy.
• Whole stream lost: published outbox records are not automatically all replayed. Restore/replay must be coordinated and tested.

Capacity and deployment evidence#

Signals to collect#

• API request errors and latency; health/readiness failures.
• Unprocessed raw events, oldest backlog age, outbox publication delay and quarantined events.
• Escalation delay, overdue jobs and worker-loop progress.
• Notification attempts, failures, unknown outcomes and provider acceptance latency.
• PostgreSQL connections, replication/WAL health, storage and backup freshness.
• NATS stream replica health, pending messages and durable consumer progress.

Local observability stack#

The on-prem Compose observability profile provides Prometheus, Grafana with a provisioned dashboard, and an OTel Collector configuration. Grafana reporting and update checks are disabled in that profile. Keep private monitoring UIs behind customer access controls.

OpenTelemetry boundary#

AckNexa exposes Prometheus metrics and structured JSON logs. The included collector accepts internal OTLP metrics and exposes them for local Prometheus. A customer Collector can also scrape the application's metrics. Native application OTLP distributed traces are not implemented; do not assume trace correlation exists because a collector is deployed.

Keep metrics off public ingress. The Helm API ingress publishes only the API path prefixes, not /metrics; Prometheus scrapes the in-cluster API Service. The API's /metrics can require a bearer token (METRICS_TOKEN), and the bundled Compose nginx answers /metrics on the API host with 404.

Health checks are not delivery checks#

Readiness proves a limited dependency check, not that every provider, schedule or escalation path works. Combine probes with queue/progress metrics and periodic authenticated end-to-end tests using dedicated internal recipients.

Independent alerting#

Monitor AckNexa from another customer-controlled failure domain, and route critical platform alerts through a channel that does not rely only on AckNexa. A self-hosted Alertmanager and an independent dead-man check can run entirely on the private network. A dashboard on the same failed host cannot page about its own disappearance.

What to protect#

• PostgreSQL: configuration, incidents, raw events, jobs, notification attempts, outbox and audit.
• NATS stream state is optional: the workers recover every unprocessed raw event from PostgreSQL when the stream is empty. Use supported snapshots or a quiesced backup if you also want the queue files.
• ACKNEXA_CONFIG_KEY and other recovery secrets, protected independently from the database copy.
• TLS materials, IdP configuration/backups, deployment values, image digests and approved release artifacts.
• Customer-created monitoring dashboards and alert configuration. Redis is not authoritative for the current core.

Small-install backup#

bash scripts/onprem_backup.sh \
  /secure/path/acknexa.env \
  /secure/backups/acknexa-NEW

Use a new absolute destination. By default (ACKNEXA_BACKUP_MODE=live) the script takes a consistent database snapshot while ingress, API and workers keep paging, and saves the protected configuration and a checksum; NATS is not copied. ACKNEXA_BACKUP_MODE=quiesced instead pauses the Web, API, workers and NATS, also copies the NATS files, and restarts only previously running services; paging is paused while it runs. Supply the same optional Compose overlays through ACKNEXA_COMPOSE_OVERRIDES when applicable.

Restore safely#

1. Verify backup checksums and image/database/broker version compatibility.
2. Provision a new isolated target, not the live source. Restore protected configuration and the encryption key.
3. Restore the PostgreSQL dump into an empty database. The small-install restore script then starts with an empty NATS stream and the workers recover unprocessed raw events from PostgreSQL. If you restore NATS data or snapshots, use the supported, stopped-target procedure; do not merge files into a live stream.
4. Start matching application versions without test seeding. Restored pending escalations resume paging as soon as workers start, so point the target at test recipients unless this is a real cut-over. Inspect tenant configuration, audit history, pending work and incident state.
5. Use dedicated test recipients to verify delivery and ACK before switching traffic. Measure actual recovery time and data loss against agreed targets.

Production recovery#

Use continuous WAL/PITR and encrypted backups to customer-controlled NAS, another site, or an internal object store as appropriate. Cloud S3 is not required. Exercise primary loss, a host/failure-domain loss, broker quorum loss, and restoration of encrypted provider configuration.

Do not replay blindly#

A lost stream may include work whose outbox row was already marked published. Resetting every publication flag can create unwanted replay. Recovery must identify unprocessed work, respect durable idempotency, and account for ambiguous provider acceptance. Backup success alone is not a disaster-recovery pass.

Plan a release#

1. Read the CHANGELOG upgrade notes for every release you skip. A historical migration may require a coordinated stop rather than a rolling deployment.
2. Run the paging upgrade check with the new image against the production database, and fix its findings.
3. Take and verify a recoverable backup, including encryption keys and state-service recovery information. Do not start an upgrade while a database dump is running: migrations wait behind it, and paging queries then queue behind the migrations.
4. Build or preload approved images and record immutable digests. Rebuilding and restarting are separate actions.
5. Run migrations with the schema-owner identity. Keep the runtime credential separate in production.
6. Roll out matching API, worker and Web images, then verify readiness and the end-to-end incident workflow.
7. Observe queue age, errors, provider outcomes, ACK latency and audit continuity before declaring success.

The upgrade check gate#

Release 1.0.0-rc.1 changes who may be paged: rotation and override participants are paged only while they are an acting member of the schedule's team, and organization viewers are never paged. Anyone else is skipped afterwards, which can leave a schedule with nobody on call. The read-only upgradecheck command reports this before you migrate.

docker run --rm --entrypoint upgradecheck IMAGE_OF_THE_NEW_RELEASE \
  paging --database-url "$DATABASE_URL" --format text

Helm runs the check as a pre-install/pre-upgrade hook Job before the migration Job, and findings fail the release. Compose runs an upgradecheck service that must succeed before migrate, API and worker start. On a fresh database, and once that migration has been applied, the gate passes without checking. After reviewing findings, continue with upgradeCheck.allowFindings=true (Helm) or ACKNEXA_ALLOW_UPGRADE_FINDINGS=true (Compose).

Why restart did not update the UI#

Restarting a container starts the same image again. Rebuild or pull the intended approved image, then recreate the relevant service. Make sure the browser is using the correct installation and port; a local test stack and a default development stack can coexist with different images and databases.

Helm and migration hooks#

The production chart runs the upgrade check and then a pre-install/pre-upgrade migration hook. Index builds in this release's migrations block writes to the affected tables while they run, the largest being on audit_events; rehearse on a restored copy to know how long they take. Required database Secrets, CA ConfigMaps and pull credentials must exist before these hooks run. Do not assume an External Secrets resource created in the same release has synchronized in time for first install.

Rollback boundaries#

An image rollback does not automatically reverse a database migration. Only use a previous image if it is compatible with the current schema. Do not run destructive down migrations as an automatic rollback step. Follow an explicitly tested recovery plan if schema/data restoration is required.

Production acceptance#

• Complete the real monitoring-source to responder ACK workflow for every enabled channel.
• Prove operation with internet egress denied for a private-only deployment.
• Test backup restore, private-CA rotation, failover and a representative load profile.
• Verify tenant isolation, least-privilege access, secret handling and audit preservation.
• Record PASS, FAIL or NOT TESTED rather than treating manifests or local mocks as production evidence.

API base and specification#

Use your deployment's API origin, not the public documentation site's URL. Paths below are relative to that origin. The OpenAPI document is available as an API specification download. It describes implemented contracts but is not a promise that every current administration field is exhaustively modeled.

Authentication#

Password sign-in posts organization, email and password to /v1/auth/login and returns a bearer token with its expiry. Authenticated calls use Authorization: Bearer TOKEN. Store tokens securely, observe expiry/revocation, and do not put them in query strings. The Web UI uses a server-side cookie proxy rather than exposing its bearer token to browser storage.

{
  "organization": "your-organization-slug",
  "email": "[email protected]",
  "password": "YOUR_PASSWORD"
}

API keys and service accounts#

For scripts, CI and other systems, use an API key instead of a person's session. Organization administrators manage keys under API Keys in Administer, and each person can manage their own personal keys. Send the key as Authorization: Bearer YOUR_API_KEY.

• The token is shown once at creation; only its hash is stored. Expiry is optional, from 1 to 730 days.
• An organization can have at most 50 active organization keys, and each person at most 10 active personal keys.
• A service account cannot sign in, and is left out of people pickers, on-call candidates, call routing, reports and postmortem owners. Revoking the key deactivates it.
• Whatever its role, a key cannot create API keys, generate a SCIM token, change notification rules or contact details, create a calendar feed link or register a push device; those need a signed-in session.
• Creation and revocation are audited with the key name and prefix, never the token. A revoked or expired key, or one whose owner is deactivated, gets 401.

Core endpoints#

Version-aware response#

{ "version": 7 }

The example is an acknowledge request. Read the actual current incident version; do not hardcode it. Resolution additionally accepts a reason. If a concurrent change causes 409, refresh and reassess before resubmitting.

Administration endpoints#

The /v1/admin routes require organization_admin. Resource areas include organization, users, teams, services, schedules, policies, integrations, routing-rules, providers, audit, api-keys, status-pages, call-routes, ticketing, webhooks and scim. Integration tests, provider tests and onboarding completion have separate actions. Do not infer a supported HTTP verb merely from a resource name.

Error handling#

Automation rules#

• Reuse an idempotency key only for retries of the same integration event.
• Use bounded backoff and distinguish a failed response from an ambiguous committed request.
• Do not log bearer tokens, webhook keys, provider configuration, or raw customer payloads by default.
• Some listing endpoints are bounded, not exhaustive paginated exports. Validate coverage before using them for reporting.
• Test API clients against the exact deployed release; the schema download is a reference, not a hosted API service.

What is implemented#

AckNexa can send voice notifications (paging calls from escalation policies) to an operator-configured HTTPS gateway and accept signed, sequenced callbacks. The gateway translates this contract into the customer's CUCM, SIP or PBX operations. For these paging calls no SIP stack or CUCM connector is bundled; the bridge and site-specific call testing remain customer responsibilities.

Inbound hotline calls are a different feature: live call routing includes a ready-made CUCM gateway.

Configure the adapter#

• Set VOICE_GATEWAY_URL to the private HTTPS endpoint and VOICE_GATEWAY_TOKEN to a random bearer token of at least 32 characters; _FILE is supported.
• Configure the API and workers consistently. In Helm, enable notifications.voiceGateway and provide its URL and Secret.
• Install the private CA, allow only the required network path, and restrict the gateway's permitted dial destinations.
• Ensure organization-level voice provider settings do not override the deployment gateway unintentionally.

Outbound notification#

AckNexa POSTs a JSON Message containing ID, Channel, Destination, Title, Body, IncidentID, Severity, Service, Environment and Status. The request uses Authorization: Bearer TOKEN and Idempotency-Key: NOTIFICATION_UUID. The gateway may return its call ID in X-Request-ID.

The gateway must authenticate, validate destinations, and persist idempotency before returning 2xx. Return success only for durable acceptance. Do not redirect: redirects are not followed. A successful HTTP call is not evidence that a phone rang or a person responded.

Callback and acknowledgement#

{
  "event_id": "unique-gateway-event",
  "notification_id": "NOTIFICATION_UUID",
  "state": "answered",
  "sequence": 3,
  "digit": "1"
}

POST callbacks to /v1/voice/callback. Supported states are queued, dialing, ringing, answered, no_answer, busy, failed and completed. Only explicit digit 1 with answered invokes acknowledgement; answering alone does not ACK the incident.

Use X-AckNexa-Timestamp with an RFC3339 timestamp and X-AckNexa-Voice-Signature with the hex HMAC-SHA256 of timestamp + '.' + raw body. The HMAC key is SHA256 of the per-organization callback secret. The signature has no sha256= prefix on this voice endpoint. Clock skew must stay within five minutes.

Credential provisioning and replay#

A trusted operator provisions the callback key hash in voice_gateway_credentials for the organization. Use a separate secret from the outbound bearer token. Callback event IDs must be unique and sequence values increase per notification. Duplicate event IDs and stale sequences are handled without regressing call state; the responder must still be authorized.

Acceptance test#

1. Send a notification to a dedicated internal test extension and verify gateway idempotency under retry.
2. Observe queued/ringing/answered callbacks, then test digit 1 and verify the incident ACK and cancelled pending escalation work.
3. Exercise no-answer, busy, duplicate/stale callbacks, wrong signatures, expired timestamps and revoked responder access.
4. Record actual provider/call evidence before claiming CUCM/PBX interoperability.

I cannot see admin options#

1. Confirm you are using the intended application URL and installation.
2. Check that your organization role is organization_admin, not only team_admin.
3. Sign out and back in after a role change, and refresh the browser.
4. Verify the deployed Web/API image versions. Rebuild and recreate services if code changed; restart alone retains the old image.

Login or setup fails#

• Verify the organization slug, email and password; check whether the user is active.
• If first-run setup is absent, check whether an organization already exists or the setup token is missing.
• For origin errors, match ACKNEXA_WEB_ORIGIN exactly to the browser scheme, hostname and port.
• For private HTTPS, install the right CA on servers and browsers; do not disable certificate verification.
• For OIDC, verify issuer discovery, redirect URI, claims, pre-provisioned user and issuer/subject binding.

Webhook accepted but no incident#

1. Record the returned raw event ID and whether the response said duplicate.
2. Inspect integration event history and Delivery Health for processing/quarantine.
3. Confirm the intended integration is enabled, the service is active, and a rule has not routed the alert elsewhere.
4. Check workers, database, broker and backlog age. Remember that 202 is acceptance, not completion.
5. If quarantined, correct the underlying issue and use the audited retry action.

Incident exists but nobody was reached#

1. Check the service's active escalation policy and its targets.
2. Inspect the effective on-call responder and any override at the incident time.
3. Verify active recipient, contact destination and channel preferences.
4. Inspect notification attempts for configuration, authentication, TLS, DNS or provider errors.
5. Test the actual selected provider. Confirm that an organization override is not disabling or replacing the deployment adapter.
6. For private-only networks, remove unavailable cloud channels from the operational policy.

ACK failed or another notification arrived#

A 409 indicates state changed; refresh and inspect before acting again. A network failure is not proof that an action failed to commit, so re-read the incident. A provider request authorized before ACK can still finish afterward; inspect the attempt timestamps rather than assuming escalation restarted.

Backup or upgrade problem#

Stop and preserve evidence. Do not delete volumes, reset outbox flags, edit audit rows, or force a down migration to make the error disappear. Check exact image/schema compatibility and use the backup and recovery procedure on a separate target.

If an upgrade stopped at the paging check, read its findings with docker compose logs upgradecheck or kubectl logs on the acknexa-upgradecheck Job, fix the team memberships it reports, and run the upgrade again. See the upgrade check gate.

Prepare a useful support report#

• State the deployment model, image versions, affected time window and sanitized request/event IDs.
• Describe expected and observed behavior, including HTTP status and whether it is reproducible.
• Include relevant redacted logs and health/backlog observations.
• Never attach passwords, provider keys, session tokens, private images or raw customer payloads to a public issue.

Your deployment operator is the first point of contact for your installation. To reach the AckNexa team, write to [email protected]. No support entitlement or response time is implied.

Release status#

What has been checked for this release:

• Continuous integration passes: Go unit tests and PostgreSQL/NATS integration tests with the race detector, govulncheck, npm audit, a gitleaks secret scan, the Web build and UI tests, deployment contract tests and the container image builds. Release builds run the full suite first.
• An upgrade check runs before migrations in both Helm and Compose and stops an upgrade that would silently stop paging people. See the upgrade check gate.

What remains before general availability:

• A real-world pilot deployment.
• An external penetration test.
• SCIM validation against a live Okta or Microsoft Entra ID tenant; so far it has been tested with SCIM test clients only.
• Store distribution of the mobile apps; for this release the operator builds the Android and iOS apps from source, and neither is in the App Store or Google Play.
• Validation of the CUCM gateway against a real CUCM cluster; so far it has been tested against an Asterisk-based CUCM stand-in.

Implemented core#

• Authenticated Alertmanager, Icinga, Alerta, PagerDuty Events API v2 compatible and generic event ingestion, plus heartbeat monitors.
• Normalization, severity mapping, routing history, duplicate protection and incident lifecycles.
• Teams, services, users, schedule layers/overrides, escalation policies and maintenance windows.
• Web incident response, acknowledgement, resolution, notes, audit and Delivery Health.
• Status pages with public titles and email subscribers, stakeholder updates, and postmortems with action items.
• Organization administration, provider configuration/testing, API keys and service accounts, reports and guided onboarding.
• Local passwords, SSO (OpenID Connect), LDAP / Active Directory sign-in, SCIM 2.0 provisioning and break-glass accounts.
• Audit export (JSON Lines or CSV) and audit streaming to HTTPS collectors or TLS syslog.
• Several organizations per deployment.
• Self-hosted Compose and Helm deployment paths, customer CA and private registry configuration, file-backed secrets and the upgrade check.

Optional adapters#

Known limitations#

• No WhatsApp adapter, email intake or SNMP trap intake.
• Additional organizations are created by the operator with the bootstrap command; there is no API or Web screen to add one, and no organization delete.
• Incident numbers are shared across all organizations of a deployment, so each organization sees gaps in its numbering.
• The mobile apps keep one account per server address, so a phone can be signed in to only one organization of a deployment at a time.
• Notification and subscriber emails are minimal plain-text messages; there are no branded templates.
• No Web-based OIDC identity unlinking/rebinding.
• No exhaustive minute-by-minute future on-call coverage timeline.
• No full organization-admin configuration screens for every team-admin workflow.
• No unbounded UI exports, complete analytics/reporting product, or turnkey retention/compliance suite.

Verification is not certification#

CI runs the backend race/integration tests, the Web UI tests and the deployment contract tests, and database backup/restore has been exercised. AckNexa also comes with an end-to-end lab with real Prometheus, Alertmanager and Icinga 2, the mobile apps and a CUCM stand-in, for operators to repeat these checks. None of this certifies production failover, point-in-time recovery targets, capacity, private registry interoperability, real SMTP/IdP/PBX installations, or end-user mobile delivery at your site.

Commercial and legal information#

AckNexa is self-hosted on your infrastructure; it is not offered as a hosted subscription, and it is not open source. Pricing, licence terms and support are arranged directly: write to [email protected].

Does AckNexa require a public cloud?#

No production-critical core workflow requires a vendor-hosted AckNexa control plane or public cloud API. Customer PostgreSQL, NATS, Web/API/workers, ingress, secrets, monitoring and backups can run on premises.

Can it run without internet access?#

The core can operate in an isolated private network once images, configuration and dependencies are provisioned. Use internal notification channels and private identity/DNS/PKI. APNs, FCM, the push relay and SaaS providers still require their own external connectivity.

Is this website a hosted application?#

No. It is a separately deployable public product site and guide. Your administrator provides the URL and credentials for your own AckNexa installation.

Is Redis mandatory?#

Redis is supplied in the on-prem profile for customer-controlled use, but the current core does not depend on a cache for incident correctness. PostgreSQL remains authoritative.

Does provider accepted mean the responder was notified?#

It proves only the provider accepted the request. Delivery to a device or inbox and an explicit human acknowledgement are separate outcomes. Inspect attempts and verify with real recipients.

Can I connect Cisco CUCM directly?#

For hotline calls, yes: live call routing includes a gateway (an Asterisk image plus the voicegateway service) that takes calls over a SIP trunk from CUCM, with no UCCX, CUBE scripting or IVR licence. It has been tested against a CUCM stand-in, not a real CUCM cluster, so validate it on yours. Paging calls from escalation policies still go through Twilio or a customer-operated HTTPS voice bridge; see Internal voice gateway.

Is there an Android app?#

Yes. A native Android app (Kotlin, Jetpack Compose, push through Firebase Cloud Messaging) is available alongside the iOS app. For this release the operator builds both from source; neither is in an app store. See Web and mobile access.

Can one deployment serve several organizations?#

Yes. Data and access are scoped per organization. The operator creates each organization after the first with the bootstrap command. Incident numbers are shared across the organizations of one deployment, and a phone can be signed in to only one organization per server address. See Users and access.

Is the platform production-certified?#

No. Release 1.0.0-rc.1 is a release candidate for pilot deployments, and the existence of these pages or manifests is not certification. Production HA/DR, scale and real-provider gates require measured acceptance on the intended infrastructure. Read Capabilities & limitations.

Does documentation search send my query elsewhere?#

No. This website searches a local static index in your browser. No analytics or external search service is included. Your hosting operator may independently retain normal HTTP access logs.

How do I contact the AckNexa team?#

Write to [email protected]. Report a suspected vulnerability to [email protected] instead, never in a public issue. For access, accounts and incidents in your own installation, your deployment administrator remains the first contact.

Product terms#