Escalation Tiers and Reaction Time for OCAP

A reaction plan that names an owner but not a deadline is only half a plan. Escalation tiers close that gap: each out-of-control signal is assigned a first responder and a reaction-time budget, and when the budget expires without acknowledgment the alert is promoted to the next tier automatically. This how-to defines the tiers (operator, line lead, quality engineer), builds a timer-driven state machine that escalates an unacknowledged alert, and measures the two numbers auditors ask for — time-to-acknowledge and time-to-contain. It extends the owner and tier fields introduced in the out-of-control action plans (OCAP) guide into a live promotion mechanism.

The design principle is that an alert must never come to rest in an unacknowledged state. Every tier has an expiry; if no one acknowledges before it, the alert climbs rather than sits. That single rule is what prevents the "acknowledge and ignore" failure — here it becomes "if you do not acknowledge, someone above you will have to."

Prerequisites

Confirm these before wiring up escalation:

  • Python 3.10+ (dataclasses, enum, and datetime from the standard library; no third-party packages required)
  • An action record from the OCAP engine carrying an initial escalation_tier and severity
  • Agreed reaction-time budgets per tier and severity — for example five minutes at the operator tier for a high-severity signal
  • A monotonic time source or injectable clock, so the state machine can be tested deterministically without real waiting
  • A destination for promotion events, typically the real-time alert notification channels that page the next tier

Step 1 — Define the tiers and their reaction-time budgets

Model the tiers as an ordered enum and the SLA as a lookup from (tier, severity) to a time budget. The budget shrinks as severity rises: a critical signal gives the operator far less time before it climbs. Formally, the reaction-time budget for tier $t$ at severity $s$ is $\tau(t, s)$, and an alert acknowledged at elapsed time $a$ meets its SLA when $a \le \tau(t, s)$.

from __future__ import annotations

from datetime import timedelta
from enum import IntEnum


class Tier(IntEnum):
    OPERATOR = 1
    LINE_LEAD = 2
    QUALITY_ENGINEER = 3


class Severity(IntEnum):
    LOW = 1
    MEDIUM = 2
    HIGH = 3
    CRITICAL = 4


# Reaction-time budget tau(tier, severity): time allowed before promotion.
SLA: dict[tuple[Tier, Severity], timedelta] = {
    (Tier.OPERATOR, Severity.CRITICAL): timedelta(minutes=2),
    (Tier.OPERATOR, Severity.HIGH): timedelta(minutes=5),
    (Tier.OPERATOR, Severity.MEDIUM): timedelta(minutes=15),
    (Tier.LINE_LEAD, Severity.CRITICAL): timedelta(minutes=5),
    (Tier.LINE_LEAD, Severity.HIGH): timedelta(minutes=10),
    (Tier.LINE_LEAD, Severity.MEDIUM): timedelta(minutes=30),
    (Tier.QUALITY_ENGINEER, Severity.CRITICAL): timedelta(minutes=15),
    (Tier.QUALITY_ENGINEER, Severity.HIGH): timedelta(minutes=30),
    (Tier.QUALITY_ENGINEER, Severity.MEDIUM): timedelta(hours=2),
}


def budget(tier: Tier, severity: Severity) -> timedelta:
    """Reaction-time budget for a tier and severity, with a safe default."""
    return SLA.get((tier, severity), timedelta(minutes=10))

Verify: assert budget(Tier.OPERATOR, Severity.CRITICAL) < budget(Tier.OPERATOR, Severity.MEDIUM) — a more severe signal must always give less time — and that an unlisted pair returns the ten-minute default rather than raising.

Step 2 — Model the alert as an escalation state machine

Represent each live alert as a small state machine with an explicit status and the tier it currently sits at. The legal transitions are narrow: an open alert can be acknowledged, escalated, or contained; a contained alert is terminal. Encoding the states explicitly stops an alert from, say, being contained before it was ever acknowledged.

from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum
from typing import Optional


class Status(Enum):
    OPEN = "open"
    ACKNOWLEDGED = "acknowledged"
    CONTAINED = "contained"


def _now() -> datetime:
    return datetime.now(timezone.utc)


@dataclass
class Alert:
    """A live OCAP alert progressing through escalation tiers."""
    alert_id: str
    severity: Severity
    tier: Tier
    status: Status = Status.OPEN
    raised_at: datetime = field(default_factory=_now)
    acknowledged_at: Optional[datetime] = None
    contained_at: Optional[datetime] = None
    tier_entered_at: datetime = field(default_factory=_now)
    escalations: int = 0

    def deadline(self) -> datetime:
        """When the current tier must have acknowledged by."""
        return self.tier_entered_at + budget(self.tier, self.severity)

Verify: a freshly built Alert has status is Status.OPEN, escalations == 0, and a deadline() that equals raised_at plus the tier budget. The deadline is what the timer in the next step checks against.

Step 3 — Promote an unacknowledged alert when its budget expires

The promotion logic is a pure function of the alert and the current time, which keeps it testable. If the deadline has passed and no one has acknowledged, advance the tier, reset the tier clock, and count the escalation. At the top tier there is nowhere higher to go, so it stays and is marked for direct intervention.

def check_escalation(alert: Alert, now: Optional[datetime] = None) -> bool:
    """Promote an unacknowledged, uncontained alert past its deadline.

    Returns True if a promotion happened. Pure w.r.t. `now` for testability.
    """
    now = now or _now()
    if alert.status is not Status.OPEN:
        return False
    if now < alert.deadline():
        return False

    if alert.tier < Tier.QUALITY_ENGINEER:
        alert.tier = Tier(alert.tier + 1)
        alert.tier_entered_at = now
        alert.escalations += 1
        return True

    # Already at the top tier: no higher tier, hold for direct intervention.
    return False


def acknowledge(alert: Alert, now: Optional[datetime] = None) -> None:
    """Record acknowledgment; stops escalation."""
    if alert.status is not Status.OPEN:
        raise ValueError(f"Cannot acknowledge an alert in status {alert.status.value}.")
    alert.acknowledged_at = now or _now()
    alert.status = Status.ACKNOWLEDGED


def contain(alert: Alert, now: Optional[datetime] = None) -> None:
    """Record containment; terminal state."""
    if alert.status is Status.CONTAINED:
        raise ValueError("Alert is already contained.")
    alert.contained_at = now or _now()
    alert.status = Status.CONTAINED

Verify: build a high-severity operator-tier alert, call check_escalation at a time just past the five-minute budget, and assert the tier advances to LINE_LEAD with escalations == 1. Call it again past the new deadline and assert it reaches QUALITY_ENGINEER; a third expiry returns False because there is no higher tier.

Step 4 — Measure time-to-acknowledge and time-to-contain

The two metrics that prove the plan works are how fast someone took ownership and how fast the situation was contained. Time-to-acknowledge is $\text{TTA} = t_{\text{ack}} - t_{\text{raised}}$ and time-to-contain is $\text{TTC} = t_{\text{contained}} - t_{\text{raised}}$. Compute them from the timestamps the state machine already records, and count escalations as a signal that a tier is under-resourced.

def metrics(alert: Alert) -> dict[str, Optional[float]]:
    """Time-to-acknowledge and time-to-contain in seconds, plus escalation count."""
    tta = (
        (alert.acknowledged_at - alert.raised_at).total_seconds()
        if alert.acknowledged_at else None
    )
    ttc = (
        (alert.contained_at - alert.raised_at).total_seconds()
        if alert.contained_at else None
    )
    return {
        "time_to_acknowledge_s": tta,
        "time_to_contain_s": ttc,
        "escalations": alert.escalations,
    }

Verify: an alert acknowledged 90 seconds after it was raised must report time_to_acknowledge_s == 90.0; an unacknowledged alert reports None, not zero, so unmet acknowledgments are never mistaken for instant ones. Aggregated across many alerts, the median time-to-contain and the escalation rate per tier are the numbers to trend for staffing and SLA tuning.

Verification

Drive the full lifecycle on an injectable clock so no real time passes, and assert promotion, acknowledgment, and the derived metrics together:

from datetime import datetime, timedelta, timezone


def test_escalation_lifecycle() -> None:
    t0 = datetime(2026, 7, 16, 8, 0, tzinfo=timezone.utc)
    alert = Alert("A-1", Severity.HIGH, Tier.OPERATOR,
                  raised_at=t0, tier_entered_at=t0)

    # Nobody acknowledges within the 5-minute operator budget -> promote.
    assert check_escalation(alert, now=t0 + timedelta(minutes=6)) is True
    assert alert.tier is Tier.LINE_LEAD and alert.escalations == 1

    # Line lead acknowledges at t0 + 8 min, then contains at t0 + 12 min.
    acknowledge(alert, now=t0 + timedelta(minutes=8))
    contain(alert, now=t0 + timedelta(minutes=12))

    m = metrics(alert)
    assert m["time_to_acknowledge_s"] == 8 * 60
    assert m["time_to_contain_s"] == 12 * 60
    assert m["escalations"] == 1

    # Acknowledged alerts no longer escalate.
    assert check_escalation(alert, now=t0 + timedelta(hours=1)) is False
    print("escalation OK:", m)


if __name__ == "__main__":
    test_escalation_lifecycle()

Expected output resembles escalation OK: {'time_to_acknowledge_s': 480.0, 'time_to_contain_s': 720.0, 'escalations': 1}. The injectable now argument is what makes this deterministic — never let the state machine read the wall clock inside a test, or the assertions become flaky.

Root-cause table

Symptom Cause Fix
Alerts sit unacknowledged for hours No timer promotes them; escalation is manual Run check_escalation on a fixed tick so every open alert is re-evaluated (Step 3)
A critical signal escalates as slowly as a minor one SLA budget not indexed by severity Key the budget on both tier and severity so severity shrinks the deadline (Step 1)
Alert gets contained before it was acknowledged State transitions not enforced Model explicit statuses and guard illegal transitions in acknowledge/contain (Steps 2–3)
Escalation counter keeps climbing at the top tier Promotion not capped at the highest tier Return False at the quality-engineer tier and hold for direct intervention (Step 3)
Time-to-acknowledge shows zero for ignored alerts Missing timestamp coerced to zero Return None when the timestamp is absent, never a numeric zero (Step 4)
Tests are flaky on timing State machine reads the real clock Inject now into every time-dependent call for deterministic tests (Verification)

Up one level: Out-of-Control Action Plans (OCAP).