Real-Time Alert Notification Channels for SPC

Getting an out-of-control signal in front of the right human within seconds is the difference between a contained excursion and a scrapped lot. This guide covers the delivery layer of SPC alerting and OCAP response: how to push a detected violation to Slack, Microsoft Teams, email, or SMS through a single pluggable transport abstraction, route by severity, and throttle hard enough that the channel stays trusted instead of muted.

A notification layer is deceptively simple to prototype and brutally easy to get wrong in production. A one-line webhook call ships an alert today; three weeks later a persistent out-of-control run has posted four hundred identical messages, the quality channel is muted, and the one genuinely new signal is buried. The engineering work is not the transport — it is the formatting contract, the severity-to-channel routing, and the deduplication and rate-limiting gate that sits in front of every transport.

What Breaks Without a Channel Layer

Teams that wire detection straight to a webhook hit the same three failures in order.

Alarm fatigue. A control chart in a sustained out-of-control state re-fires on every incoming point. A Nelson rule that trips on point 1 of a shifted run will trip again on points 2 through 40. Naive alerting sends forty notifications for one incident. Operators learn within a shift that the channel cries wolf, and they stop reading it — so the layer that was supposed to shorten reaction time has lengthened it.

No context, no action. A message that says "Line 3 out of control" forces the operator to open a dashboard, find the chart, identify which rule tripped, and look up the reaction plan. Every second of that is downtime. A well-formed alert carries the rule, the characteristic, the measured value against its limit, and the specific out-of-control action plan (OCAP) step to execute, plus a link to the chart snapshot.

Wrong audience, wrong urgency. An informational zone-B warning and a critical spec-limit breach cannot share a delivery path. Routing everything to one email distribution list means criticals wait behind noise; routing everything to SMS means people silence their phones. Severity has to select the channel and the on-call target.

There is a subtler coupling worth naming. Aggressive rolling-window limit recalibration is sometimes proposed as a cure for alarm fatigue — if the limits keep re-centering on the shifted process, the alarms stop. That trades a delivery problem for a statistical one: limit chasing quietly absorbs a real special cause into the centerline and certifies an out-of-control process as stable. The correct fix for alarm fatigue lives here, in dedup and throttling, not in the limit math.

SPC alert notification channel pipeline A left-to-right flow. An out-of-control signal enters a formatter that attaches the rule, characteristic, measured value against its limit, and the OCAP action. The formatted alert reaches a deduplication and throttle gate that also routes by severity. Alerts that pass the gate fan out to three transports: Slack, Microsoft Teams, and an email or SMS transport. Alerts that are duplicates within the cooldown window or that exceed the rate limit branch downward to a suppressed-duplicate sink and are counted, not delivered. OOC signal rule tripped Formatter rule, value vs limit, OCAP + chart link Dedup + throttle cooldown, token bucket, severity route Slack block message Microsoft Teams adaptive card Email / SMS critical fallback Suppressed duplicate counted, not delivered route by severity duplicate / rate-limited
Every signal is formatted with full context, then passes a single dedup-and-throttle gate that routes by severity to Slack, Teams, or email/SMS. Duplicates within the cooldown and rate-limited bursts branch to a suppressed sink and are counted rather than delivered.

Channel and Throttle Specification

Two pieces of this layer are worth specifying precisely: the token-bucket rate limiter and the deduplication window, because both are what stand between a real incident and an alarm storm.

The token bucket for a given routing key holds up to $b$ tokens and refills at $r$ tokens per second. Each delivery consumes one token; a request is suppressed when the bucket is empty:

$$\text{tokens}(t) = \min\!\big(b,\; \text{tokens}(t_0) + r\,(t - t_0)\big)$$

Deliver only if $\text{tokens}(t) \ge 1$, then decrement by one. The capacity $b$ sets the allowed burst — how many notifications a genuinely new, rapidly escalating condition can send back-to-back — while $r$ sets the sustained ceiling. For a slow drift you might choose $b = 3$ and $r = \tfrac{1}{60}$ (one token per minute), permitting a 3-message burst then at most one message per minute per key.

Deduplication is a coarser, earlier gate. Within a cooldown of $T_c$ seconds, a repeat of the same routing key is dropped outright. If a single incident spans $L$ chart points arriving every $\Delta t$ seconds, naive alerting emits $L$ notifications; with a cooldown the count collapses to:

$$N = \left\lceil \frac{L \, \Delta t}{T_c} \right\rceil$$

So a 40-point run at one point per 5 seconds ($L\Delta t = 200\text{ s}$) with $T_c = 300$ s yields a single notification instead of forty. The routing key is what makes this safe. Keying on $(\text{process\_id}, \text{characteristic}, \text{rule}, \text{severity})$ means a different rule, a different characteristic, or an escalation to a higher severity produces a new key and breaks through the cooldown immediately — the storm is suppressed, but a genuinely new condition is not.

Parameter Symbol Typical value Role
Bucket capacity $b$ 3 tokens Allowed burst per routing key
Refill rate $r$ 1/60 per s Sustained notification ceiling
Dedup cooldown $T_c$ 300 s Window in which a repeat key is dropped
Routing key — 4-tuple Grants break-through to new conditions

The mechanics of collapsing a run into a single notification, choosing keys, and coalescing are covered in depth on deduplicating and throttling SPC alert storms.

When to Use Each Channel vs Alternatives

Channel selection is a severity decision, not a preference.

  • Slack or Microsoft Teams for informational and warning-level signals — zone-B and zone-C excursions, single-point Western Electric zone rule trips. A chat channel keeps the shift's quality context in one searchable thread and is the right home for the majority of alerts. Rich formatting (blocks, adaptive cards) carries the full context inline. The practical mechanics are on sending SPC alerts to Slack and Microsoft Teams.
  • Email for warning-level digests, shift summaries, and any signal that needs a durable record outside the chat retention window. Email is a poor real-time channel — treat it as a record and escalation path, not a primary alert.
  • SMS or phone-based paging for critical, spec-limit or safety-relevant breaches only. SMS bypasses a silenced app but has no formatting and a hard length limit, so the message must be a terse pointer: process, rule, and "open OCAP." Reserve it for the tier where a missed alert means scrap or a safety event.
  • Ticketing and MES write-back are not a human notification channel and belong to a different stage — when an alert must open a durable work item or annotate the process record, hand off to routing SPC alerts to ticketing and MES rather than overloading a chat transport.

The transport abstraction below exists so that adding SMS beside Slack is a new class implementing one method, not a rewrite of the routing logic.

Production-Ready Python Implementation

The following is a complete, runnable notification layer: an Alert value object that produces its own dedup key, an abstract Transport with Slack and Teams stubs that accept an injected webhook client (no URL is hard-coded here — the client is constructed and injected by the caller), a formatter, a token-bucket rate limiter, and a Notifier that routes by severity and applies the dedup/throttle gate before any transport is touched.

from __future__ import annotations

import logging
import time
from abc import ABC, abstractmethod
from dataclasses import dataclass, field
from enum import IntEnum
from typing import Protocol

logger = logging.getLogger(__name__)


class Severity(IntEnum):
    """Ordered so that a higher value is a more urgent alert."""
    INFO = 10
    WARNING = 20
    CRITICAL = 30


class TransportError(RuntimeError):
    """Raised by a transport when delivery fails after its own retries."""


@dataclass(frozen=True)
class Alert:
    """A single out-of-control signal ready for delivery.

    chart_url is a site-relative link to a rendered chart snapshot, never
    an external URL — the notification layer stays inside the intranet.
    """
    process_id: str
    characteristic: str
    rule: str
    value: float
    ucl: float
    lcl: float
    severity: Severity
    ocap_action: str
    chart_url: str
    ts: float = field(default_factory=time.time)

    def dedup_key(self) -> str:
        """Identity for suppression: same process, characteristic, rule, and
        severity collapse into one incident; a change in any of them breaks
        through the cooldown as a genuinely new condition."""
        return f"{self.process_id}|{self.characteristic}|{self.rule}|{int(self.severity)}"


class WebhookClient(Protocol):
    """Minimal contract for an injected chat webhook client.

    Implementations wrap a configured URL and handle transport retries; the
    notifier never sees the URL. Raises on non-recoverable failure.
    """
    def post(self, payload: dict) -> None: ...


class Transport(ABC):
    """A delivery channel. Subclasses translate an Alert into a channel-native
    payload and hand it to their injected client."""

    name: str

    @abstractmethod
    def send(self, subject: str, body: str, alert: Alert) -> None:
        """Deliver the alert or raise TransportError on failure."""


class SlackTransport(Transport):
    name = "slack"

    def __init__(self, client: WebhookClient) -> None:
        self._client = client

    def send(self, subject: str, body: str, alert: Alert) -> None:
        payload = {
            "blocks": [
                {"type": "header", "text": {"type": "plain_text", "text": subject}},
                {"type": "section", "text": {"type": "mrkdwn", "text": body}},
                {"type": "section", "text": {"type": "mrkdwn",
                                             "text": f"*OCAP:* {alert.ocap_action}"}},
                {"type": "actions", "elements": [
                    {"type": "button", "text": {"type": "plain_text", "text": "Open chart"},
                     "url": alert.chart_url}]},
            ]
        }
        try:
            self._client.post(payload)
        except Exception as exc:  # normalize any client error into our type
            raise TransportError(f"slack delivery failed: {exc}") from exc


class TeamsTransport(Transport):
    name = "teams"

    def __init__(self, client: WebhookClient) -> None:
        self._client = client

    def send(self, subject: str, body: str, alert: Alert) -> None:
        card = {
            "type": "message",
            "attachments": [{
                "contentType": "application/vnd.microsoft.card.adaptive",
                "content": {
                    "type": "AdaptiveCard", "version": "1.4",
                    "body": [
                        {"type": "TextBlock", "text": subject, "weight": "Bolder", "size": "Large"},
                        {"type": "TextBlock", "text": body, "wrap": True},
                        {"type": "TextBlock", "text": f"OCAP: {alert.ocap_action}", "wrap": True},
                    ],
                    "actions": [
                        {"type": "Action.OpenUrl", "title": "Open chart", "url": alert.chart_url}],
                },
            }],
        }
        try:
            self._client.post(card)
        except Exception as exc:
            raise TransportError(f"teams delivery failed: {exc}") from exc


def format_alert(alert: Alert) -> tuple[str, str]:
    """Build a channel-agnostic (subject, body) pair carrying full context:
    which rule, which characteristic, the value against its nearer limit,
    and the process id."""
    limit = alert.ucl if alert.value >= alert.ucl else alert.lcl
    side = "UCL" if alert.value >= alert.ucl else "LCL"
    subject = f"[{alert.severity.name}] {alert.process_id} — {alert.rule}"
    body = (
        f"*{alert.characteristic}* value {alert.value:.4g} breached {side} {limit:.4g}. "
        f"Process `{alert.process_id}`."
    )
    return subject, body


@dataclass
class TokenBucket:
    """Per-key token bucket. Refills continuously; each delivery costs one token."""
    capacity: float
    refill_per_sec: float
    tokens: float = field(init=False)
    updated: float = field(default_factory=time.time)

    def __post_init__(self) -> None:
        self.tokens = self.capacity

    def allow(self, now: float | None = None) -> bool:
        now = time.time() if now is None else now
        elapsed = max(0.0, now - self.updated)
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_per_sec)
        self.updated = now
        if self.tokens >= 1.0:
            self.tokens -= 1.0
            return True
        return False


class Notifier:
    """Routes alerts to transports by severity behind a dedup + throttle gate.

    routes maps a Severity to the ordered list of transports that should
    receive alerts at that level. The gate runs before any transport so a
    storm never reaches the network.
    """

    def __init__(
        self,
        routes: dict[Severity, list[Transport]],
        dedup_ttl_s: float = 300.0,
        bucket_capacity: float = 3.0,
        refill_per_sec: float = 1.0 / 60.0,
    ) -> None:
        if not routes:
            raise ValueError("at least one severity route is required")
        self._routes = routes
        self._dedup_ttl = dedup_ttl_s
        self._bucket_capacity = bucket_capacity
        self._refill = refill_per_sec
        self._last_seen: dict[str, float] = {}
        self._buckets: dict[str, TokenBucket] = {}
        self.suppressed_count = 0

    def _suppressed(self, alert: Alert, now: float) -> bool:
        key = alert.dedup_key()
        last = self._last_seen.get(key)
        if last is not None and (now - last) < self._dedup_ttl:
            return True  # duplicate inside the cooldown window
        bucket = self._buckets.setdefault(
            key, TokenBucket(self._bucket_capacity, self._refill))
        if not bucket.allow(now):
            return True  # rate limited by the token bucket
        self._last_seen[key] = now
        return False

    def notify(self, alert: Alert) -> list[str]:
        """Deliver an alert. Returns the names of transports that accepted it;
        an empty list means the alert was suppressed or every transport failed."""
        now = time.time()
        if self._suppressed(alert, now):
            self.suppressed_count += 1
            logger.info("suppressed alert key=%s", alert.dedup_key())
            return []

        transports = self._routes.get(alert.severity, [])
        if not transports:
            logger.warning("no route for severity=%s", alert.severity.name)
            return []

        subject, body = format_alert(alert)
        delivered: list[str] = []
        for transport in transports:
            try:
                transport.send(subject, body, alert)
                delivered.append(transport.name)
            except TransportError as exc:
                # one channel failing must not block the others
                logger.error("transport %s failed: %s", transport.name, exc)
        return delivered

Wire it up by constructing each transport with a webhook client the caller has configured, then mapping severities to channels — for example {Severity.WARNING: [slack, teams], Severity.CRITICAL: [slack, sms]}. The Notifier never learns a URL; it only knows names and severities.

Validation and Testing

  • Suppression is testable offline. Feed the Notifier a fake transport that records calls, replay a 40-point out-of-control run through notify, and assert exactly one delivery plus suppressed_count == 39. This is the single most important test in the layer.
  • Break-through must be preserved. After the run above, submit an alert with a different rule or a higher severity and assert it delivers immediately. A dedup layer that also swallows new conditions is worse than no layer.
  • Transport isolation. Make one transport raise TransportError and assert the other still receives the alert and appears in the returned list. A single channel outage must never silence the rest.
  • Formatter completeness. Assert every rendered message contains the rule, the characteristic, the value, the breached limit, and the OCAP action. An alert missing its reaction plan forces a dashboard round-trip and defeats the purpose.
  • Clock injection. TokenBucket.allow and _suppressed accept an explicit now so tests advance time deterministically instead of sleeping. Never write a throttle test that calls time.sleep.
  • Idempotency at the source. The detection layer should emit at most one signal per point per rule. If upstream double-fires, dedup will mask it — verify the rule detection stage is itself deduplicated so the notifier is not papering over a bug.

Failure Modes and Edge Cases

Symptom Root cause Fix
Channel floods with identical messages Dedup key too specific (includes timestamp or point index) so every point is "new" Key on process, characteristic, rule, and severity only; never include the value or time
A genuinely new excursion is never delivered Dedup key too coarse (e.g. process only) so a new rule collapses into an open incident Add rule and severity to the key so escalations break through
Critical alert lost during a chat outage Single transport, no severity fallback Route CRITICAL to two independent transports (chat plus SMS)
One bad webhook blocks all channels Exception from one transport not caught, aborts the loop Catch TransportError per transport; log and continue
Throttle never releases Token bucket refill rate set to zero or clock not advancing Set refill_per_sec > 0; inject a monotonic clock, not wall-clock in tests
Messages arrive with no action Formatter omits the OCAP step Make ocap_action a required field on Alert; assert it renders
Alarms silenced right when the process shifts Fatigue "solved" by recalibrating limits instead of throttling Throttle here; keep limits gated per rolling-window recalibration guidance

Compliance Notes

  • IATF 16949, Clause 9.1.1.1 requires a documented reaction plan for out-of-control conditions. The notification is the first link in that chain — the OCAP step must travel with the alert, and every delivery (and every suppression) needs a timestamped record for the reaction-time evidence an audit expects.
  • AIAG SPC Reference Manual (2nd ed.) frames the reaction to a special-cause signal. Suppressing duplicates is defensible only when the suppression is logged with its key and count; silently dropping signals is not.
  • ISO 9001:2015, Clause 9.1.3 expects analysis of process data. A suppressed-alert counter and a delivery log are the artifacts that show signals were acted on, not lost.
  • 21 CFR Part 11 (regulated environments) requires that the alert-and-reaction record be attributable and tamper-evident; treat the notification log as part of the electronic batch record, not as ephemeral chat history.

Frequently Asked Questions

Should I use email or SMS as the primary SPC alert channel?

No. Use a chat channel (Slack or Microsoft Teams) as the primary real-time channel because it carries rich formatting and keeps the shift's context searchable in one thread. Reserve email for durable records and digests, and reserve SMS strictly for critical, spec-limit or safety-relevant breaches where bypassing a silenced app matters. Routing everything to SMS trains people to silence it.

How do I stop an out-of-control run from sending hundreds of alerts?

Put a deduplication and throttle gate in front of every transport. Deduplicate on a routing key of process, characteristic, rule, and severity within a cooldown window, and rate-limit each key with a token bucket. A 40-point run then collapses to a single notification. Crucially, because the key includes the rule and severity, a different rule or an escalation still breaks through immediately.

Isn't recalibrating control limits a simpler cure for alarm fatigue?

It is a trap. Re-centering limits on a shifted process does stop the alarms, but it does so by absorbing a real special cause into the centerline — the limit-chasing failure mode. That certifies an out-of-control process as stable and will not survive an audit. Fix alarm fatigue in the delivery layer with dedup and throttling, and keep limit recalibration gated behind a change order.

Why route by severity instead of sending everything everywhere?

Because urgency and audience differ. Informational and warning signals belong in a chat channel where they are searchable and non-intrusive; critical breaches need a channel that bypasses a silenced device. Fanning every signal to every channel reproduces alarm fatigue on the high-urgency path, so criticals end up buried behind noise exactly when speed matters most.

How do I add a new channel like PagerDuty later?

Implement the Transport interface — one send method that translates an Alert into the channel's payload and hands it to an injected client — then add the new transport to the severity route map. No routing, dedup, or throttle code changes. That single-method contract is the whole reason for the transport abstraction.

Up one level: SPC Alerting and OCAP Response. For the detection layer that feeds these channels, start from the home page.