Routing SPC Alerts to Ticketing and MES Systems

Routing is the delivery backbone of the SPC alerting and OCAP response layer: the component that takes a detected out-of-control signal and reliably lands it in the systems that record and act on it. A router receives a rule violation, decides what kind of event it is by assigning an alarm code and a severity, decides where it must go, and delivers it there with the reliability a quality event demands — exactly once in effect, even when the underlying network calls fail and retry. This guide specifies that router as a dependency-injected abstraction and gives a runnable Python implementation with idempotency, retries, and a dead-letter path.

What Breaks Without a Router

Teams that skip the router wire each detection rule directly to each destination. Rule 1 posts to the ticketing system; a run-rule violation posts to the MES; a second characteristic adds two more wires. The combinations multiply until the delivery logic is scattered across the codebase, untestable, and impossible to reason about during an incident. Worse, each wire re-implements the two things that are genuinely hard — idempotency and retries — slightly differently, so some paths flood and others silently drop.

The most damaging concrete failure is duplicate incidents. A sustained special cause re-fires the same rule on every scan, once every few seconds. A naive integration opens a fresh ticket on each firing, so a single shift-long excursion buries the quality engineer under hundreds of near-identical tickets and the real signal is lost in the noise. The mirror-image failure is silent loss: a transient network error on the one scan that mattered drops the event entirely, and because nothing retried, no record of the excursion ever reaches the system of record. A router exists to make both impossible — to collapse repeated firings onto one incident and to guarantee that a firing which should be delivered eventually is.

Alert router fanning a violation to a ticketing adapter and an MES adapter A left-to-right flow. A detected violation enters an alert router that maps it to an alarm code and severity and derives an idempotency key from the process, rule, and time window. The router fans the resulting event to two dependency-injected adapters in parallel: a ticketing adapter that upserts an incident in the ticketing system, and an MES adapter that writes the event to the manufacturing-execution system and historian. A separate dashed path sends events whose retries are exhausted to a dead-letter queue for later reprocessing. Violation rule fired Alert router map code · severity idempotency key Ticketing adapter upsert incident MES adapter write event Dead-letter retries exhausted Ticket system Jira / ServiceNow MES / historian quality state
The router classifies the violation, derives an idempotency key, and fans the event out to injected ticketing and MES adapters; events whose retries are exhausted divert to a dead-letter queue rather than being lost.

Specification: Keys, Severity, and Backoff

Three quantities define the router's behavior, and each is worth stating precisely.

The idempotency key is what makes repeated firings collapse onto one record. It is a digest of the identity of the signal, not of the individual scan. With a bucketing window of $\Delta$ seconds, the window index for a point at epoch time $t$ is $w = \lfloor t / \Delta \rfloor$, and the key is

$$k = H\big(\text{process\_id} \parallel \text{rule\_id} \parallel w\big)$$

where $H$ is any stable hash and $\parallel$ denotes concatenation. Every scan within the same $\Delta$-second window that trips the same rule on the same process produces the same $k$, so the destination adapter upserts one record instead of creating many. Choosing $\Delta$ trades responsiveness against grouping: a window equal to the OCAP reaction-time budget groups a single excursion into a single incident without merging genuinely separate events.

Severity is a deterministic function of the rule that fired. A single point beyond the $3\sigma$ limit (Nelson rule 1) is the strongest single-point evidence of a special cause and maps to the highest severity; the run and trend rules, which accumulate evidence over several points, map lower. The mapping is a fixed table, not a computation:

Rule that fired Alarm code Severity
Point beyond $3\sigma$ (Nelson 1) SPC-BEYOND-3S Critical
Nine points one side of center (Nelson 2) SPC-SHIFT-9 Major
Six points trending (Nelson 3) SPC-TREND-6 Major
Two of three in zone A (Western Electric) SPC-2OF3-2S Major
Four of five in zone B (Western Electric) SPC-4OF5-1S Minor

Retry delay governs delivery reliability. On the $i$-th retry (counting from zero) the router waits a capped, jittered exponential backoff

$$d_i = \min\!\left(d_{\max},\; d_0 \cdot 2^{\,i}\right)\cdot\left(1 + U\right)$$

where $d_0$ is the base delay, $d_{\max}$ the ceiling, and $U$ a small uniform jitter that spreads retries so a recovering destination is not hammered by every stream at once. After the retry budget is exhausted the event goes to the dead-letter path rather than being dropped.

When to Route vs. Notify Directly

Routing to a system of record and pushing a real-time notification are complementary, not interchangeable, and the choice of what to do with a given signal is deliberate:

  • Route to ticketing and MES for every signal that must leave an auditable trail — which, for control-charted characteristics under a reaction plan, is all of them. The ticket is the durable record of the nonconformity and its disposition; the MES write-back puts the quality state where the shop-floor systems already read.
  • Notify a human channel in addition, for signals that need a reaction now. Notification is about latency to a person; routing is about durability of the record. A critical signal does both; a minor run-rule signal may route to a ticket without paging anyone.
  • Do neither beyond logging only for signals the plant has explicitly decided are informational. That decision itself belongs in the OCAP decision table, not hard-coded in the router.

The router does not decide whether a point is out of control — that is the job of the rule detection layer, whose output the router consumes. Keeping that boundary crisp means a reaction can never disagree with the chart. The two focused guides under this area build the concrete destinations: pushing out-of-control alerts to Jira and ServiceNow for the ticketing adapter, and writing SPC events back to MES and historians for the shop-floor write-back.

Production-Ready Python Implementation

The router below is dependency-injected: it knows nothing about Jira, ServiceNow, or any historian, only about a TicketingAdapter protocol that concrete backends implement. That keeps the routing, classification, idempotency, and retry logic in one testable place while the ticketing and MES write-back adapters vary independently.

from __future__ import annotations

import hashlib
import logging
import random
import time
from dataclasses import dataclass
from enum import IntEnum
from typing import Callable, Protocol

logger = logging.getLogger(__name__)


class Severity(IntEnum):
    INFO = 1
    MINOR = 2
    MAJOR = 3
    CRITICAL = 4


# Deterministic map from detection rule to an alarm code and severity.
# Rule 1 (a single point beyond 3 sigma) is the strongest single-point signal.
RULE_SEVERITY: dict[str, tuple[str, Severity]] = {
    "nelson_1": ("SPC-BEYOND-3S", Severity.CRITICAL),
    "nelson_2": ("SPC-SHIFT-9", Severity.MAJOR),
    "nelson_3": ("SPC-TREND-6", Severity.MAJOR),
    "we_zone_a": ("SPC-2OF3-2S", Severity.MAJOR),
    "we_zone_b": ("SPC-4OF5-1S", Severity.MINOR),
}


@dataclass(frozen=True)
class Violation:
    """One out-of-control signal emitted by the detection layer."""
    process_id: str
    rule_id: str
    timestamp: float  # epoch seconds of the offending point
    value: float
    center: float
    ucl: float
    lcl: float


@dataclass(frozen=True)
class AlertEvent:
    """A classified, keyed alert ready for delivery to an adapter."""
    key: str
    process_id: str
    rule_id: str
    alarm_code: str
    severity: Severity
    violation: Violation


class TicketingAdapter(Protocol):
    """Interface every routing destination must implement.

    ``upsert`` MUST be idempotent on ``event.key``: called twice with the same
    key it updates one record and never creates a second.
    """

    name: str

    def upsert(self, event: AlertEvent) -> str:
        ...


class InMemoryDedupStore:
    """Records keys already fully delivered. Swap for Redis/DB in production."""

    def __init__(self) -> None:
        self._seen: dict[str, float] = {}

    def seen(self, key: str) -> bool:
        return key in self._seen

    def mark(self, key: str) -> None:
        self._seen[key] = time.time()


class AlertRouter:
    """Classifies SPC violations and delivers them to injected adapters.

    Delivery is at-least-once with capped exponential backoff; effect is
    at-most-once because adapters upsert on the idempotency key. Events whose
    retry budget is exhausted are handed to ``dead_letter`` rather than lost.
    """

    def __init__(
        self,
        adapters: list[TicketingAdapter],
        dedup_store: InMemoryDedupStore,
        window_seconds: int = 300,
        max_retries: int = 4,
        base_delay: float = 0.5,
        max_delay: float = 8.0,
        dead_letter: Callable[[AlertEvent, Exception], None] | None = None,
        sleep: Callable[[float], None] = time.sleep,
    ) -> None:
        if not adapters:
            raise ValueError("At least one adapter is required.")
        if window_seconds <= 0:
            raise ValueError("window_seconds must be positive.")
        self.adapters = adapters
        self.dedup = dedup_store
        self.window_seconds = window_seconds
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.max_delay = max_delay
        self.dead_letter = dead_letter
        self._sleep = sleep

    def _key(self, v: Violation) -> str:
        """Stable idempotency key: process, rule, and the time-window bucket."""
        window = int(v.timestamp // self.window_seconds)
        raw = f"{v.process_id}|{v.rule_id}|{window}"
        return hashlib.sha1(raw.encode("utf-8")).hexdigest()

    def _classify(self, rule_id: str) -> tuple[str, Severity]:
        return RULE_SEVERITY.get(rule_id, ("SPC-UNKNOWN", Severity.MINOR))

    def route(self, v: Violation) -> dict[str, str]:
        """Route one violation to every adapter. Returns {adapter_name: record_id}."""
        key = self._key(v)
        if self.dedup.seen(key):
            logger.info("Duplicate signal suppressed for key %s (%s)", key, v.process_id)
            return {}

        alarm_code, severity = self._classify(v.rule_id)
        event = AlertEvent(key, v.process_id, v.rule_id, alarm_code, severity, v)

        results: dict[str, str] = {}
        all_ok = True
        for adapter in self.adapters:
            try:
                results[adapter.name] = self._deliver(adapter, event)
            except Exception as exc:  # noqa: BLE001 - boundary: never crash the pipeline
                all_ok = False
                logger.error("Delivery to %s failed for %s: %s", adapter.name, key, exc)
                if self.dead_letter is not None:
                    self.dead_letter(event, exc)

        # Only suppress future re-fires once the event is safely recorded
        # everywhere; a partial failure stays eligible so the next scan retries.
        if all_ok:
            self.dedup.mark(key)
        return results

    def _deliver(self, adapter: TicketingAdapter, event: AlertEvent) -> str:
        last_exc: Exception | None = None
        for attempt in range(self.max_retries + 1):
            try:
                return adapter.upsert(event)
            except Exception as exc:  # noqa: BLE001 - retryable transport error
                last_exc = exc
                if attempt == self.max_retries:
                    break
                delay = min(self.max_delay, self.base_delay * (2 ** attempt))
                delay *= 1.0 + random.random() * 0.25  # jitter to de-synchronize retries
                logger.warning(
                    "Retry %d/%d to %s in %.2fs", attempt + 1, self.max_retries,
                    adapter.name, delay,
                )
                self._sleep(delay)
        assert last_exc is not None
        raise last_exc

A concrete backend implements TicketingAdapter.upsert against its own client — a REST client for a ticketing system such as Jira or ServiceNow, or a historian/MES writer — injected in, so the same router serves every destination. The upsert contract is the load-bearing part: the adapter must search by event.key and update the existing record if one is found, which is what turns at-least-once delivery into at-most-once effect.

Validation and Testing

  • Idempotency under repeat firing. Feed the same violation timestamp ten times and assert the ticketing adapter's upsert is called with the same key each time and that only one record results. This is the single most important test in the layer.
  • Window boundary behavior. Two firings straddling a $\Delta$-second boundary must produce two keys, and two firings inside one window must produce one. Test both sides of the boundary explicitly so a persistent excursion groups but genuinely separate events do not merge.
  • Retry then succeed. Inject an adapter that raises twice and then returns; assert the router retries, that the injected sleep is called with increasing delays, and that the final result carries the record id.
  • Dead-letter on exhaustion. Inject an always-failing adapter and assert the event is passed to the dead-letter callback and that the key is not marked seen, so the next scan retries rather than silently dropping the signal.
  • Classification coverage. Assert every rule id the detection layer can emit maps to an alarm code and severity, and that an unknown rule falls back to a safe default rather than raising inside the routing path.

Failure Modes and Edge Cases

Symptom Root cause Fix
Hundreds of duplicate tickets for one excursion No idempotency key, or the key includes the raw timestamp instead of the window bucket Key on process, rule, and floor(t / window); upsert on that key in the adapter
A real excursion never reaches the ticket system Transient network error with no retry, event dropped Retry with capped exponential backoff and divert exhausted events to a dead-letter queue
Retries hammer a recovering backend and it fails again No jitter; every stream retries on the same schedule Add uniform jitter to the backoff delay to spread retries
Suppressed signal never re-delivers after a partial failure Key marked seen even though one adapter failed Mark the key only when every adapter succeeded; leave partial failures eligible
Separate excursions merged into one ticket Window $\Delta$ far larger than the reaction-time budget Size the window to the OCAP reaction time so one incident equals one excursion
Router crashes the whole pipeline on a backend outage Adapter exception propagates out of route Catch at the delivery boundary, log, dead-letter, and continue to the next adapter

Compliance Notes

  • AIAG SPC Reference Manual (2nd ed.) — the routed incident is where the reaction plan's evidence begins; the alarm code ties the record to the specific rule that fired.
  • IATF 16949 — requires a documented reaction plan and evidence it was executed. The ticket created idempotently on the idempotency key, with its severity and alarm code, is that evidence; the MES write-back places the quality state in the controlled shop-floor system.
  • ISO 9001:2015, clause 10.2 (nonconformity and corrective action) — an out-of-control signal is a candidate nonconformity; the router discharges the clause's "react and record" requirement automatically, once, per excursion.
  • ISO 9001:2015, clause 7.5.3 (control of documented information) — the idempotent, keyed record and its retry/dead-letter trail are controlled documented information that reconstructs the delivery after the fact.

Frequently Asked Questions

How do I stop a persistent out-of-control signal from creating duplicate tickets?

Derive an idempotency key from the process, the rule, and a time-window bucket rather than the raw point timestamp, and have the ticketing adapter upsert on that key — search for an existing record and update it, creating a new one only if none is found. Every re-firing within the same window then lands on the same key and updates one incident instead of opening a new one each scan.

What size should the deduplication window be?

Match it to the reaction-time budget in the out-of-control action plan. A window that spans a single expected excursion groups all of that excursion's firings into one incident, while remaining short enough that two genuinely separate events fall into different windows and get their own records. Windows far larger than the reaction time risk merging distinct events.

Should the router decide whether a point is out of control?

No. Detection is the sole responsibility of the rule-detection layer, and the router consumes its output. Keeping that boundary clean means the routed incident can never disagree with the control chart, and the router stays a pure delivery-and-classification component that is simple to test.

What happens when a destination system is down?

The router retries with capped, jittered exponential backoff. If the retry budget is exhausted the event goes to a dead-letter queue for later reprocessing rather than being dropped, and its idempotency key is not marked delivered, so a subsequent scan of the same excursion remains eligible to retry. The signal is never silently lost.

For the full response layer and how routing fits the reaction loop, return to SPC alerting and OCAP response.