SPC Alerting and OCAP Response: From Out-of-Control Signal to Documented Action

A control chart that raises a red point has done half a job. The other half — the half that keeps scrap out of the customer's hands and keeps the plant defensible in an audit — is what happens in the minutes after the signal. This section covers that response layer: how an out-of-control signal becomes a routed alert with an owner, a documented reaction, an escalation clock, and a closed loop of evidence. Detection tells you a process changed; alerting and reaction decide what the plant does about it, who does it, how fast, and how it is proven afterward.

The distinction matters because the two layers fail in different ways. The detection layer — the out-of-control rule detection using Nelson and Western Electric rules that evaluates every point against the frozen limits — fails when a real special cause slips past the run rules. The response layer fails when a correctly detected signal reaches nobody, reaches the wrong person, spawns a duplicate ticket every scan, or triggers a reaction that no record can later reconstruct. A plant can have flawless statistics and still ship nonconforming product because the alert died in a log file that no one watches.

From out-of-control signal to routed alert, reaction, escalation, and audit A left-to-right flow. An out-of-control signal enters an alert router. The router fans the alert out to two destinations in parallel: a ticketing and manufacturing-execution-system record, and a human notification channel. Both destinations feed a documented out-of-control action plan reaction, which then feeds a timed escalation stage. Beneath every stage runs an immutable audit log that captures the signal, the routing decision, the reaction taken, and the escalation and closure, tying each to a process, rule, owner, and timestamp. Signal rule fired · 3σ / run Alert router code · severity · owner Ticketing / MES incident · event write-back Notification chat · on-call channel OCAP reaction decision table · owner Escalation reaction-time tiers IMMUTABLE AUDIT LOG process_id · rule · alarm code · owner · timestamps · disposition — captured at every stage
A detected signal enters a router that assigns an alarm code, severity, and owner, fans out in parallel to a system-of-record write and a human channel, drives a documented reaction, and escalates on a clock — with every stage written to an immutable log.

Engineering Context: Why Detection Alone Is Not Enough

Consider what a bare detection layer produces: a boolean column, out_of_control, appended to a stream of subgroups. On its own that column changes nothing on the shop floor. Somebody has to see it, decide it is real, know what to do, do it before the next bad part is produced, and leave a trail that proves all of the above. Each of those verbs is a failure mode when it is left implicit.

The first failure is routing loss. A signal that is logged but not delivered is indistinguishable, from the customer's perspective, from a signal that never fired. The second is ownerless drift — an alert broadcast to a channel everybody can see and nobody owns, which is how a genuine excursion sits unacknowledged through a shift change. The third is reaction ambiguity: an operator who sees a red point but has no defined procedure will improvise, and improvised reactions to statistical signals are neither repeatable nor defensible. The fourth is untimed escalation — a reaction that is technically assigned but has no clock, so a stalled response never surfaces to a supervisor. The fifth is audit gaps, where the disposition of a nonconformity cannot be reconstructed months later when a customer or registrar asks.

This section builds the machinery that closes each gap. An alert router assigns an alarm code, a severity, and an owner to every signal, then guarantees delivery to both a system of record and a human. An out-of-control action plan — an OCAP — converts the signal into a specific, pre-authorized reaction keyed to the rule that fired and the characteristic involved. An escalation policy puts a reaction-time clock on that reaction and promotes it up a tier when the clock expires. And an immutable log threads through all of it, so the path from signal to closure is a single traceable record. The context an operator needs to judge a signal — the chart itself, the offending point, the recent trend — comes from the dynamic Plotly control chart rendering layer, which the alert links back to rather than duplicating.

There is also a statistical cost buried in getting this layer wrong. A Shewhart chart under only the three-sigma rule already carries a false-alarm rate of roughly one in 370 points on a stable process; adding run and zone rules raises detection power but also raises the nuisance-alarm rate. If every one of those alarms opens a ticket and pages a person, the response layer manufactures the alarm fatigue that eventually gets it ignored — and an ignored alerting system is worse than none, because it creates a false sense of coverage. The machinery below is therefore as much about restraint — collapsing a persistent signal onto one incident, throttling repeat notifications, routing by severity — as it is about delivery. Reliable response and disciplined suppression are two views of the same design.

The Life of a Signal

The three areas below are easiest to hold together by following one excursion from red point to closed record. Suppose the bore-diameter characteristic on line 7 begins to drift as a cutting tool wears, and after nine consecutive subgroups fall above the centerline the detection layer fires Nelson rule 2 — a sustained shift. That firing is the only thing the response layer treats as ground truth; everything downstream is reaction, not re-judgment.

The router receives the violation and classifies it: alarm code SPC-SHIFT-9, severity Major, owner set to the line's process engineer. It derives an idempotency key from the process, the rule, and the current time window, then fans the event out in parallel. The ticketing adapter opens one incident carrying that key; the MES write-back inserts a quality-event row in the OPEN disposition with the point value and the limits it violated; the notification path posts to the line's channel with a link back to the live chart so the engineer sees the shift, not just a code. Meanwhile the OCAP decision table, indexed by rule 2 on a bore diameter, returns the pre-authorized reaction — quarantine the parts made since the shift began, verify the tool offset, and confirm the correction on the next subgroups — together with the reaction-time budget and the escalation tier.

From there a clock runs. If the process engineer acknowledges and acts within the budget, they advance the disposition to CONTAINED and then DISPOSITIONED; the MES row and the ticket both update in place, keyed by the same identity, and the loop closes. If the budget expires with no acknowledgment, escalation promotes the alert to the next tier — the shift supervisor — automatically, because silence is itself a signal worth surfacing. Throughout, the shift persists on the chart, so the rule keeps firing every scan; but the idempotency key suppresses the re-fires, so one excursion remains one incident, one row, and one notification thread. Every transition — routed, notified, reacted, escalated, closed — lands in the immutable log under that single key. Reconstructing the whole episode later is a matter of following one identifier. The three areas below are the parts that make that thread exist.

Response Layer Map

The section decomposes the response layer into three areas: how alerts are routed to systems, how the human reaction is defined and governed, and how people are notified in real time. Each area has its own build-out.

Area Role in the response layer
Routing SPC Alerts to Ticketing and MES Systems The delivery backbone — an alert router that maps rule violations to alarm codes and severity, opens incidents in ticketing systems idempotently, and writes the quality event back to the MES and historian with delivery guarantees.
Out-of-Control Action Plans (OCAP) The reaction logic — a decision table that turns each rule-plus-characteristic combination into a specific pre-authorized action, an owner, and an escalation tier with a bounded reaction time.
Real-Time Alert Notification Channels The human-delivery path — pushing alerts to chat and on-call channels with the deduplication and throttling that keep a persistent signal from becoming an alert storm.

Routing Alerts to Ticketing and MES Systems

The routing area is the plumbing between a statistical result and the systems that record and act on it. Its central abstraction is an alert router: a component that receives a detected violation, decides what kind of event it is (a severity, an alarm code), decides where it must go, and delivers it there with the reliability guarantees a quality event demands. Without this abstraction, teams wire each detection rule directly to each destination, and the combinatorial mess that results is impossible to test or audit. A router collapses that into one place where mapping, severity, retries, and idempotency are defined once.

Two properties dominate the design. The first is idempotency: a special cause that persists across many scans re-fires the same rule on the same process every few seconds, and each re-fire must reconcile against the existing incident rather than opening a new one. The router therefore derives a stable key — process, rule, and time window — so that the twentieth firing of a sustained shift updates one ticket instead of creating twenty. The second is delivery guarantee: incident creation and event write-back are network calls that fail, so the router must retry with backoff, deduplicate on the key, and divert to a dead-letter path when a destination is unreachable, rather than dropping the quality event on the floor.

The key implementation decision in this area is where the idempotency window comes from. Set it too short and a single sustained excursion fragments into many incidents as consecutive windows tick over; set it too long and two genuinely separate events merge into one record and the second goes unremarked. The defensible choice is to size the window to the reaction-time budget defined in the OCAP, so one window corresponds to one opportunity to react — a decision that ties this area's plumbing directly to the reaction logic in the OCAP area rather than treating the two as independent.

Under this area sit two focused guides: pushing out-of-control alerts to Jira and ServiceNow, which builds the ticketing adapter and its idempotency key, and writing SPC events back to MES and historians, which persists the detected event, its limits, and its disposition into the shop-floor systems so the quality state lives where operators already look. Start with the routing area itself for the router abstraction and the adapter interface both guides depend on.

Out-of-Control Action Plans (OCAP)

An OCAP is the documented, pre-authorized answer to the question an operator asks the instant a chart signals: what do I do now? The AIAG SPC Reference Manual frames it as the reaction plan that a control chart requires to be complete — a chart without a defined reaction is a measurement, not a control. The engineering task is to encode that reaction as data rather than as tribal knowledge: a decision table indexed by the rule that fired and the characteristic it fired on, returning a concrete action, the owner responsible, and the tier at which the response escalates if it is not resolved in time.

Encoding the reaction as a table has three payoffs. It makes the reaction repeatable — every operator on every shift gets the same instruction for the same signal. It makes it testable — the table is a pure function from signal to action that can be unit-tested against every rule-and-characteristic combination. And it makes it auditable — the exact reaction prescribed for a given signal, and the version of the plan in force when it fired, are both recoverable. This is precisely what IATF 16949's reaction-plan requirement and ISO 9001:2015 clause 10.2 on nonconformity and corrective action expect a plant to be able to demonstrate.

The key implementation decision here is the granularity of the table's index. Keying the reaction on the rule alone is too coarse — a point beyond three sigma on a safety-critical torque means something very different from the same rule firing on a cosmetic dimension. Keying it on every possible signal-and-characteristic pair is too fine to maintain. The workable middle is to key on the rule together with a characteristic class (safety-critical, functional, cosmetic) so that the table stays small enough to review yet expressive enough that the reaction and its reaction-time budget scale with the consequence of the defect. That classification is also what lets the router assign severity consistently, which is why the two areas share the same rule taxonomy.

The two guides in this area are building an OCAP decision table in Python, which implements the lookup from signal to prescribed action, and escalation tiers and reaction time for OCAP, which adds the clock and the tier promotion that keep a reaction from stalling. Begin with the OCAP area overview for the decision-table model and how it consumes signals from the detection layer.

Real-Time Alert Notification Channels

Routing puts an event into a system of record; notification puts it in front of a person while it still matters. The two are separate concerns deliberately — a ticket that is created correctly but sits unread for an hour has still failed the operator on the line. The notification area covers delivering alerts to the channels people actually watch during a shift: team chat and on-call rotations, with the message carrying enough context (the process, the rule, a link back to the live chart) for a first responder to act without hunting.

The hard problem in notification is not sending — it is not over-sending. A sustained out-of-control condition, or a burst of correlated characteristics tripping at once, can generate hundreds of messages in minutes, and an operator who is flooded stops reading. The area therefore treats deduplication and throttling as first-class: collapse repeat firings of the same signal into one thread, rate-limit per process, and suppress downstream channels once an incident is acknowledged. This mirrors the idempotency logic in the routing area but applied to human attention rather than system records.

The key implementation decision in this area is which severities are allowed to interrupt a person and which merely accumulate in a record. Paging on every signal trains responders to mute the channel; paging on nothing defeats the point of real-time delivery. The practical policy routes only Major and Critical signals to an interruptive channel, sends Minor signals to a passive feed that a responder scans on their own cadence, and lets the OCAP's severity assignment — not the notification code — decide which is which. This keeps the human-attention budget spent on the signals that actually warrant a mid-shift interruption.

Its two guides are sending SPC alerts to Slack and Microsoft Teams, which builds the channel adapters and message payloads, and deduplicating and throttling SPC alert storms, which keeps a persistent signal from drowning the responders. The notification area overview ties both to the router so the same signal drives both the record and the human alert.

Implementation Principles

Across all three areas, a handful of principles keep the response layer reliable and defensible:

  • Route on a stable key, never on the raw firing. Derive the alert identity from process, rule, and window so a persistent signal maps to one incident and one notification thread. Idempotency is the single most important property in the layer; without it, every downstream system floods.
  • Separate detection from reaction. The layer consumes signals; it does not re-evaluate the rules. Keep the rule detection as the sole source of truth for whether a process is out of control, so a reaction can never disagree with the chart.
  • Every alert has exactly one owner. Broadcast-to-everyone is broadcast-to-no-one. The router assigns an owner at the moment of routing; the OCAP names the role responsible; escalation reassigns explicitly when a clock expires.
  • Put a clock on every reaction. A reaction without a bounded reaction time cannot be known to be late. Escalation tiers convert silence into a signal of its own.
  • Deliver at least once, act at most once. Retries and dead-letter paths guarantee the event is not lost; idempotency keys guarantee the retries do not multiply the effect.
  • Close the loop in an immutable log. Signal, routing decision, prescribed action, owner, escalation, and disposition are one traceable record. If the path from red point to closed nonconformity cannot be reconstructed, the layer has not done its job.
  • Fail loud, never silent. A router that cannot reach a destination must dead-letter and surface the failure, not swallow it. A delivery path that quietly stops working recreates the exact blind spot the layer exists to remove, and it does so invisibly.
  • Version the reaction, not just the record. The OCAP in force changes over time as processes and tooling change. Stamp each reaction with the plan version that prescribed it, so a disposition reviewed a year later is judged against the rules that actually applied when it was made.

These principles are deliberately conservative. The response layer is the part of an SPC system that touches people, tickets, and shop-floor systems directly, so its failures are the most visible and the most consequential. Favoring restraint, single ownership, and reconstructable records over cleverness is what keeps it trusted — and a trusted alerting system is the only kind that changes behavior on the floor.

Compliance and Standards

The response layer is where a plant's statistical practice meets its quality-management obligations. An auditor rarely challenges the arithmetic of a control limit; what they probe is what the plant did when a chart signaled, whether the reaction was defined in advance, whether it was followed, and whether the record proves it. Every one of those questions is answered by this layer rather than by the calculation engine, which is why the standards below govern the response machinery specifically. Anchor it to them explicitly:

  • AIAG SPC Reference Manual (2nd ed.) — establishes that a control chart is incomplete without a reaction plan, the OCAP concept this section implements. The manual's reaction-plan guidance is the statistical basis for the decision-table model.
  • IATF 16949 — requires a documented reaction plan for control-charted characteristics and evidence that it was followed. The router's owner assignment, the OCAP table, and the immutable log together produce that evidence.
  • ISO 9001:2015, clause 10.2 (nonconformity and corrective action) — governs what a plant must do when a nonconformity is detected: react, evaluate, correct, and record. An out-of-control signal is a candidate nonconformity, and this layer is the mechanism that discharges the clause's requirements automatically and traceably.
  • ISO 9001:2015, clause 9.1.1 (monitoring, measurement, analysis, and evaluation) — requires that the organization determine what needs monitoring, the methods, and when the results are evaluated. Timed escalation and audited disposition are how an automated system satisfies the evaluation half of the clause.
  • ISO 9001:2015, clause 7.5.3 (control of documented information) — the immutable, versioned alert-and-reaction record is the controlled documented information that proves conformance after the fact.

Cite these by name and clause in the plant's quality documentation; the response layer's purpose is to make every reaction it drives traceable back to them.

This section sits alongside the calculation and detection stack; return to the statistical-process-control.org home for the full topic map.