Pushing Out-of-Control Alerts to Jira and ServiceNow

This guide builds the ticketing adapter that turns an out-of-control signal into an incident in a system such as Jira or ServiceNow, and — critically — keeps a signal that persists across many scans from opening a fresh ticket every few seconds. It implements the upsert contract that the routing area's alert router depends on: a create-or-update operation keyed on an idempotency string derived from the process, the rule, and the time window. The client that talks to the ticketing REST API is injected, so the same adapter shape serves either backend and stays fully testable against a fake.

Prerequisites

  • Python 3.10+ with the AlertEvent, Violation, and TicketingAdapter definitions from the routing area importable.
  • A ticketing client object injected in, exposing search, create, and update methods and carrying its own authentication (an API token or OAuth session) — construct it in your composition root, never inside the adapter.
  • A custom field or label on the ticketing project that can hold the idempotency key and is indexed for search (for example a single-line text field spc_key).
  • A severity-to-priority mapping agreed with the quality team, so the adapter can set the ticket priority from the alert severity.
  • Upstream signals arriving from the rule-detection layer as Violation records with a valid process_id, rule_id, and point timestamp.

Step 1 — Define the incident payload

Map the AlertEvent to the fields the ticketing system expects. Keep the mapping in one function so both create and update paths build an identical body, and stamp the idempotency key into its own searchable field. Priority is derived from severity; the summary and description carry the statistical context a responder needs.

from __future__ import annotations

from typing import Any

# Alert severity -> ticket priority. Agree these labels with the quality team.
PRIORITY_BY_SEVERITY = {
    "CRITICAL": "P1",
    "MAJOR": "P2",
    "MINOR": "P3",
    "INFO": "P4",
}


def build_incident_fields(event: "AlertEvent", key_field: str = "spc_key") -> dict[str, Any]:
    """Build the create/update body for an SPC incident, shared by both paths."""
    v = event.violation
    summary = (
        f"[{event.alarm_code}] {event.process_id}: {event.rule_id} "
        f"(value {v.value:.4g}, limits {v.lcl:.4g}..{v.ucl:.4g})"
    )
    description = (
        f"Out-of-control signal on {event.process_id}.\n"
        f"Rule: {event.rule_id}\n"
        f"Alarm code: {event.alarm_code}\n"
        f"Point value: {v.value:.6g} (center {v.center:.6g})\n"
        f"Control limits: LCL {v.lcl:.6g}, UCL {v.ucl:.6g}\n"
        f"Signal time (epoch): {v.timestamp:.0f}"
    )
    return {
        "summary": summary,
        "description": description,
        "priority": PRIORITY_BY_SEVERITY.get(event.severity.name, "P3"),
        key_field: event.key,  # the idempotency key, in a searchable field
    }

The idempotency key lives in spc_key rather than being buried in free text, because the next step searches on it. Never encode the key only in the summary — summaries get edited by responders and the search then breaks.

Step 2 — Search for an existing incident by key

Before creating anything, ask the ticketing system whether an open incident already carries this key. Both Jira's REST search (a JQL query over the custom field) and ServiceNow's REST table query (an encoded query over the field) return matching records; the injected client hides that difference behind a search method that takes the key field and value and returns a list of record ids.

def find_existing(client: Any, key_field: str, key: str) -> str | None:
    """Return the id of an open incident carrying this idempotency key, or None."""
    matches = client.search(field=key_field, value=key, open_only=True)
    if not matches:
        return None
    if len(matches) > 1:
        # More than one open ticket for one key means a prior race created a
        # duplicate; return the oldest and let a cleanup job merge the rest.
        matches = sorted(matches, key=lambda m: m["created"])
    return matches[0]["id"]

The open_only=True filter matters: a closed incident from a previous, resolved excursion with the same window bucket should not be reopened silently. If the excursion recurs after closure, a new ticket is the correct behavior because it represents a new event to disposition.

Step 3 — Implement the idempotent upsert

Now assemble the adapter. upsert searches by key, updates the incident if one exists, and creates it otherwise, returning the record id either way. This is the exact method the router calls, and because it keys on event.key, calling it twenty times for one persistent excursion touches a single ticket.

class RestTicketingAdapter:
    """Ticketing adapter for a Jira- or ServiceNow-style REST backend.

    The transport client is injected, so this class is backend-agnostic and can
    be unit-tested against a fake client with no network access.
    """

    def __init__(self, client: Any, name: str, key_field: str = "spc_key") -> None:
        self.client = client
        self.name = name
        self.key_field = key_field

    def upsert(self, event: "AlertEvent") -> str:
        """Create or update the incident for event.key; return the record id."""
        fields = build_incident_fields(event, self.key_field)
        existing_id = find_existing(self.client, self.key_field, event.key)

        if existing_id is not None:
            # Update in place: refresh priority and append the latest observation.
            self.client.update(existing_id, fields)
            return existing_id

        return self.client.create(fields)

The adapter deliberately raises nothing itself — if client.create or client.update throws on a transport error, the exception propagates to the router, whose retry and dead-letter logic handles it. Keeping retry policy in the router and idempotency in the adapter is what makes at-least-once delivery combine with at-most-once effect.

Step 4 — Wire the adapter into the router

Construct the client in your composition root, wrap it in the adapter, and hand the adapter to the router alongside any others (such as the MES write-back adapter). One signal then drives both a ticket and a shop-floor write.

def build_router(jira_client: Any, mes_client: Any):
    """Compose the router with a ticketing adapter and an MES adapter."""
    from your_routing_module import AlertRouter, InMemoryDedupStore

    ticketing = RestTicketingAdapter(jira_client, name="jira", key_field="spc_key")
    # mes = MesWriteBackAdapter(mes_client, name="mes")  # from the sibling guide
    return AlertRouter(
        adapters=[ticketing],  # add mes here in production
        dedup_store=InMemoryDedupStore(),
        window_seconds=300,
    )

Verification

Prove idempotency with a fake client before touching a live project. The fake records create and update calls in memory, so the test asserts that a repeated signal produces exactly one ticket.

class FakeClient:
    def __init__(self) -> None:
        self.records: dict[str, dict] = {}
        self._next = 0

    def search(self, field, value, open_only=True):
        return [
            {"id": rid, "created": r["_created"]}
            for rid, r in self.records.items()
            if r.get(field) == value and (not open_only or r["_open"])
        ]

    def create(self, fields):
        self._next += 1
        rid = f"INC-{self._next}"
        self.records[rid] = {**fields, "_open": True, "_created": self._next}
        return rid

    def update(self, rid, fields):
        self.records[rid].update(fields)


def test_persistent_signal_makes_one_ticket():
    client = FakeClient()
    adapter = RestTicketingAdapter(client, name="jira")

    # Same event key delivered five times (a persistent excursion).
    event = make_event(key="abc123", process_id="LINE7-BORE-DIA")  # test helper
    ids = {adapter.upsert(event) for _ in range(5)}

    assert len(ids) == 1                      # one ticket, not five
    assert len(client.records) == 1           # nothing duplicated
    print("verified single ticket:", ids.pop())

Expected output is a single incident id and one record in the fake store. If five ids come back, the idempotency key is not reaching a searchable field — check that build_incident_fields writes spc_key and that the fake's search matches on it.

Root-Cause Table

Symptom Cause Fix
A new ticket appears every scan for one excursion The idempotency key is not written to a searchable field, so find_existing never matches Stamp event.key into a dedicated indexed field and search on it, not on the summary
Two open tickets share one key A race created a duplicate before the first was searchable Return the oldest match, alert, and let a cleanup job merge; consider a unique constraint on the key field
A resolved excursion silently reopens search did not filter to open incidents Pass open_only=True so a closed ticket with the same window is not revived; a recurrence should open a new ticket
Wrong priority on the ticket Severity name missing from the priority map Cover every Severity value in PRIORITY_BY_SEVERITY with a safe default
Transport errors crash the pipeline The adapter caught and swallowed exceptions Let transport errors propagate so the router's retry and dead-letter logic owns them

For the router abstraction, retry policy, and the full delivery backbone this adapter plugs into, return to routing SPC alerts to ticketing and MES systems.