Enforcing Spec Limits and Range Checks on Ingest

Every measurement batch that reaches a control chart has to clear three distinct gates at ingest: is the value a valid number in the right units and dtype, is it physically plausible for the sensor that produced it, and does it conform to the engineering specification for the part. This guide builds a single fail-closed validation function that enforces all three, routes bad rows to quarantine with a traceable reason code, and — critically — keeps spec-limit rejects separate from ingest errors. It is a hands-on technique within batch data validation and error handling.

The one distinction that governs everything below: a spec-limit violation is a real reject of a conforming-or-not part, while an ingest error is bad data that never should have entered the pipeline. Confuse them and you either scrap good parts or chart corrupt numbers. And spec limits are not control limits — a point outside a spec limit is a nonconforming unit, not a statistical signal.

Prerequisites

Confirm these before wiring the validator into ingest:

  • Python 3.10+ with pandas >= 2.0 and numpy >= 1.24 installed (pip install "pandas>=2.0" "numpy>=1.24")
  • Engineering LSL and USL per characteristic, sourced from the drawing or control plan — not from historical data, and not from any control chart
  • Physical range bounds per sensor (full-scale min and max, and any impossible values such as a negative diameter) documented from the gauge or transmitter datasheet
  • The expected dtype and unit of every measurement column (for example millimeters as float64, not a locale-formatted string), so a units mismatch is caught, not silently coerced
  • A schema contract already applied by validating CSV batch uploads against SPC schemas, so required columns are guaranteed present before this step
  • A quarantine sink (table, dead-letter file, or queue) that preserves the original row index for traceability back to the MES transaction

Step 1 — Separate the three gates conceptually

Before writing code, fix the three checks and their meanings, because each produces a different downstream action.

Dtype and units. A measurement exported as the string "12,4" (locale decimal comma) or in inches when the schema expects millimeters is an ingest error. The value is uninterpretable, so it fails closed and goes to quarantine with reason DTYPE_UNIT. It never reaches a spec check, because you cannot judge conformance of a number you cannot trust.

Physical range / plausibility. A diameter of -9999, a temperature of 1e6, or a pressure below absolute zero is physically impossible for the sensor. This is also an ingest error — a dropped reading or a railed transmitter — and quarantines with reason RANGE. It is emphatically not a spec reject; the part was never measured.

Engineering spec limits (LSL/USL). A cleanly measured, physically plausible diameter of 10.42 mm against a spec of 10.00 ± 0.30 is a real nonconformance. The data is good; the part is out of tolerance. This row is valid data that must still flow to the chart and disposition system, tagged SPEC_VIOLATION — you do not discard it, because a nonconforming part is a quality event to be counted, not a data defect to be hidden.

The spec formula is simply

$$\text{conforming} \iff \text{LSL} \le x_i \le \text{USL},$$

where LSL and USL come from the engineering specification. These are fixed tolerance boundaries set by design. They have nothing to do with the statistically derived control limits an X-bar chart computes from process data — a point can sit comfortably inside the spec yet still be out of statistical control, and vice versa. That distinction is developed in full in process capability analysis (Cp, Cpk, Pp, Ppk), which is exactly the ratio of spec width to process spread.

Step 2 — Enforce dtype and units, failing closed

Coerce every measurement column to numeric first, so a stray string becomes NaN rather than crashing a later comparison or silently promoting the column to object dtype. Any value that fails to coerce is an ingest error.

import numpy as np
import pandas as pd


def check_dtype_units(frame: pd.DataFrame, value_col: str) -> pd.Series:
    """Return a boolean mask: True where a value is an uninterpretable ingest error.

    A non-numeric payload (locale comma, unit suffix, sentinel text) coerces to
    NaN and is treated as a fail-closed dtype/unit error.
    """
    coerced = pd.to_numeric(frame[value_col], errors="coerce")
    original_na = frame[value_col].isna()
    # New NaN introduced by coercion == a value that was present but unparseable.
    return coerced.isna() & ~original_na

Verify in isolation: pass a frame containing "12,4" and "10.1" and assert the comma string is flagged while the clean numeric string is not. Failing closed here means the ambiguous value is rejected, never guessed at — a validator that repairs "12,4" into 12.4 fabricates data.

Step 3 — Enforce physical range and plausibility bounds

Range bounds catch dropped readings and railed sensors. Reject anything outside the sensor's documented full-scale window; never clip, because clipping invents a plausible-looking number that hides the fault.

def check_physical_range(
    values: pd.Series, low: float, high: float
) -> pd.Series:
    """Return a boolean mask: True where a value is physically impossible.

    `low`/`high` are the sensor's full-scale limits, not engineering tolerances.
    NaN (already-missing or coercion failure) is not flagged here; it is handled
    by the dtype gate so reason codes stay distinct.
    """
    if low >= high:
        raise ValueError("physical range must have low < high")
    present = values.notna()
    return present & ((values < low) | (values > high))

Verify: inject a -9999 sentinel into a diameter column bounded [0.0, 50.0] and assert it is flagged RANGE, while an in-range 10.42 — even though it will later fail the spec check — is not flagged here. The gates must stay orthogonal: a bad-data reject and a bad-part reject are different events.

Step 4 — Evaluate engineering spec limits (LSL/USL)

Now, and only now, judge conformance. A row that reaches this gate is a trustworthy number, so an out-of-spec value is a genuine nonconformance, not a data problem. Tag it, keep it, and let it flow downstream — the part still happened and must be counted.

def check_spec_limits(
    values: pd.Series, lsl: float, usl: float
) -> pd.Series:
    """Return a boolean mask: True where a valid measurement is out of tolerance.

    LSL/USL are engineering spec limits from the drawing or control plan.
    These are NOT control limits: a conforming part can still be out of
    statistical control, and an out-of-spec part can still sit inside 3-sigma
    control limits. Spec conformance and statistical control are independent.
    """
    if lsl >= usl:
        raise ValueError("spec limits must have LSL < USL")
    present = values.notna()
    return present & ((values < lsl) | (values > usl))

Verify: with a spec of 10.00 ± 0.30 (LSL 9.70, USL 10.30), assert that 10.42 is flagged SPEC_VIOLATION and 10.05 is not. Confirm that a SPEC_VIOLATION row is never sent to quarantine — it is valid data.

Step 5 — Compose the fail-closed validator

Combine the gates so that the two ingest-error checks (dtype, range) route to quarantine while the spec check only labels. The function returns two frames: clean for charting and disposition, and quarantine for the dead-letter path. Every row keeps its original index.

def validate_on_ingest(
    frame: pd.DataFrame,
    value_col: str,
    phys_range: tuple[float, float],
    spec_limits: tuple[float, float],
) -> tuple[pd.DataFrame, pd.DataFrame]:
    """Validate a measurement batch at ingest against dtype, range, and spec gates.

    Ingest errors (unparseable dtype/units, physically impossible values) fail
    closed and are quarantined. Spec-limit violations are real nonconformances:
    they are labeled but kept in the clean frame for charting and disposition.

    Returns
    -------
    (clean, quarantine) : tuple of pd.DataFrame
        `clean` carries a `status` column (PASS or SPEC_VIOLATION) and coerced
        numeric values; `quarantine` carries a `reason` column and the original
        payload. Original index preserved on both.
    """
    out = frame.copy()
    coerced = pd.to_numeric(out[value_col], errors="coerce")

    dtype_bad = check_dtype_units(out, value_col)
    range_bad = check_physical_range(coerced, *phys_range)

    reason = pd.Series(pd.NA, index=out.index, dtype="object")
    reason[range_bad] = "RANGE"
    reason[dtype_bad] = "DTYPE_UNIT"  # dtype wins: an unparsed value has no range

    quarantine = out[reason.notna()].copy()
    quarantine["reason"] = reason[reason.notna()]

    clean = out[reason.isna()].copy()
    clean[value_col] = coerced[reason.isna()]
    spec_bad = check_spec_limits(clean[value_col], *spec_limits)
    clean["status"] = np.where(spec_bad, "SPEC_VIOLATION", "PASS")
    return clean, quarantine


# Usage:
# clean, quarantine = validate_on_ingest(
#     batch, "diameter_mm", phys_range=(0.0, 50.0), spec_limits=(9.70, 10.30)
# )

Verification

Run the whole gate against a fixture that exercises all four outcomes — clean pass, spec violation, range error, and dtype error — and assert each row lands in the right place with the right label.

import pandas as pd

batch = pd.DataFrame(
    {
        "diameter_mm": ["10.05", "10.42", -9999.0, "12,4"],
        "part_id": ["A1", "A2", "A3", "A4"],
    }
)

clean, quarantine = validate_on_ingest(
    batch, "diameter_mm", phys_range=(0.0, 50.0), spec_limits=(9.70, 10.30)
)

# A1 conforms, A2 is out of spec but valid data -> both stay in clean.
assert list(clean["part_id"]) == ["A1", "A2"]
assert list(clean["status"]) == ["PASS", "SPEC_VIOLATION"]

# A3 (railed sensor) and A4 (locale comma) are ingest errors -> quarantine.
assert list(quarantine["part_id"]) == ["A3", "A4"]
assert list(quarantine["reason"]) == ["RANGE", "DTYPE_UNIT"]

# Nothing was lost: every original row is accounted for.
assert len(clean) + len(quarantine) == len(batch)
print("all four outcomes routed correctly")

Expected output: all four outcomes routed correctly. The load-bearing assertions are that the spec violation stays in clean (a nonconforming part is a counted event, not a discarded one) and that row count is conserved (nothing is silently dropped).

Root-cause table

Symptom Cause Fix
Good parts scrapped as bad data Spec violations routed to quarantine instead of labeled Keep SPEC_VIOLATION rows in the clean frame; only dtype and range errors quarantine (Steps 4–5)
Control chart limits balloon, no alarms fire A -9999 sentinel passed the gate and inflated the range estimate Enforce physical bounds and reject sentinels; never clip (Step 3)
A measurement column silently becomes object dtype Locale decimal comma or a unit suffix left a string in the column Coerce with pd.to_numeric(errors="coerce") and fail closed on the resulting NaN (Step 2)
Operators told a part is defective when it is fine Spec limits confused with statistically derived control limits Judge conformance against LSL/USL only; treat control limits as a separate statistical question (Step 4)
Rejected rows vanish with no trace Validator deletes instead of quarantining Route failures to a dead-letter sink with a reason code and the original index (Step 5)

Preserve the original index and the reason code on every quarantined row so each rejection traces back to a physical MES transaction. This is a traceability requirement, not a convenience (ISO 9001:2015 clause 7.1.5 on monitoring and measuring resources; IATF 16949 clause 9.1.1.1 on statistical control; NIST/SEMATECH e-Handbook §6.3.2 against undocumented data removal). For teams pulling batches over the wire, carry the reason code through the same audit path used when connecting Python to MES and SCADA systems.

FAQ

Why are spec limits not the same as control limits?

Spec limits (LSL and USL) are fixed engineering tolerances set by the design of the part — they define what makes a unit conforming. Control limits are computed from the process data itself and describe the natural, statistical variation the process actually exhibits. A process can run in perfect statistical control yet produce parts outside spec (it is capable of nothing better), and it can produce every part inside spec while drifting out of statistical control. Because they answer different questions, this ingest gate judges conformance against spec limits only and leaves statistical control to the charting layer. Their relationship is quantified by capability indices such as Cpk.

Should an out-of-spec measurement be quarantined?

No. An out-of-spec measurement is good data describing a nonconforming part, so it must flow downstream to be counted, charted, and dispositioned. Quarantine is only for ingest errors — values that are uninterpretable (wrong dtype or units) or physically impossible (a railed or dropped sensor). Hiding a spec violation in quarantine would understate your defect rate and corrupt capability analysis. Label it SPEC_VIOLATION, keep it in the clean stream, and let the quality system act on it.

Up one level: Batch Data Validation and Error Handling. For the full ingestion sequence see Manufacturing Data Ingestion and Preprocessing.