Set inspection thresholds with a confusion matrix

What the test resolves

Threshold tuning must balance defective escapes and good-part rejects. Overall accuracy can conceal escapes when most samples are good.

This is a vendor-neutral commissioning method. Check the exact camera, lens, sensor or controller manual before changing settings. The numerical worksheet is hypothetical, not a Deyi Vision measurement or a product specification.

Operating prerequisites

Define positive as defective and predicted positive as reject. Establish independent ground-truth labels and split data by physical part/lot so repeated images cannot leak across tuning and validation.

Define a positive result as a defective part and a positive prediction as a reject, then record that convention wherever the matrix appears. Obtain truth labels independently of the detector being tuned. Set the intended cost and allowable rate of missed defects and false rejects before selecting a threshold. Avoid mixing uncertain labels into the tuning set without a documented adjudication method; disagreement in truth can imitate classifier error.

Bench and records

Prepare labeled samples covering defect severity, positions and material lots. Preserve continuous detector scores, image IDs and disputed labels; agree escape/reject limits before seeing results.

Create a part ledger with unique physical part ID, lot, defect label, label evidence, raw image, detector score and split assignment. Keep repeated images of the same part and related lot observations within the same split to reduce leakage. Reserve a validation set that is not consulted during threshold tuning. Record the score direction and preprocessing version so an apparently lower threshold always has an unambiguous interpretation.

Run the test

1. Tune candidate thresholds only on the development set, keeping the reject inequality explicit.
2. Count TP, FN, FP and TN with fixed axis/label order for each candidate.
3. Choose the operating point against agreed limits, then lock threshold and preprocessing.
4. Run the untouched validation set and a production-representative mixture; report both counts and denominators by defect type.

Evaluate candidate thresholds on the tuning set and retain the individual predictions, not only aggregate rates. Construct counts with explicitly named true and predicted axes, checking each mismatch against its raw image and truth evidence. Select a candidate using the predeclared operational policy, then freeze it. Run the untouched validation set once for acceptance; if it prompts another threshold change, obtain independent validation evidence for the revised candidate.

Hypothetical worksheet

Hypothetical100 defective and1000 good parts give TP92,FN8,FP20,TN980: recall92%, false rejects2%, precision82.14%. A lower threshold yielding TP98,FN2,FP50,TN950 reduces escapes but raises false rejects to5%; neither point is automatically acceptable.

Positive = defective part; predicted positive = reject
TP=92; FN=8; FP=20; TN=980
recall = 92 / (92 + 8) = 92%
false_reject_rate = 20 / (20 + 980) = 2%
precision = 92 / (92 + 20) = 82.14%
At a second hypothetical threshold: TP=98,FN=2,FP=50,TN=950

The example shows a tradeoff: fewer missed defects accompany more false rejects at the second operating point. Recall uses all truly defective parts, false-reject rate uses all truly acceptable parts, and precision uses all rejected parts. Preserve these denominators with the counts. Neither example is automatically acceptable without the process limits. A precision measured on this constructed test mixture should not be presented as the expected precision under a different production defect prevalence.

Test sequence

Interpret the evidence

Read the matrix by inspecting both error cells, then review each cell's representative parts for common failure mechanisms. Separate defect subtypes, positions and material conditions when those distinctions affect risk. A good overall recall can conceal poor performance on a required rare defect. Likewise, repeated harmless texture can dominate false rejects. The diagram supports diagnosis, but the saved part ledger is needed to connect aggregate performance to actionable cases.

Troubleshooting

• Very high accuracy with low defect recall is class imbalance, not success.
• Repeated shots of one part in both sets leak information; resplit by part/lot before claiming validation.

If validation performance collapses after tuning, investigate leakage, material shifts and changes in score preprocessing before searching another threshold. If score polarity is reversed, correct the interpretation rather than relabeling the truth classes. If a supposedly defective sample is ambiguous, preserve the disagreement and obtain independent adjudication. Observing no missed defects in a finite set does not establish a zero escape probability; report the tested counts and scope honestly.

Acceptance and handoff

Accept only the locked setting meeting separate escape and false-reject limits on independent data. State counts, sample mix and untested defect classes. Zero observed escapes on a finite set does not establish zero escape probability.

Archive the raw files, settings, sample identifiers and decision limits together. Discuss the documented requirements with Deyi Vision using the related product and preparation guides below.

Approve the frozen threshold only when the independent set satisfies the agreed class-specific limits and required defect coverage. Verify the production pipeline uses the same score direction, preprocessing and recipe version, and that borderline or invalid inputs follow the stated policy. Archive individual predictions, class counts and error examples. Define which changes in lighting, materials, camera settings or defect population require new validation instead of allowing silent threshold drift.