How to validate a machine vision application: a practical checklist

Why validation comes before procurement

Most failed vision projects do not fail at the camera; they fail because nobody agreed, in writing and in numbers, on what the system was supposed to do. An integrator is handed a vague brief ('detect bad parts'), buys plausible hardware, and discovers on the line that 'bad' meant six different defect types, three of which are invisible under the chosen lighting. Validation is the discipline of answering every open question on a bench, with real parts, before money and downtime are committed. It is the cheapest insurance in the entire project.

Treat validation as a gate with a fixed sequence: define the inspection, gather the evidence (samples), prove you can image the feature, size the optics and sensor, control the part in motion, decide the algorithm strategy, and only then run a proof of concept (POC) scored against pre-agreed acceptance criteria. Each step can kill the project early and cheaply, which is exactly what you want. A defect you cannot photograph is a showstopper you would far rather discover in week one than after installation.

This checklist is deliberately ordered so the hardest, most failure-prone questions come first. Imaging physics and sample variability sink far more projects than software does. By the time you are choosing between classical and deep-learning tools, the riskiest unknowns should already be closed.

Step 1 - Define the defect and the tolerance

Write a defect catalogue. For each defect class, record a plain-language description, a representative image or sketch, the smallest instance that must be caught (in millimetres or as a fraction of the feature), and the largest acceptable instance that must pass. That last point is critical and frequently skipped: inspection is about a decision boundary, not about 'finding flaws'. A 0.1 mm scratch may be cosmetic and acceptable while the same scratch at 0.5 mm is a reject. The system is being asked to place a line between pass and fail, so that line has to exist on paper first.

Separate the two fundamentally different tasks the word 'inspection' hides: metrology (measuring a dimension, position or count against a numeric spec with a stated tolerance) and defect detection (deciding whether an anomaly is present). Gauging a bore diameter to a few tens of microns is a different engineering problem from spotting a random surface stain, and the two often need different cameras, lighting and algorithms. List which features need true measurement and which need presence/absence or cosmetic judgement.

Quantify the smallest feature of interest. This single number, the smallest defect dimension you must resolve, drives the entire resolution budget later. If the team cannot agree on it, that disagreement is the real deliverable of this step. Also capture the cost asymmetry: a missed defect that ships a bad part (a false accept, or escape) usually carries a very different cost than a good part wrongly rejected (a false reject), and that imbalance will shape your acceptance criteria.

• For each defect: description, reference image, minimum detectable size, maximum acceptable size.
• Classify each check as metrology (dimension/position/count with tolerance) or defect detection (presence/cosmetic).
• State the cost asymmetry: which is worse, a missed defect (escape) or an over-reject?
• Define the smallest feature of interest in millimetres - it sets the resolution budget.

Step 2 - Collect representative samples (good, bad, edge)

A vision system can only be validated against the variation you put in front of it. Before any imaging, collect a physical sample set that spans the real population: clearly good parts, clearly bad parts for every defect class, and - most importantly - the ambiguous edge cases that sit near the decision boundary. Edge cases are where systems live or die. A POC that only sees obvious 'golden' parts and obvious gross failures will look brilliant in a demo and collapse on a production line full of marginal parts.

Capture the natural nuisance variation the line will throw at the system but that is not itself a defect: part-to-part colour and finish differences, allowed positional play in the fixture, batch and supplier changes, surface texture, and cosmetic marks that are acceptable. If the system must tolerate these, they have to be in the sample set, or you will be tuning against a fantasy. For deep-learning approaches this is doubly true, since the model learns only the distribution it is shown; for classical approaches the same variation determines how tight your thresholds can be without triggering false rejects.

Be honest about quantities and rarity. Common defects are easy to collect; the dangerous ones are often rare, which makes them both the most important to catch and the hardest to sample. Plan how rare defects will be obtained - saved from scrap bins, deliberately manufactured, or seeded - and label everything with ground truth (which part is which class, ideally adjudicated by the quality team). Keep a sealed validation subset that is never used during tuning, so the final acceptance test runs against parts the system has never seen.

• Three buckets: confident good, confident bad (every defect class), and borderline edge cases.
• Include allowed nuisance variation (finish, fixturing play, batch/supplier changes) explicitly.
• Plan for rare-defect scarcity: scrap, seeded or manufactured samples with ground-truth labels.
• Hold back an untouched validation set for the final acceptance run.

Step 3 - Prove the imaging: lighting, lens, camera and filters

The most important rule in machine vision: if a trained operator cannot reliably see the defect in the raw, unprocessed image, no algorithm will reliably find it either. So the first imaging task is not to choose a camera, it is to make the feature pop, and that is overwhelmingly a lighting problem. The geometry of illumination - bright field versus dark field, diffuse versus directional, backlight, coaxial/on-axis, dome, low-angle grazing - decides whether a scratch, dent, edge or print becomes high contrast or vanishes into the surface. Engineers routinely spend more effort on lighting than on any other single component, because it is the cheapest place to win contrast and the most expensive place to get wrong.

Match the technique to the feature. Backlighting turns dimensional gauging and presence checks into clean silhouettes with crisp edges. Low-angle dark-field illumination makes surface topography - scratches, embossing, engraving - light up against a dark background. Diffuse dome or coaxial light suppresses glare on specular, curved or glossy metallic surfaces. Then consider wavelength: choosing a colour of light (or a monochrome camera with a bandpass filter) can boost contrast between two materials, a polarising filter can cut specular hot-spots, and infrared or UV can reveal features that are invisible under white light. These choices interact with the sensor: monochrome cameras are generally more light-sensitive and resolve finer detail than colour ones of the same resolution, because they have no colour filter array and need no demosaicing, while colour is needed when the defect or the sort is defined by colour itself.

The lens is the other half of the optical chain and is not a place to economise. It must resolve the smallest feature - its resolving power, often expressed in line-pairs per millimetre, has to comfortably exceed what the sensor's pixel pitch demands - cover the sensor format without vignetting, and provide enough depth of field for the part's height variation. For precision gauging, a telecentric lens holds magnification constant across its working range, so a part's apparent size does not change as it moves nearer or further within that range, which removes the parallax error that wrecks measurement to tens of microns. Working distance, mounting space and whether you can physically fit the light and camera around the part are practical constraints to check against the real machine, not assume.

Close this step with a documented imaging recipe: the exact lighting geometry, colour and intensity, the lens and aperture, the working distance, the camera and exposure, and any filter. Save reference images of good, bad and edge samples captured under that recipe. If the contrast margin between a borderline-good and a borderline-bad part is thin in those images, stop and fix the optics now - tightening software thresholds against a weak image only buys false rejects.

• Lighting first: pick geometry (backlight, dark field, diffuse/dome, coaxial, low-angle) for the feature.
• Use wavelength and filters (bandpass, polariser, IR/UV) to add contrast or kill glare.
• Lens must out-resolve the sensor, cover the format and hold depth of field; consider telecentric for precision gauging.
• Acceptance for this step: the defect is obvious to a human in the raw frame, with margin on the edge cases.

Step 4 - Budget field of view and resolution

Resolution is an engineering budget, not a marketing number. Three quantities lock it together: the field of view (FOV) you must cover, the size of the smallest feature you must resolve, and how many pixels you need across that feature to make a reliable decision. Sampling theory sets a hard floor - you need at least two pixels across a feature just to register it - but two is never enough in practice once blur, noise and sub-pixel positioning are accounted for. As a robust working rule, allocate at least 3 to 5 pixels across the smallest defect: nearer 3 for a high-contrast presence check, 5 or more for measurement or subtle cosmetic work.

Work it through with a simple chain. Spatial resolution (millimetres per pixel) is the FOV dimension divided by the sensor's pixel count along that dimension. Multiply the smallest feature by your required pixels-per-feature to get the spatial resolution you need, then divide the FOV by that to get the minimum sensor pixel count. For example, a 100 mm wide FOV that must resolve a 0.2 mm defect with 3 pixels across it needs about 0.2 / 3 = 0.067 mm per pixel, which is roughly 1500 pixels across the width - and you should then add headroom for optical blur, lens MTF roll-off and part alignment. Treat these as design figures, not guarantees: the lens, the lighting contrast and the algorithm all erode the resolution the datasheet implies.

Resolution does not live in isolation. Pushing pixel count up raises the data rate, which feeds back into the interface choice and the processing budget you will confirm later, and a larger sensor needs a larger-format lens and shrinks depth of field at a given aperture. If a single camera cannot cover the FOV at the detail you need, that is a signal to split the view across multiple cameras, move the part under a line-scan camera, or tighten the fixture - not to chase an ever-higher megapixel count that the optics cannot deliver anyway.

• Spatial resolution (mm/px) = FOV dimension / sensor pixels along that axis.
• Minimum pixels per axis = (FOV / smallest feature) x pixels-per-feature (3-5 px).
• Add margin for optical blur, MTF roll-off and alignment; the datasheet figure is a ceiling, not a promise.
• If one camera cannot cover the FOV at the needed detail, use multiple cameras, line scan or a tighter fixture.

Step 5 - Lock down motion, triggering and timing

A perfect image on a static bench means nothing if the part blurs, drifts or is mistimed on the line. Decide first whether you inspect the part stationary (indexed to a stop) or in continuous motion. Continuous motion demands a short exposure to freeze it, and a short exposure demands a lot of light, usually delivered as a bright source strobed in sync with the trigger. It also points strongly to a global-shutter camera, which exposes the whole sensor at one instant; a rolling shutter exposes rows sequentially and will skew or smear anything moving during the readout.

Pin down triggering and synchronisation. Something physical - a photoelectric sensor, a proximity switch, or an encoder pulse on a moving web - must tell the camera when to grab, and that signal has to be repeatable so the part lands in the same place in every frame. Estimate the positional jitter the trigger and fixture allow, and confirm it still fits inside the FOV margin you budgeted in Step 4. For line-scan imaging the encoder is not optional: line rate must be locked to web speed or the image stretches and compresses with every speed change.

Finally, build a timing budget against the real cycle time. Add up exposure, sensor readout, image transfer over the chosen interface, and processing, and confirm the total fits within the time the line allows per part, with margin. A system that is accurate but too slow fails just as surely as one that is fast but wrong. Running an interface or a processor near its ceiling invites dropped frames and intermittent faults that are painful to debug on a production floor, so size both above your peak load, not at it.

• Static-and-indexed or moving? Motion forces short exposure, more light and usually a global shutter.
• Trigger from a real, repeatable signal (photoeye, proximity switch, encoder); check positional jitter against FOV margin.
• Line scan needs encoder-locked line rate or the image distorts with speed.
• Build a cycle-time budget: exposure + readout + transfer + processing must fit with margin.

Step 6 - Choose the algorithm strategy: classical, deep learning or both

Only once the image is proven and the part is under control does the software strategy become a sensible question - and the honest answer is that the imaging usually decides it. Classical (rule-based) machine vision is the right tool for features you can specify: dimensions, positions, edges, blob counts, presence/absence and code reading, under controlled lighting. It is deterministic, fast, explainable, needs no training data, and is the only credible choice for true metrology, where you must report a number with a stated tolerance.

Deep learning earns its place where a feature is real and obvious to a human but genuinely hard to write a rule for: variable cosmetic defects on textured or natural surfaces, defects of no fixed shape, or sorting by overall appearance. The trade is data and explainability - a model is only as good as the labelled, representative samples from Step 2, it generalises poorly to variation it never saw, and its decisions are harder to justify and audit. It does not rescue a weak image; a deep network on a low-contrast frame fails the same way a threshold does.

Many production lines use both: classical tools for the gauging and code reading, a learned model for the cosmetic judgement, fused into one pass/fail. Whichever you pick, the choice must trace back to the defect catalogue. If a defect cannot be specified geometrically and cannot be reliably labelled from samples, you have a sampling or imaging problem to fix before any algorithm will help - which is exactly why this decision comes near the end of the checklist, not the start.

• Classical vision: measurable, repeatable, well-lit features; deterministic, explainable, no training data, mandatory for metrology.
• Deep learning: variable, hard-to-specify cosmetic defects; needs labelled, representative data and is harder to audit.
• Neither approach fixes a bad image - resolve imaging and sampling first.
• Combining both is common: classical for gauging and codes, learned models for cosmetic judgement.

Step 7 - Run a scored proof of concept against acceptance criteria

A proof of concept without numbers is a demo, not a validation. Before running it, agree the acceptance criteria in writing: the required detection rate per defect class (especially for the costly escapes from Step 1), the maximum tolerable false-reject rate on good parts, measurement accuracy and repeatability where you are gauging, and the cycle time the line demands. These targets come straight from the cost asymmetry you defined at the start, so the escapes you most want to avoid get the tightest bar.

Run the POC on the sealed validation set held back in Step 2 - parts the system has never been tuned against - and score every result against ground truth in a confusion matrix: true and false accepts, true and false rejects, per defect class. Headline accuracy hides the failures that matter, so read the matrix, not the average. For measurement tasks, run a repeatability study (image the same parts many times, and across operators or setups if relevant) to separate the system's own variation from real part variation; a gauge whose noise swamps the tolerance cannot be trusted whatever its average reads.

Treat the POC as a documented experiment with a verdict, not a sales meeting. Record the full imaging recipe, the sample provenance and labels, the software configuration, and the scored results, so the test can be repeated and audited. If a criterion is missed, the ordered checklist tells you where to look - thin contrast sends you back to lighting and optics, missed edge cases to the sample set, blur or mistiming to motion and triggering. Passing every pre-agreed criterion on unseen parts is what converts a promising bench setup into a system you can buy and install with confidence.

• Set numeric targets up front: detection rate per defect class, max false-reject rate, measurement accuracy/repeatability, cycle time.
• Score on the untouched validation set with a confusion matrix per class - never on parts used for tuning.
• For gauging, run a repeatability study; the system's own variation must be small against the tolerance.
• Document recipe, samples, config and results; on failure, walk the checklist back to the offending step.