Insights
Ultrasound Phantom Comparisons: Measure Repeatability Before Declaring Improvement
When a processing change improves an ultrasound phantom metric, does the difference exceed acquisition and analysis variability? A source-led OEM comparison protocol with explicit repeatability and transfer limits.
By ALEVA | Orange County, California, USA | November 8, 2026
Engineering question: When an ultrasound processing change improves a phantom metric, how do you establish whether the change exceeds variation introduced by acquisition and analysis?
A better image is a measurement claim
A side-by-side image can help identify a candidate improvement. An OEM acceptance decision needs a defined measurement procedure: what was measured, where it was measured, and which conditions were held constant. A processing revision and a different probe placement should not silently become the same experimental variable.
A 2026 peer-reviewed study by Yu and colleagues evaluated ultrasound imaging performance using freehand scans of a random hypoechoic sphere phantom. It assessed a depth-dependent lesion signal-to-noise ratio and examined variability associated with operators and scanning speed. Its results support evaluating the measurement procedure alongside the imaging configuration; they do not establish a universal acceptance limit for every phantom, probe or product. [1]
Define the comparison before collecting images
For a proposed OEM verification plan, first state the engineering claim. Is the revision intended to improve target detectability at a specified depth, preserve a resolution measure, or reduce an artifact? Choose the measurement and analysis region before inspecting the A/B result. If several metrics matter, specify which must improve and which must remain within an agreed tolerance.
Record the phantom model and serial number, probe, acquisition settings, processing revision and analysis procedure. Preserve the source data required to repeat the calculation. For the proposed protocol, document environmental conditions and follow the handling instructions for the actual phantom in use.
NIST's Engineering Statistics Handbook separates short-term repeatability, day-to-day reproducibility and run-to-run stability in its treatment of measurement variability. This is foundational measurement guidance, not an ultrasound-specific acceptance standard. It motivates keeping the sources of variation visible instead of reporting only one pooled spread. [2]
Separate three questions
Analysis stability: Reprocess the same saved acquisition with the same analysis settings. This checks whether the analysis can reproduce its own result. It does not test acquisition repeatability.
Acquisition repeatability: Repeat the capture under the defined procedure, including independent repositioning when placement is part of normal use. Treat closely spaced frames from one unchanged placement as related observations rather than automatically counting each frame as an independent experiment.
Transfer to another operator or setup: Repeat the procedure with the additional operator, device or site relevant to the intended handover. Keep those results identifiable so that a change in operator is not hidden inside one pooled average.
Use a paired comparison with visible limits
Where practical, compare A and B under matched acquisition conditions and counterbalance their order. Report the paired differences together with the spread across independent captures. Retain depth-specific results when the engineering claim depends on depth. A single overall score can conceal an improvement in one region and a deterioration in another.
A fixture can reduce placement variability, but it may test a narrower situation than freehand use. Reprocessing identical raw data isolates processing differences, but it does not exercise a changed acquisition chain. Select the experiment that answers the intended question, then state what it leaves untested.
The OEM decision
The proposed acceptance record should contain the frozen protocol, configuration identifiers, independent-capture definition, results, observed variation and decision rule. Evaluate the paired differences with an uncertainty or statistical analysis appropriate to the design; a comparison of the mean difference with one standard deviation alone is not a universal decision rule. If the evidence does not resolve the predefined improvement criterion, report the result as inconclusive. Collecting more nearly identical frames is not automatically the remedy.
This is an engineering test-design proposal, not evidence of measured ALEVA hardware performance or clinical effectiveness. Product-specific acceptance limits require an agreed application and validation plan.
References
Yu et al. “Assessing imaging performance of ultrasound systems using a random hypoechoic sphere phantom with freehand scanning.” Medical Physics, 53(1), e70278, published January 15, 2026. Peer-reviewed. DOI: 10.1002/mp.70278; public full text.
NIST/SEMATECH. Engineering Statistics Handbook, Chapter 2: Measurement Process Characterization (2003 publication record), section 2.4.4, Analysis of variability. Foundational guidance; accessed November 8, 2026. Publication record.