Testing, Iteration and Feedback
Test planning, sample sizes, user testing, designed experiments, failure analysis and learning from each build.
Plan the test before you build
A test is a question with a number attached. “It should be strong” is not a test; “it should hold 5 kg for 1 000 cycles with less than 1 mm deflection” is. Write the test first, because it decides what the prototype must be and what you must measure.
| Test plan item | Example |
|---|---|
| Question | Does the latch hold under a 30 N pull for 5 000 cycles? |
| Pass / fail criterion | No breakage; release force within 5 to 15 N throughout |
| Samples | 5 prototypes from the same process (see below) |
| Method and set-up | Cycling rig, a force gauge, a counter; room temperature |
| Measurements | Release force every 500 cycles; photos of cracks |
| Decision | Pass: keep the design; fail: lengthen the hook and repeat |
Measure properly
- Use instruments that resolve at least 10 times finer than the tolerance: to judge a ±0.1 mm feature you need a resolution of 0.01 mm or better (calipers read 0.01 mm but are good to about 0.03 mm).
- Repeat measurements and use the average; note the spread.
- Calibrate or check against a known standard (a gauge block, a reference weight) before the test.
- Record raw numbers, units and conditions (temperature, date, who), not just conclusions: see Data Logging.
- Photograph the set-up and the failures.
Worked example: how sure is the average?
- Five measurements of a force have a standard deviation of 0.12 N around a mean of 10.00 N. The 95 % confidence interval of the mean is ± t × s ÷ √n = 2.776 × 0.12 ÷ √5 = ±0.15 N (t = 2.776 for 4 degrees of freedom).
- With 20 measurements the interval narrows to ±0.06 N (t = 2.093): four times as many samples halve the uncertainty. A difference smaller than the interval is not a real difference.
How many samples?
If you test n parts and none fails, how reliable can you claim the design to be? For a zero-failure (success-run) test the number of samples needed to demonstrate a reliability R with confidence C is n = ln(1 − C) ÷ ln R.
| Reliability to show | Confidence | Samples, no failures allowed |
|---|---|---|
| 90.0 % | 90 % | 22 |
| 95.0 % | 90 % | 45 |
| 99.0 % | 90 % | 230 |
| 95.0 % | 95 % | 59 |
| 99.0 % | 95 % | 299 |
| 99.9 % | 90 % | 2 302 |
Worked example: what does a passed test of 30 parts show?
- To show 90 % reliability at 90 % confidence you need n = ln 0.1 ÷ ln 0.9 = 22 parts with no failures.
- If 30 parts all pass, the failure rate could still be as high as 1 − 0.051/30 = 9.5 % (95 % confidence): the “rule of three” gives 3 ÷ 30 = 10.0 %. Thirty passes do not prove the failure rate is zero.
- Prototypes are usually too few for statistics: use them to find how it fails, and use bigger samples later (see From Prototype to Production).
Testing with users
| Test users | Problems found (model) |
|---|---|
| 1 | 31 % |
| 3 | 67 % |
| 5 | 84 % |
| 8 | 95 % |
| 10 | 98 % |
| 15 | 100 % |
Worked example: five users or fifteen?
- If each user finds about 31 % of the problems, n users find 1 − (1 − 0.31)n: five users find 84 % and fifteen find 100 %. To reach 85 % you need 6 users.
- The better plan is three rounds of five with fixes in between: the fixes clear the common problems so that the next five users find new ones, which one big test of fifteen users cannot do.
- The model is a rule of thumb: real problems differ in how easy they are to find, and rare ones need more users.
- Watch, do not explain. Hand over the prototype with a task and keep quiet; where people hesitate is where the design is unclear.
- Choose users like your customers, not colleagues who know the project.
- Record what people do (video, notes) rather than only what they say.
- Ask open questions (“what did you expect?”) rather than leading ones.
- Test the rough prototype (see Low-Fidelity Prototyping: Mock-ups and Models): users forgive roughness and say more about the idea.
Designed experiments
Changing one factor at a time is slow and misses interactions. A factorial experiment changes all the factors together in a planned pattern and separates their effects.
Worked example: three factors in eight runs
- Three factors, each at a low (−) and a high (+) level, need 23 = 8 runs for a full factorial (4 for a half-fraction). For five factors a full factorial takes 32 runs, a half fraction 16, while one-factor-at-a-time takes 6 runs but cannot see interactions.
- A print-strength test: A = layer height, B = infill, C = nozzle temperature; the result is the breaking load in newtons.
| A | B | C | Breaking load (N) |
|---|---|---|---|
| − | − | − | 62 |
| + | − | − | 71 |
| − | + | − | 65 |
| + | + | − | 80 |
| − | − | + | 64 |
| + | − | + | 74 |
| − | + | + | 66 |
| + | + | + | 85 |
- The effect of A is the average of the runs with A at + minus the average with A at −: (71 + 80 + 74 + 85) ÷ 4 − (62 + 65 + 64 + 66) ÷ 4 = 77.5 − 64.3 = 13.3 N. In the same way B = 6.3 N, C = 2.8 N and the interaction AB = 3.8 N.
- Factor A matters most; B helps; C and the AB interaction are small. Test them with replicates (the repeat of each run, here 2) to be sure the differences are above the scatter.
Test to failure and find the margin
- Test at least one prototype to failure: how and where it breaks shows where to strengthen it, and how much margin you have.
- Safety factor = failure load ÷ design load. If the design load is 50 N and the part fails at 140 N, the safety factor is 2.8.
- Allow for the scatter: a single failure load is one sample. Several parts show the spread; design to the low end.
- Look at the fracture: a brittle snap across layers, a ductile yield, a joint pulling out: each means a different fix.
- Do it safely: eye protection, guards and a safe distance; stored energy releases violently.
Accelerated and environmental tests
You cannot wait years to learn how a product ages. Accelerated tests raise a stress (temperature, load, humidity, vibration) to speed up the failure mechanism. For temperature-driven processes the Arrhenius model gives the acceleration factor.
AF = exp[(Ea ÷ k) × (1 ÷ Tuse − 1 ÷ Ttest)]
Worked example: a burn-in at 85 °C
- With Ea = 0.7 eV, a use temperature of 25 °C and a test at 85 °C: AF = exp[(0.7 ÷ 8.617×10−5) × (1 ÷ 298.15 − 1 ÷ 358.15)] = 96.
- 1 000 h at 85 °C then stand for 96 015 h at 25 °C, i.e. about 11.0 years. The result is only as good as the assumed activation energy and the relevance of the mechanism.
- Other tests: thermal cycling (cracks and fatigue), humidity, vibration and shock, drop, salt spray, and ingress protection for water and dust.
Failure modes and priorities
A failure mode and effects analysis (FMEA) lists what can go wrong, scores each by severity (S), occurrence (O) and detection (D; high means hard to detect), each from 1 to 10, and multiplies them into a risk priority number: RPN = S × O × D. Fix the largest first.
| Failure mode | S | O | D | RPN |
|---|---|---|---|---|
| Snap hook breaks | 8 | 5 | 4 | 160 |
| Screw boss strips | 5 | 6 | 3 | 90 |
| Battery connector reversed | 9 | 3 | 2 | 54 |
| Lid does not seal | 4 | 5 | 6 | 120 |
| Radio detuned by the case | 6 | 4 | 7 | 168 |
The highest RPN here is “Radio detuned by the case” (168): test or redesign it first. The scores are subjective: use them to focus the discussion, not as measurements. Note that a severe failure (S = 9) deserves action even if its RPN is lower.
Learn from every iteration
Worked example: a test week
- 6 tests a day for 5 days is 30 tests. Allow a day for each of: set-up and calibration, testing, analysis and writing, redesign, and building the next version.
- Write a one-page summary of each build: the question, the test, the result, the decision, the next step.
- Look for the cause, not the symptom: ask why a failure happened several times in a row until you reach something you can change.
- Change one thing at a time unless you are running a designed experiment.
- Keep the failed parts and label them: the fracture surface is data.
- Share results with the team and with suppliers: a second view finds what you missed.
- Stop when the requirement is met (see The Prototyping Process and Its Stages).
Troubleshooting
| Problem | Likely cause | Try |
|---|---|---|
| Results are inconsistent | Uncontrolled variables, poor measurement | Control conditions; repeat; improve the instrument; use more samples |
| Prototype passes the test but fails in use | The test did not represent the real use | Test in the real environment; watch real users |
| Cannot tell which change helped | Several changes at once | Revert; change one at a time or run a designed experiment |
| Users all say it is great | Leading questions, friendly users | Observe behavior; recruit neutral users; ask for the worst thing |
| Test takes too long | Real-time test of a slow process | Accelerate the stress; test the weakest part |
| Data lost or unclear | No record of conditions | Log conditions and raw data every time |
Related
- Prototyping Techniques: the overview of this guide.
- The Prototyping Process and Its Stages: what to test at each stage.
- From Prototype to Production: pilot-run testing.
- Data Logging: recording measurements.
FAQ
How many prototypes do I need to test?
Enough to see the spread: five for a first look at how it fails; 22 with no failures to show 90 % reliability at 90 % confidence. Statistical claims need more samples than prototypes usually allow.
How many users should test my prototype?
About five per round, in several rounds with fixes between them: five find about 84 % of the problems on the usual model.
Why test to failure?
It shows how the part breaks and how much margin the design has, which a pass-only test cannot.
What is a designed experiment?
A planned set of runs that varies several factors together so that their effects and interactions can be separated with fewer runs than one-at-a-time testing.
How do I know a difference is real?
Compare it with the scatter: if the difference is smaller than the confidence interval of the measurements, it may be noise. Repeat or use more samples.
References
- Montgomery, D. C., Design and Analysis of Experiments, Wiley; Meeker, W. Q. and Escobar, L. A., Statistical Methods for Reliability Data, Wiley.
- Nielsen, J. and Landauer, T. K., “A mathematical model of the finding of usability problems”, INTERCHI 1993; Nielsen, J., Usability Engineering.
- AIAG, FMEA Handbook; JEDEC JESD47 (stress-test-driven qualification); IEC 60529 (IP codes).
Last reviewed: October 2026. Statistical rules and model values are standard but depend on assumptions: use them as guides, and verify with real tests.