Measuring Tools and Accuracy
Rules, calipers, micrometers and indicators, resolution against accuracy, thermal expansion, reading technique and an error budget.
Resolution, accuracy and repeatability
Three words that are often confused. Resolution is the smallest step the instrument can show (the last digit). Accuracy is how close its reading is to the true value. Repeatability is how closely it gives the same reading again. A caliper that shows 0.01 mm has a resolution of 0.01 mm but an accuracy of about ±0.03 mm: the last digit means little.
| Instrument | Resolution | Accuracy (typical) | Use |
|---|---|---|---|
| Steel rule | 0.5 mm | ±0.3 to ±0.5 mm (reading) | Layout, rough work |
| Tape measure | 1 mm | ±1 to ±2 mm per meter | Carpentry, long lengths |
| Vernier or dial caliper | 0.02 or 0.01 mm | ±0.03 mm | General machining and checking |
| Digital caliper | 0.01 mm | ±0.02 to ±0.03 mm | The workshop workhorse |
| Micrometer | 0.01 mm (0.001 digital) | ±0.002 to ±0.005 mm | Precision diameters and thickness |
| Dial test indicator | 0.01 mm (0.002 fine) | Relative only | Setting work true on a mill or lathe |
| Gauge blocks | — | ±0.0002 to ±0.001 mm | Reference and calibration |
Worked example: which instrument for ±0.05 mm?
- A tolerance of ±0.05 mm has a band of 0.10 mm. The gauge-maker’s rule is that the instrument should be ten times better than the band: 0.10 ÷ 10 = 0.010 mm. A good minimum is four times: 0.025 mm.
- Calipers at ±0.03 mm accuracy give a ratio of 0.10 ÷ 0.03 = 3.3: too coarse to certify a ±0.05 mm part (you would accept bad parts and reject good ones). Use a micrometer (±0.005 mm, ratio 20).
- Tolerance grades and fits are in Fits and Tolerances.
The instruments
| Instrument | How it works | Good practice |
|---|---|---|
| Steel rule | Engraved graduations at 0.5 or 1 mm | Measure from the 1 cm mark, not the end (worn end); view square to the scale |
| Tape measure | Hook that slides by its own thickness | The hook is loose on purpose (1.5 mm: it makes inside and outside measurements agree); hold flat and straight |
| Calipers | Fixed and sliding jaws; vernier, dial or digital | Measure near the root of the jaws; light, constant pressure; zero closed |
| Micrometer | Screw with a 0.5 mm pitch and a thimble of 50 divisions | Use the ratchet; clean the faces; check the zero |
| Dial test indicator | Lever or plunger that magnifies a movement | Mount rigidly; the stylus must be square to the motion to avoid cosine error |
| Combination and engineer’s square | A blade at 90° to a stock | Check with the flip test below |
| Feeler and radius gauges | Strips and templates of known size | Use with a light touch; thin strips bend |
| Surface plate and gauge blocks | A flat reference and stacked blocks of exact size | Handle with care; wring the blocks; keep them oiled |
Worked example: reading a micrometer and a vernier
- Micrometer: the sleeve shows 7.5 mm (including the half-millimeter line) and the thimble line 28 is at the index: 7.5 + 28 × 0.01 = 7.78 mm.
- Vernier (0.02 mm): the zero lies just past 12 mm and the 17th vernier line aligns with a main-scale line: 12 + 17 × 0.02 = 12.34 mm.
- Tape measure: the hook’s play of 1.5 mm over a 3 m length is 0.05 %: fine for carpentry, not for engineering.
Technique: how to read well
- Look square on. Parallax makes a mark appear in the wrong place: with a gap of 1.0 mm between the rule and the work, viewing 10° off square gives an error of 1.0 × tan(10°) = 0.18 mm. Bring the scale in contact with the work.
- Mark with a scribe, not a pencil. A pencil line is about 0.5 mm wide, a scribed line 0.05 mm: ten times finer. Use a marking knife on wood and layout blue on metal.
- Jaws squarely on the work. A caliper tilted 1° with the part 30 mm from the scale gives an error of 30 × sin(1°) = 0.52 mm (the Abbe error: measure close to the scale).
- Light, constant pressure. The jaws are springy: heavy pressure closes them and gives too small a reading. A micrometer’s ratchet gives the same force every time.
- Clean the faces and the work. A speck of swarf is 0.01 mm or more.
- Measure a round part at several angles: it may be oval or tapered.
- Read first, then decide. Write the number down before you work out what to do about it.
An error budget
Every measurement is the sum of several independent errors. If they are independent, they add as the root of the sum of squares (RSS), not as a straight sum, because they are unlikely to all be at their worst together.
| Source | Size (95 %) | Square (×10⁻³ mm²) |
|---|---|---|
| Display resolution | ±0.005 mm | 0.025 |
| Instrument accuracy | ±0.030 mm | 0.900 |
| Jaw pressure and angle | ±0.020 mm | 0.400 |
| Part form (out-of-round) | ±0.015 mm | 0.225 |
| Temperature (3 K on 50 mm steel) | ±0.002 mm | 0.003 |
Worked example: adding the errors
- The straight sum of the five terms is ±0.072 mm. The root-sum-square is √(0.025 + 0.900 + 0.400 + 0.225 + 0.003)×10⁻³ = ±0.039 mm.
- The instrument accuracy (±0.030 mm) and the jaw pressure (±0.020 mm) dominate: using a micrometer instead cuts the instrument term to ±0.005 mm and the total to ±0.017 mm (with a ratchet for the pressure and the same part and temperature).
- Use the same tool and the same hands for comparing parts: many errors cancel (they are systematic), and what remains is the repeatability.
Temperature
Solids expand when warm. Steel grows about 11.7 µm per meter per kelvin (a coefficient of 11.7 × 10⁻⁶ K⁻¹), aluminum about 23.1. Measurements are defined at 20 °C.
Worked example: a warm part
- A 100 mm steel part 10 K warmer than 20 °C is longer by 11.7 × 10⁻⁶ × 100 × 10 = 11.7 µm; an aluminum part by 23.1 µm. A micrometer shows both.
- Holding a 25 mm part in the hand warms it by about 12 K: 3.5 µm in steel. Let parts and instruments reach the same temperature, and avoid handling the jaws.
- Measuring an aluminum part with a steel rule at a different temperature: both change, but by different amounts; for fine work measure at 20 °C.
- See Fits and Tolerances for the effect on fits.
Squareness, flatness and straightness
To check a square, draw a line along the blade against a straight edge, flip the square and draw it again. If the square is true, the lines coincide; if not, they diverge by twice the error.
Worked example: the flip test
- The lines are 0.20 mm apart at the end of a 150 mm blade. The error is half of that over the length: 0.20 ÷ (2 × 150) = 0.001 rad = 0.04°.
- Over a 300 mm part this is 0.20 mm out of square. A good engineer’s square is better than 0.01 mm over 150 mm.
- Flat: check with a straight edge and a light behind it: a gap that lets 0.02 mm of light through can be seen. Straight: sight along the edge, or use a taut string.
- Cosine error: a dial indicator whose stylus is tilted 5° from the line of motion reads 0.38 % too much (0.076 mm on a 20 mm movement).
Repeatability and averaging
Worked example: ten readings of a 25 mm gauge block
- The readings are 25.01, 25.03, 24.99, 25.02, 25.00, 25.03, 25.01, 24.98, 25.02, 25.01 mm. The mean is 25.010 mm and the sample standard deviation s = 0.016 mm.
- The standard error of the mean is s ÷ √10 = 0.005 mm; with the Student t factor 2.262 (95 %, 9 degrees of freedom) the mean is known to ±0.012 mm.
- The mean is 25.010 against the true 25.000 mm: the bias is +0.010 mm. Averaging reduces random scatter, not a bias: only calibration against a reference removes a bias.
- A reading that does not repeat means the technique or the part is not stable: find out why before you average.
Checking and calibrating
- Check calipers with the jaws closed (zero), and on a gauge block or a known standard at two sizes (say 10 mm and 50 mm), inside and outside.
- Check a micrometer with a gauge block or the setting rod supplied: 25 mm for a 25–50 mm micrometer.
- Check a square with the flip test; a rule against a known straight edge.
- Keep a reference: a gauge block or a ground bar of known size is a cheap way to catch a bad instrument.
- Record the checks and repeat once or twice a year, or after any knock. Send an instrument for calibration if it must be traceable.
- Store them in cases, oiled lightly, with the jaws open a little to avoid corrosion marks.
Troubleshooting
| Problem | Likely cause | Try |
|---|---|---|
| Readings do not repeat | Technique, dirt, temperature, worn instrument | Clean; use constant pressure; let parts settle; check against a gauge block |
| Caliper reads differently for inside and outside | Worn or bent jaws | Check the jaws against a gauge block; replace the caliper |
| Digital caliper jumps or resets | Flat battery, dirt on the scale | New battery; clean the scale with a dry cloth |
| Part fits one day and not the next | Temperature or humidity | Measure at the same temperature; allow time |
| Cuts are not square | Square or fence out, blade deflection | Flip-test the square; check the fence |
| Measurements differ between two people | Technique and pressure | Agree a method; use a ratchet or a go/no-go gauge |
Related
- Shop Tools: the overview of this guide.
- Hand Tools, Marking Out and Cutting: marking out, and the tools these measurements guide.
- Sharpening, Maintenance and Tool Care: keeping measuring tools in calibration.
- Fits and Tolerances and GD&T: what the measurement is for.
FAQ
Are digital calipers accurate enough?
For most workshop work yes, at about ±0.02 to ±0.03 mm; for ±0.05 mm tolerances or finer use a micrometer, and always check them against a gauge block.
What is the difference between accuracy and resolution?
Resolution is the smallest step the display shows; accuracy is how close the reading is to the true value. A display of 0.01 mm does not mean an accuracy of 0.01 mm.
Why does a tape measure hook move?
The hook slides by its own thickness so that inside and outside measurements both read correctly.
How often should I calibrate?
Check against a reference every few months, after a knock, and send for calibration yearly if you need traceability.
References
- ISO 3611 (micrometers), ISO 13385 (calipers) and ISO 3650 (gauge blocks).
- JCGM 100, Guide to the Expression of Uncertainty in Measurement (GUM).
- Oberg et al., Machinery’s Handbook (measuring instruments, thermal expansion).
- Mitutoyo, Quick Guide to Precision Measuring Instruments.
Last reviewed: October 2026. Accuracies and errors are typical: check your own instruments against a reference.