Numbers and Floating Point

Integers

Computers store whole numbers exactly, in a fixed number of bits, so they overflow. An n-bit unsigned integer holds 0 to 2ⁿ − 1; a signed integer (two’s complement, see Bits, Bytes and Data Formats) holds −2ⁿ⁻¹ to 2ⁿ⁻¹ − 1. Python integers grow without limit; C, Rust (in release builds), Java and most hardware wrap or are undefined.

Floating-point numbers

Most real numbers are stored in IEEE 754 floating-point format: a sign, an exponent and a mantissa, so that value = (−1)^sign × 1.mantissa × 2^exponent. A 32-bit float has 24 significant bits (about 7 decimal digits); a 64-bit double has 53 (about 16 digits).

	float (32-bit)	double (64-bit)
Significant bits	24 (23 stored)	53 (52 stored)
Decimal digits	about 7	about 15 to 16
Machine epsilon (gap above 1.0)	2−23 = 1.192e-7	2−52 = 2.220e-16
Largest finite value	3.403e+38	1.798e+308
Smallest normal value	1.175e-38	2.225e-308

Rounding error and what it does

Each floating-point operation rounds its result, so it has a relative error of at most ε ÷ 2 (about 1.1 × 10⁻¹⁶ for a double). One operation is harmless; the trouble is that errors accumulate through millions of operations, and that some operations amplify them. The rules of thumb:

• Use double by default on a PC; use float only when memory or speed demand it, and then check the accuracy.
• Errors usually grow like √n for random rounding and like n for a steady bias, so a long sum or a long simulation needs care.
• Algorithms matter more than precision: a stable formula in single precision beats a naive one in double.
• Test with known answers and with different precisions: if the result changes in the third digit when you switch from float to double, it is not trustworthy to more than three.
• Floating-point addition is not associative: (a + b) + c may differ from a + (b + c), so the order of a sum (and parallel or vectorized sums) changes the last digits.

Adding many numbers

def kahan_sum(values):
    total = 0.0
    c = 0.0                       # running compensation for lost low-order bits
    for v in values:
        y = v - c
        t = total + y
        c = (t - total) - y       # what was lost when adding y to total
        total = t
    return total

import math
math.fsum([0.1] * 10)             # 1.0 exactly; sum([0.1] * 10) gives 0.9999999999999999

Cancellation: subtracting nearly equal numbers

Subtracting two numbers that are almost equal cancels their leading digits and leaves the rounding error of the inputs as the leading digits of the answer: catastrophic cancellation. The cure is usually to rearrange the formula so that the subtraction does not happen.

Comparing numbers

Never test floating-point numbers for equality. Test whether they are close, using a relative tolerance for large numbers and an absolute one near zero.

#include <math.h>
#include <float.h>
int nearly_equal(double a, double b, double rel, double abs_tol) {
    double diff = fabs(a - b);
    return diff <= abs_tol || diff <= rel * fmax(fabs(a), fabs(b));
}
/* use rel about 1e-9 for doubles, 1e-5 for floats; abs_tol a few times the noise of the data */

• Compare with a tolerance that means something: the noise of the measurement, not a magic 1e-6.
• Use < and > freely; they are safe, it is == that is not.
• Do not use a float as a loop counter: for (x = 0; x != 1.0; x += 0.1) never ends. Count with an integer and compute x from it.
• Test exact values only when they are exactly representable (powers of two, small integers).

Special values: NaN, infinity and negative zero

Value	How it arises	Behavior
+∞ / −∞	1.0 ÷ 0.0 (in IEEE arithmetic), overflow past the largest value	∞ + 1 = ∞; 1 ÷ ∞ = 0; ∞ − ∞ is NaN. (Python raises ZeroDivisionError for 1.0 / 0.0 instead.)
NaN (not a number)	0.0 ÷ 0.0, √(−1), ∞ − ∞, a failed conversion	Propagates through every calculation; NaN ≠ NaN, so test with isnan() (math.isnan, np.isnan)
−0.0	Negative results that underflow to zero	Equals +0.0 in comparisons but can change the sign of 1 ÷ x and atan2
Subnormal numbers	Values below the smallest normal number	Gradual underflow: precision falls; on some CPUs they are very slow

• Check inputs and intermediate results for NaN and infinity in data pipelines: one NaN in a sum makes the whole sum NaN, which is often how you find a bad sensor value.
• Missing data is not zero: represent it as NaN (or a masked value), not 0 or −999.
• On microcontrollers the behavior of division by zero, NaN and overflow depends on the chip and the compiler options: avoid relying on it.

Rounding and significant figures

• Round only for display, not in the middle of a calculation: rounding early adds errors that accumulate.
• Rounding rules: Python’s round() and the IEEE default round half to even, so round(2.5) is 2 and round(3.5) is 4 (and round-half-up would give 3 for 2.5). This removes a bias when many numbers are rounded; spreadsheets and some languages round half up, so results can differ in the last digit.
• Significant figures: a result cannot be more precise than its least precise input. A length of 12.3 mm (three significant figures) times a width of 4.56 mm gives 56.088 mm², which you report as 56.1 mm², not 56.088.
• Display with formats, not by converting to text yourself: f"{x:.3g}" (three significant figures), f"{x:.2f}" (two decimals), "%.3g" in C.
• Money and exact decimals: use integers (cents: 10 + 20 = 30 exactly) or a decimal type (Python decimal), never binary floats.

Units and uncertainty

A number in a program has no unit, so keep track of it yourself: put it in the name (length_mm), convert at the edges of the program, and use one system inside. Mixing units has destroyed spacecraft (NASA’s Mars Climate Orbiter, 1999, was lost because one program gave thrust in pound-force seconds and another expected newton seconds). A units library such as pint (Python) catches the mistakes automatically.

Choosing a number type

For…	Use	Why
Counts, indexes, states, IDs	An integer type of the right width (uint32_t, size_t)	Exact; faster than floats; no rounding
Bytes and bit fields	uint8_t, uint16_t	Exact width and no sign surprises (see Bits, Bytes and Data Formats)
Time stamps and long counters	64-bit integer (microseconds or nanoseconds)	No precision loss, no early wrap-around
Measurements and engineering math on a PC	double	About 16 digits: more than any sensor
Large arrays, graphics, machine learning, microcontrollers with an FPU	float (or 16-bit float formats)	Half the memory and bandwidth; check the accuracy
Microcontrollers without an FPU	Integers and fixed-point	Float math is slow and large (see Embedded Systems)
Money, exact decimals	Integer cents or a decimal type	Binary floats cannot represent 0.1
Exact big numbers	Python int, big-integer libraries	No overflow

Related

• Programming Languages: the overview of this guide.
• Bits, Bytes and Data Formats: integers as bit patterns.
• Numerical Methods: how these errors affect equation solving, integration and simulation.
• Embedded Systems: Memory and Number Formats: fixed-point math on a microcontroller.
• Data Logging: precision, rounding and time stamps of measured data.

FAQ

Why is 0.1 + 0.2 not 0.3?

Neither 0.1 nor 0.2 can be stored exactly in binary; each is rounded to the nearest representable value, and their sum rounds again to 0.30000000000000004. Compare with a tolerance, or use integers (cents) or a decimal type for exact decimals.

Should I use float or double?

double unless memory or speed force you otherwise. The extra precision is cheap on a PC, and a float’s seven digits are quickly used up by sums and small differences.

What is NaN and why does it spread?

Not-a-number is the result of an undefined operation (0 ÷ 0, √−1). Any calculation that uses it also gives NaN, and it compares unequal to everything, even itself, so use isnan(). One bad sensor value can turn a whole average into NaN.

How many digits should I show?

No more than the data support: the number of significant digits of the least precise input, or one more for intermediate values. Compute the uncertainty, and round the result to match it.

References

• IEEE 754-2019, Standard for Floating-Point Arithmetic.
• Goldberg, D., “What every computer scientist should know about floating-point arithmetic”, ACM Computing Surveys, 1991.
• Kahan, W., “Pracniques: further remarks on reducing truncation errors”, Communications of the ACM, 1965 (compensated summation); Higham, N., Accuracy and Stability of Numerical Algorithms.

Last reviewed: October 2026. Digits shown come from IEEE doubles computed in the page builder: they are reproducible on any IEEE 754 machine.