Chapter 4
Numerical Errors and Uncertainty in Computation
These notes explain three short experiments on floating-point representation and precision, cancellation, and order of summation. Source files in MATLAB, Python, C++, and Fortran and instructions for running them are available on the Chapter 4 codes page. Inspect the program output and change the numbers to explore the behavior.
1. Representation and precision
Compare 0.1 + 0.2 with 0.3 and examine their
difference. In ordinary double precision the difference is about
5.55 × 10−17. A short decimal display may hide it.
The decimal fractions are rounded when stored in binary, and their
sum is rounded again.
The programs also print the gap between 1 and the next larger
representable number: approximately 1.19 × 10−7
in single precision and 2.22 × 10−16 in double
precision. This is the convention for epsilon used here. Finally,
compare 1 + 1e-8 in both precisions: the correction is
lost in single precision and retained, with rounding, in double
precision. Change the correction to see where that behavior changes.
2. Cancellation and a better formula
For positive x, compare the mathematically equivalent expressions
sqrt(1 + x) - 1 and x / (sqrt(1 + x) + 1).
As x decreases, the direct expression loses relative accuracy because it subtracts nearly equal quantities. At x = 10−16 it normally returns zero in double precision, while the second expression gives approximately 5 × 10−17. The second formula avoids this cancellation and serves as a numerical reference in the program output; it is still evaluated with finite precision.
Extend the range of x or repeat the calculation in single
precision. In Python, use np.float32 for all operands
and np.sqrt for the square root to keep this calculation
in single precision. Observe where the two results begin to disagree.
3. Order of summation
Set a = 1016, b = 1, and
c = −1016. The exact sum is 1. In ordinary
double precision, (a + b) + c gives 0, while
(a + c) + b gives 1. In the first grouping the unit
contribution is lost before the large terms cancel.
Change the magnitude of a, keeping c =
−a, and find when the groupings begin to differ.
This example shows that summation order matters; no single ordering
guarantees accuracy for every mixture of positive and negative terms.
For comparison in Python, try
math.fsum([a, b, c]),
a function designed for accurate floating-point summation. It gives
1 for this example.
Different errors need different checks
- Roundoff: compare precision, summation order, or an algebraically improved formula.
- Discretization or truncation error: refine the grid or step, increase the approximation order, or retain more series terms and check convergence.
- Implementation error: test the code and check against a known solution. See the Chapter 3 resources for recording verification checks.
- Input uncertainty: vary inputs over their plausible uncertainty ranges and examine the changes in the result.
- Modeling error: test assumptions and compare with physical evidence. More arithmetic precision does not correct an inadequate model.
Documentation and further reading
- Python: Floating-Point Arithmetic — Issues and Limitations. An introduction to binary fractions, rounding, and printed output.
- NumPy: finfo. Precision and range information for floating-point types.
- MATLAB: eps. Floating-point spacing and relative precision.
- David Goldberg: What Every Computer Scientist Should Know About Floating-Point Arithmetic. A more detailed treatment of rounding, cancellation, and floating-point arithmetic.