fpchecker-doc

Floating-point Exceptions and GPU Applications

Mon, 19 Jul 2021 00:00:00 +0000

A floating-point exception occurs when an attempted arithmetic operation produces an abnormal result. For example, an exception arises when a program attempts to compute the square root of a negative number or when a division by zero occurs.

In traditional CPU-based programming languages and system software, several methods exist to detect floating-point exceptions. For example, in Linux systems, when exceptions occur, one of two things can happen. By default, the exception is simply noted in the floating-point status word, and the program can check the status word to find out which exceptions happened. Alternatively, traps for exceptions can be enabled, enabling the program to receive the SIGFPE signal (see this). The default action for this signal is to terminate the program, but the effect of the signal can be changed.

GPUs are quickly dominating the HPC market and are becoming pervasive in the largest supercomputers. However, the support to detect floating-point exceptions in GPUs is sometimes limited and, in some cases, null. For example, NVIDIA GPUs have no mechanism to detect that a floating-point exception occurred according to the CUDA Programming Guide [1] version 11.

The CUDA Programming Guide also states other deviations of the CUDA implementation of floating-point arithmetic concerning the IEEE-754 standard, including that double-precision floating-point absolute value and negation are not compliant with IEEE-754 with respect to NaNs.

Detecting Exceptions in CUDA

Since the traditional mechanisms to detect floating-point exceptions don’t exist in CUDA, programmers are left with almost no option other than using printf statements in the application source code to catch the result of exceptions. Printing out the results of computations makes sense because the result of exceptions—infinity and NaN—propagate quickly to other calculations. For example, 2 x ∞ = ∞. So there is a good chance of observing such results in the application’s output.

Photo by Nana Dua on Unspalsh.com

A significant downside is that printing in standard output from millions of GPU threads can be slow and error-prone. For example, some exceptions can be miss-detected, and they can affect control-flow without programmers even noticing them.

FPChecker Detection

FPChecker can detect the result of exceptions in CUDA (and in the CPU code). For example, the tool detects operations that produce NaN, positive infinity, negative infinity, and subnormal numbers (underflows). Furthermore, it informs programmers of the location of the exception (file and line of code) and the first CUDA thread that encounters the exception. This information can be used in combination with a debugger to understand the origin of the error.

Mitigating floating-point exception often involves a deep understanding of the application’s algorithm, the modification of inputs, and adding if/then conditions in the code to deal with specific quantities—for example, if the denominator of a division is zero, some different action may be needed.

References

[1] CUDA C++ Programming Guide: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html

What can be Done About Floating-Point Cancellation

Mon, 12 Jul 2021 00:00:00 +0000

Scientific visualization of a very large simulation of a Rayleigh–Taylor instability problem.

One of the difficulties of dealing with floating-point arithmetic arises when subtracting numbers. When we subtract nearby quantities, the most significant digits in the operands match and cancel each other. This is called cancellation.

Example of Cancellation

Let’s look at an example. Suppose we have two quantities, a and b:

where x, m, and n are decimal digits. Suppose that due to the available precision in the system, we can only store the 12 most significant digits. As a result, when a and b are stored, the mmm… and nnn… digits are lost.

If we compute c = a – b, the result has only one significant digit:

The values for the uuu… digits are not necessarily zero. Therefore, the relative error of the result can be significant. The above is an example of catastrophic cancellation, which occurs because the original quantities a and b were not represented exactly with the available precision of the system. On the other hand, suppose the a and b quantities are exactly represented as floating-point numbers in the available precision. In that case, cancellation can occur, but it is considered benign (i.e., the result of the subtraction is exact).

What Can be Done?

Avoiding cancellation can be a challenge depending on the problem being solved by the application. In some cases, rearranging the mathematical formulas can remove the problem. The paper What Every Computer Scientist Should Know About Floating-Point Arithmetic [1] gives an example of how rearranging formulas helps.

Sometimes algorithmic changes or input changes are required to avoid cancellation (see [2]). In many applications, however, it’s almost impossible to remove cancellation, and programmers must embrace it and develop a solid testing infrastructure to deal with the numerical inconsistencies and errors it may yield.

Cancellation Detection in FPChecker

Cancellation is almost inevitable in scientific applications, and it occurs very frequently. For example, many numerical methods in HPC rely on calculating differences of quantities, which often suffer from cancellation.

Because of its prevalence in scientific codes, FPChecker detects cancellation only when many digits are lost in a subtraction instruction. Therefore, FPChecker only reports cancellation when at least 10 decimal digits are lost; however, this threshold can be adjusted.

References

[1] Goldberg, David. “What every computer scientist should know about floating-point arithmetic.” ACM computing surveys (CSUR) 23, no. 1 (1991): 5-48.

[2] Lee, Sang-Hyeon. “Alleviation of cancellation problem of preconditioned Navier–Stokes equations.” Journal of Computational Physics 228, no. 14 (2009): 4970-4975.

Subnormal Numbers and Compiler Optimizations: A Dangerous Combination

Mon, 05 Jul 2021 00:00:00 +0000

Subnormal numbers (previously known as denormal numbers) fill the underflow gap in floating-point arithmetic. Traditionally, an underflow computation is said to occur when the exact result of the calculation is nonzero but is smaller than the smallest normalized floating-point number. Subnormal numbers represent the result of computations that produce very small numbers but are not too small to become zero.

While subnormal numbers are generated frequently in numerical software, combining such numbers with specific compiler optimizations can produce unexpected results.

Simple Example

Let’s examine an simple example where combining subnormal numbers and compiler optimizations produce inconsistencies.

Consider the following function that executes a division operation and makes a decision based on the result of the division:

void compute(double x, double y) {
  double z = x/y;
  if (z < 1.0)
    printf("Correct branch\n"); 
  else
    printf("Incorrect branch!!\n");
}

Compiling the code on an x86_64 system with the clang compiler and optimization level -O3 (with and without the –ffast-math option), produces the expected result. If we use 1.1e-311 and 2.0e-309 for the input of x and y, the division produces 0.005499999999998974, so the code prints:

$ ./test
Correct branch

Note that the inputs x and y are subnormal floating-point numbers, but this doesn’t change the function’s behavior.

We now compile the code on a different system. We use an IBM POWER9 system and compile it with the IBM xlc compiler using -O3. In this compiler, -O3 automatically enables the non-strict IEEE mode, similar to the optimization mode of –ffast-math in the clang and gcc compilers. This time, using the same input, we get a different result:

$ ./test
Incorrect branch!!

We now lower down the optimization level in xlc to –O2, and this time we get the expected result:

$ ./test
Correct branch

What happened? How can we have a different behavior by simply changing the optimization level from –O3 to –O2? To understand what happened, we can look at the assembly code (we use the Linux objdump tool to disassemble the code). Here’s a portion of the assembly code for the –O2 case:

0000000000000000 <compute>:
  0: 00 00 4c 3c   addis   r2,r12,0
  4: 00 00 42 38   addi    r2,r2,0
  8: 91 ff 21 f8   stdu    r1,-112(r1)
  c: a6 02 08 7c   mflr    r0
 10: 80 00 01 f8   std     r0,128(r1)
 14: c0 11 01 f0   xsdivdp vs0,vs1,vs2
 18: 00 00 82 3c   addis   r4,r2,0
 1c: 00 00 84 e8   ld      r4,0(r4)
 20: 8c 03 01 10   vspltisw v0,1
 24: 78 23 83 7c   mr      r3,r4
 28: e2 03 20 f0   xvcvsxwdp vs1,vs32
 2c: 58 01 01 f0   xscmpodp cr0,vs1,vs0
 30: 20 00 81 41   bgt     50 <compute+0x50>
...

The interesting thing to note in the above assembly is that the division operation is in line 13; the xsdivdp instruction corresponds to a division in the POWER architecture. Let’s see now the assembly code for the -O3 case:

0000000000000000 <compute>:
  0: 00 00 4c 3c   addis   r2,r12,0
  4: 00 00 42 38   addi    r2,r2,0
  8: 91 ff 21 f8   stdu    r1,-112(r1)
  c: a6 02 08 7c   mflr    r0
 10: 80 00 01 f8   std     r0,128(r1)
 14: 68 11 80 f0   xsredp  vs4,vs2
 18: 00 00 62 3c   addis   r3,r2,0
 1c: 00 00 63 e8   ld      r3,0(r3)
 20: 8c 03 01 10   vspltisw v0,1
 24: 78 1b 64 7c   mr      r4,r3
 28: e2 03 00 f0   xvcvsxwdp vs0,vs32
 2c: 38 01 62 fc   fmsub   f3,f2,f4,f0
 30: 88 1d 84 f0   xsnmsubadp vs4,vs4,vs3
 34: 80 21 61 f0   xsmuldp vs3,vs1,vs4
 38: 88 19 22 f0   xsmsubadp vs1,vs2,vs3
 3c: 88 0d 64 f0   xsnmsubadp vs3,vs4,vs1
 40: 18 19 00 f0   xscmpudp cr0,vs0,vs3
 44: 1c 00 81 40   ble     60 <compute+0x60>
...

The division operation is gone with the –O3 optimization level, and it was replaced with a reciprocal operation (xsredp instruction) followed by other operations, including multiplication.

We see that with the higher optimization level (-O3), the compiler optimizes the division operation and replaces it with potentially faster operations, which is wise and can potentially improve performance. However, this optimization makes the assumption that the inputs are represented as normal floating-point numbers.

We add a printf("z = %.17g\n", z); statement to check the result of z with the –O2 and –O3 cases:

With –O2:

$ ./test 
z = 0.0054999999999989736
Correct branch

With –O3:

$ ./test
z = nan
Incorrect branch!!

The optimized code, which includes a reciprocal operation, produces a NaN; the inputs fall outside of the normal number representations (i.e., they are subnormal numbers), combined with the equivalent code produced by the optimizations results in the code taking an incorrect branch.

Conclusion

If you are using highly optimized floating-point code, it is good to know whether your application is operating on subnormal numbers. Several numerical inconsistencies can be caused by the combination of optimized code and subnormal inputs. Knowing the functions and lines of code that use subnormal numbers is an excellent first step to understand potential numerical inconsistency issues. FPChecker can perform a complete floating-point analysis of applications and report code locations that produce subnormal numbers.