`-ffast-math`

shouldn’t be used, but that you shouldn’t use it in this situation where (it seemed) you were trying to match Matlab’s results in a line-by-line translation. The compiler would undo the care taken in the translation. It would also be inappropriate when you’ve manually planned out your floating point operations to keep alike magnitudes together, since the compiler would undo that planning as well (unaware of the expected magnitudes of the operands).
When I wrote the raytracer with my intern, we were using `-ffast-math`

from day one since I knew it wouldn’t matter. And, as you mentioned, it’s not even necessarily less precise, especially if the compiler employs fused multiply-add. As an exercise, we compared the output images between `-ffast-math`

enabled and disabled. They were subtly different — it was interesting to compute the difference in Gimp and stretch it out to be visible — but the difference was completely imperceptible to us.

When adding multithreading, I had him disable `-ffast-math`

just for testing. The goal was to have bit-for-bit identical output before and after threading was added, and `-ffast-math`

would make the results non-deterministic (different optimization decisions across different builds). Individual rays were computed entirely independently of one another, so if there was a difference under multithreading then there was a mistake in the code (race, etc.).

Another place it would be appropriate is the physics engine for a *single player* game. The difference would probably be imperceptible to players. However, it may be inappropriate for multiplayer physics engines that need to synchronize their results over a network. Two different builds could diverge enough that it annoys players (rubber banding, misses that look like hits, etc.). That’s just something to measure and test.

In the past I’ve figured it’s good policy to leave it off for scientific computing. But the only way I can justify this is when either you’ve manually planned out your floating point operations, or you require deterministic output across builds. As you’ve shown, the latter doesn’t really work so well if you call transcendental functions (unless you implement them yourself). So your article, plus some recent thought, has changed my mind on this.

Unfortunately, things are weirder than that. As noted near the end of the post […]

Oops! Sorry, I missed that sentence. That’s another thing to add to the list of mysterious Matlab quirks.

Furthermore, MSVC++ on Win64 vs. Win32 are also different!

The Win32 version is probably using the old x87 ISA, working internally in 80-bit precision and using the fsin, fcos, etc. instructions. The Win64 version would only use the SSE2 (and later) ISA. So for floating point operations that’s basically running the code on two different CPU architectures.

]]>It isn’t clear to me why “you definitely shouldn’t [use /fp:fast]”? Dawson makes this point as well, which I agree with: for an arguably large majority of use cases (including much of the analysis of the sort motivating this post), the desirable property is not necessarily getting *exactly the same* answer [as the MATLAB code] down to the ulp, but getting a *sufficiently accurate* answer *more quickly*. Granted, the “sufficiently accurate” is in the eye of the beholder and specific to each use case, but we can do the numerical analysis to figure out what is acceptable for the task at hand.

“That would be the smart thing to do so that Matlab’s results would be uniform across different platforms.” Unfortunately, things are weirder than that. As noted near the end of the post, even the same MATLAB code, on two different machines with slightly different (ages of) architecture, both running Win64, can yield different results, suggesting that (1) MATLAB has some architecture-dependent switching, and thus (2) reproducing this analysis on another machine might yield interestingly different results.

Furthermore, MSVC++ on Win64 vs. Win32 are also different! And the different Win32 results appear to agree with the different MATLAB (Win64) results… although I have only spot-checked a small number of examples of this.

]]>My initial thought was that Matlab may be less strict than C++ about associativity. This is where I think you’re mistaken in your article. Exactly because of the associativity problem you mentioned, a C++ compiler is *not* allowed choose an arbitrary evaluation order of `a + b + c`

for floating point operands. The plus operator is left associative, so this expression is strictly equivalent to `(a + b) + c`

and the compiler cannot (normally) pretend otherwise. The compiler is broken otherwise. In the C99 specification there’s a specific example about this (5.1.2.3-14), and I don’t expect C++ changed this.

Side note: This demonstrates one major reason why I prefer C to C++. I wanted to verify this for myself from the standard. I don’t like having to trust some StackOverflow answer (for example). The C standard is relatively short (552 pages for C99), and I can generally find what I need to know. The C++11 standard is 1324 pages and monstrously complicated — and getting worse each time they make a new standard, such as with C++17 (~1647 pages). When you’re running into a tricky problem, you need to understand all the parts, which includes referencing the language specification. If anything, I’d give up on a number of C99’s features just to cut down on the size of that document, but the damage has already been done.

The infamous `-ffast-math`

switch for GCC and Clang relaxes the language standard and lets the compilers perform some transformations that are otherwise forbidden (particularly `-fassociative-math`

). Presumably you’re not using this flag since you definitely shouldn’t.

I didn’t know there was so much variation in the results of the transcendental functions. Until I read your article, I thought these were stricter. Now I’m currently convinced that’s the original source of the problem, exacerbated by the butterfly effect. I’d bet one of the few ways Matlab would play it loose with the standardized floating point operations is under vectorization, which is basically the only way to get decent performance from Matlab.

MinGW-w64 (unfortunately, IMHO) links against msvcrt.dll and probably directly uses most its trigonometric functions, so it’s not surprising it has the same results as MSVC++. This is essentially the same thing I discovered with qsort().

I wonder if MathWorks uses their own implementations of the transcendental functions rather than relying on their C++ compiler/runtime (or whatever they’re using). That would be the smart thing to do so that Matlab’s results would be uniform across different platforms. If this is true, then you were essentially testing Microsoft’s implementation against Matlab’s. If you ran the tests on Linux, where the C++ version of these functions would be ultimately supplied by glibc, I wonder how it would compare.

]]>On a two’s complement architecture, case 3 is always evaluated. The -1 is represented as all bits set (111..111), so the expression evaluates to 3.

On a one’s complement architecture, case 2 is always evaluated. The -1 is represented as all bits high except the lowest bit (111…110), so the expression evaluates to 2.

On a signed magnitude architecture, case 1 is always evaluated. The -1 is represented as the lowest and highest bits set (100…001), so the expression evaluates to 1.

Since you mentioned preprocessor directives, was this code used to detect the host’s signed integer representation at compile time? Digging a bit, I see that DTED uses signed magnitude to encode integers (crazy!), so I’m guessing each branch had architecture-specific code for decoding DTED integers.

]]>Yes, in the sense that the corresponding C++ expression (p[1] == p[2] == p[3] == … == p[n]), where the p[k] are of type bool, has the same problem as Rosen’s notation; this “looks” like it might evaluate to true iff all of the p[k] are equal and false otherwise, but it doesn’t.

The *intended* meaning of this expression is essentially how Python treats chained comparisons, with an implicit AND stuck in between each consecutive binary comparison.

]]>