Please enable JavaScript to view this site.

A-Shell Development History

Navigation: » No topics above this level «

INT() and FIX() Fudge Factor

Scroll Prev Top Next More

When FPROUND is not set, these functions now apply a fudge factor of .0000005 (one tenth of the standard FPROUND factor). This minimizes the problem illustrated whereby an expression like INT(140.4*100) might result in 14039 rather than 14040. The discrepancy is caused by the inherent imprecision of binary floating point representation of decimal fractions (e.g. 14039.999999975 instead of 14040.0).

There are various other ways to deal with this issue, including setting the FPROUND option, but the issue continues to catch programmers by surprise; this automatic adjustment should minimize that problem. But note that it only works for numbers up to about 8 significant digits, after which the tiny fudge factor becomes too small to matter. If you regularly work with larger numbers and aren't explicitly dealing with rounding within the application, you may want to set your rounding factor somewhat higher. See "Rounding of Floating Point Variables" in the A-Shell reference for more details.