General

Re: sum function 9-10 decimal places?

ebonymb — Wed, 01 Aug 2012 19:12:28 GMT

i had this exact same problem. I used your suggestion and it resolved my issue temporarily only to occur again today. My thought is I might be able to use the round(sum()*100) div 100 formula in a hidden field and display the result as text in a visible field. Can that be done?

Re: sum function 9-10 decimal places?

charris1980 — Thu, 02 Apr 2009 16:23:34 GMT

cool, i'll give this a try and keep an eye on it for a few days. thanks for the quick reply and explanation on the calculations!

Re: sum function 9-10 decimal places?

Jimmy — Thu, 02 Apr 2009 15:31:02 GMT

Floating point arithmetic in XPath is computed according to the IEEE 754 standard, which allocates a fixed number of bits for the computation. I don't know a whole lot about floating point arithmetic, but I do know that it is not perfectly precise, and as you have experienced, tiny fractional parts can be lost or gained along the way.

I think the following formula might do the trick to get the result to have just one or two decimal places all the time:

round(sum(contents of sum) * 100) div 100

Please give it a try.

sum function 9-10 decimal places?

charris1980 — Thu, 02 Apr 2009 14:45:08 GMT

i have a form that sum(amts) on a text box. i tell it to force 2 decimal places and it does for maybe 90% of the forms submitted, but the other 10% it does 9-10 decimal places.

for example it might add:

$152.42 + $32.01. that equals 184.43. and when filling out the form, that text box shows that. but when i submit the form it changes it to "184.42999999999998"

I can't understand why though.