[mdlug] More OOCalc math wonkiness

[Ken Siersma]

Doesn't happen in OpenOffice 2.2.0, Ubuntu 7.04.

On Nov 27, 2007 3:44 PM, Drew <drew4096 at gmail.com> wrote:

>     Using OO Calc on another project, I ended up getting a zero
> result rather abruptly
> when scaling down a Slip value. Increasing the precision of this
> parameter, so that the
> display read .001 instead of 0, changed the results. So I did a
> little test in a fresh
> spreadsheet:
>
> Start at any cell. Enter a column of numbers as follows:
>
> 0.001
> 0.002
> 0.003
> 0.004
>
> Do not change the precision of the display. The numbers just entered
> read all zeros, as the default is
> two decimal places. (not significant figures!)
>
> Now in another cell, add up all the numbers (=SUM(A1:A4)). They
> should add up to 0.01, which is within
> the (default) precision of the cell format. However, the result displayed
> is 0.
>
> Now change the precision of the cells with the list of numbers to at
> least three decimal places. The cell
> with the addition formula then shows the correct value of 0.01.
>
> Evidently, Calc is referring to cells to be used in formulas, not by
> any actual number stored separately,
> as is expected from calculators, but rather by *whatever is
> displayed* by virtue of the Font format
> setting. At least now I have a better handle on how to write the bug
> report. However, I'm not sure that this
> isn't supposed to be a "feature"; and even so, fixing it is likely
> going to require completely rewriting
> large sections of code, if the developers did what I think they did.
> Still, it's almost certainly responsible
> for the off-the-wall results that I've been getting with the least
> square analysis.
>
> BTW: I also tried this in Microsoft Excel. Excel gave the correct
> answer to within the precision setting
> *of the formula cell* regardless of the precision settings of the
> cells being added. So Excel apparently
> is at least getting this right.
>
> ----
>
> - Drew.
>
> _______________________________________________
> mdlug mailing list
> mdlug at mdlug.org
> http://mdlug.org/mailman/listinfo/mdlug
>