It looks like you're new here. If you want to get involved, click one of these buttons!
local value = 1 for i=1, 200 do value = value + 0.1 print("VALUE"..i, value) end |
VALUE1 1.1 VALUE2 1.2000000000000002 VALUE3 1.3000000000000003 VALUE4 1.4000000000000004 VALUE5 1.5000000000000004 VALUE6 1.6000000000000005 VALUE7 1.7000000000000006 VALUE8 1.8000000000000007 VALUE9 1.9000000000000008 VALUE10 2.000000000000001 VALUE11 2.100000000000001 VALUE12 2.200000000000001 VALUE13 2.300000000000001 VALUE14 2.4000000000000012 VALUE15 2.5000000000000013 VALUE16 2.6000000000000014 VALUE17 2.7000000000000015 VALUE18 2.8000000000000016 VALUE19 2.9000000000000017 VALUE20 3.0000000000000018 VALUE21 3.100000000000002 VALUE22 3.200000000000002 VALUE23 3.300000000000002 VALUE24 3.400000000000002 VALUE25 3.500000000000002 VALUE26 3.6000000000000023 VALUE27 3.7000000000000024 VALUE28 3.8000000000000025 VALUE29 3.9000000000000026 VALUE30 4.000000000000003 VALUE31 4.100000000000002 VALUE32 4.200000000000002 VALUE33 4.300000000000002 VALUE34 4.400000000000001 VALUE35 4.500000000000001 VALUE36 4.6000000000000005 VALUE37 4.7 VALUE38 4.8 .... and so on |
Likes: MoKaLux
Comments
https://discuss.ardupilot.org/t/overcoming-lua-floating-point-math-precision-errors-latitude-longitude/75710
I don't know if this happens only in the print statement or also on the math side (could you try some equality checks, eg. if 0.4 == 0.3+0.1)
Also: https://wiki.gideros.rocks/index.php/Int64
Likes: pie
edit - here it is! function round2, and equality check passed