save tables of numbers to compact file, any alternative to json?

keszegh

so i need to store rather big data sets and saving to json generates huge files, i need to store numbers of type 1166.53, or multiplying by 100, integers with at most 6 digits. for now in a json output they take up in average 7 letters, like:"1166.53," whereas they are int-s or doubleint-s at most, so they could take up 1 or at most 2 letter space, is there any supported format that saves data to some more efficient format?

ar2rsawseen

This sounds like a Google interview question about phone numbers
http://stackoverflow.com/a/5464662/2274511

Unfortunately better storing efficiency increases access time

In your case, if there are only numbers, you can probably simply save them in a comma separated list in a file, without any additional markup.
and then split it back to table of numbers
Splitting/Joining strings:
http://lua-users.org/wiki/SplitJoin

keszegh

thanks, but a json is essentially a comma separated list, as my table structure is not too involved, the number of '['-s etc is small compared to the numbers. so this way it's easier to stick with json.

i also searched for possible solutions and i've found something called luabins, would that be a good solution, what do you think?

https://github.com/agladysh/luabins/tree/master

and a usage (and combo with base64):
http://www.sambatista.com/Lucky5Games/?p=102

i wonder how much space could i save with using luabins instead of json.
of course we need some bindings, which i still never did, and so i contemplate much longer before trying to make this work with gideros.

also, while access time is important, sometimes i load such a big file, that i get 'out of memory' errors from lua, so i think space is more of an issue (or at the end i import the same table so i will get the error anyway?).

also smaller file may mean faster access time as hdd and memory are slower than cpu?