Loading large files on Android?

bigtunacan

I'm looking at loading files with the datasaver module and some of my files are fairly large. My concern is that Android will shut down an application if it locks the main thread for too long and I'm dealing with some files that are fairly large.

I'm wondering how others are dealing with this?

talis

Even there will be no error, it is always a good approach to divide the big files into small ones, if there will be no index or kind. Actually to hold index on big files can also have proplem hence the file will be a asset and in any case will be loaded into memory totally when you try to read or write. ( I am not %100 sure how the read/write file functions behave about memory management but guessing that they will load the entire file into memory )
If you will divide somehow you can init, search and get to the information much more faster.
I don't know what kind of file is it but let's suppose that it is a dictionary file just to give an example:
-You can divide the dictionary file into smaller files and give a prefix like
af_xxx.txt
go_xxx.txt
jp_xxx.txt
and so on according to word list in each files...

So in this case you can make a very simple index like the word "Gideros" will in the file which is starting with "go" prefix.

Of course as i wrote have no idea what is the purpose of your big file, but if you will share more information maybe some other ideas can be invented

bigtunacan

I won't have control over all of the files that are loaded. I create file sets that contain quiz type sets of questions that users the practice and compete with (this is an educational learning game).

Some of these sets of questions are controlled by me, but my users will also be able to create sets of questions/answers which can be used to compete against one another.

I could put some arbitrary limit and not allow files over a certain size (which I may end up doing for memory purposes), but in terms of locking the main application thread I think that will be hit before memory usage especially on lower end Android devices.

What I would like to do is spin up a background thread that does the file loading and present the user with a loading spinner or such.

If I were creating a native Android application I would do this with an IntentService

http://developer.android.com/training/run-background-service/create-service.html

I'm wondering if there is a way to do something similar with Gideros where I can spin up and control a background thread that can provide me with some type of callback mechanism once it has completed.

antix

@bigtunacan, I haven't tackled anything like this yet but if you have a giant text file with question and answer pairs you could get your program to read one line of the file each frame (or on some other timed event) and the program can run as normal but just not access the data until it has all loaded. In theory it could be setup with a callback and everything. After thinking for a while I made an example (attached)..

First the class that does the actual loading..

FileLoader = Core.class(Timer)   function FileLoader:init(delay, repeatCount) -- STARTUP self:addEventListener(Event.TIMER, self.onTimer, self) self.callBack = nil self.file = nil self.output = {} self.began = 0 self.elapsed = 0 end   function FileLoader:dispose() -- SHUTDOWN self:stop() self:removeEventListener(Event.TIMER, self.onTimer, self) self.callBack = nil if self.file then self.file.close() end self.file = nil self.output = nil self.began = nil self.elapsed = nil end   function FileLoader:beginLoad(fileName, callBack) -- BEGIN LOADING A NEW FILE self:stop() self:reset() if self.file then self.file.close() end self.output = {} self.file = io.open(fileName, "r") self.callBack = callBack or nil self.began = os.timer() self:start() end   function FileLoader:onTimer() -- PROCESS NEXT LINE IN FILE local input = self.file:read( "*l" ) -- READ ONE LINE FROM FILE if input then table.insert(self.output, input) -- INSERT LINE else self:stop() self.file:close() -- DISPOSE OF FILE self.file = nil self.elapsed = math.floor((os.timer() - self.began) * 1000) if self.callBack then self.callBack(self.output) -- CALL USER FUNCTION HERE self.callBack = nil end end end

To use it we just do something like this..

local FileLoader = FileLoader.new(10, 0) -- NEW FILE LOADER THAT READS ONE LINE EACH n MS   local function onFileLoaded(data) print( #data.." lines loaded in "..FileLoader.elapsed.."ms") end   FileLoader:beginLoad("data/somedata.txt", onFileLoaded)

So that should load the file in the background and call your callBack when the file is fully loaded.

This is just a very basic example and you could do more processing when each line is read, checking the first character for example to determine if the line is a question, or an answer, etc.

Does it help at all?

antix

I thought about this further and decided to expand my example and make another function that generates a table of questions from the data that was read. You could implement it in the FileLoader class but it is probably just better to call this function from your callback.

So in the data file that gets loaded.. Any line beginning with * is a question. The first line after a question is the correct answer and subsequent lines are false answers. This gives a good base for a multi choice question system and it accommodates as many or as few choices per question automatically.

As you iterate through the lines of text you can use some simple rules to generate the list of questions like this..

local function generateQuestions(data) -- GENERATE QUESTIONS FROM DATA local ssub, insert = string.sub, table.insert   local output = {} local question = {question = "", answers = {}} -- BLANK QUESTION for i = 1, #data do local line = data[i] if string.sub(line, 1, 1) == "*" then -- LINES BEGINNING WITH * DENOTES A QUESTION if i > 1 then -- SKIP 1ST INSERT BECAUSE IT WILL ALWAYS BE EMPTY table.insert(output, question) -- ADD NEW QUESTION end question = {question = string.sub(line ,2), answers = {}} -- NEW QUESTION WITH "*" STRIPPED OFF else table.insert(question.answers, line) -- ALL OTHER LINES ARE ANSWERS (FIRST ONE BEING THE REAL ANSWER) end end table.insert(output, question) -- ADD LAST QUESTION return output end

Now you can display a question and just present the answers in random order.

Anyway, I better get back to what I'm supposed to be doing

bigtunacan

@antix,

Thanks. As always you are very helpful. I haven't been logged in for a few days, but this is great stuff.

antix

@bigtunacan, you're welcome. I'll be making use of my class in future projects. I think it could also be adapted for use with streaming content.