Loading large files on Android?

bigtunacan · April 2016

I'm looking at loading files with the datasaver module and some of my files are fairly large. My concern is that Android will shut down an application if it locks the main thread for too long and I'm dealing with some files that are fairly large.

I'm wondering how others are dealing with this?

talis · April 2016

Even there will be no error, it is always a good approach to divide the big files into small ones, if there will be no index or kind. Actually to hold index on big files can also have proplem hence the file will be a asset and in any case will be loaded into memory totally when you try to read or write. ( I am not %100 sure how the read/write file functions behave about memory management but guessing that they will load the entire file into memory )
If you will divide somehow you can init, search and get to the information much more faster.
I don't know what kind of file is it but let's suppose that it is a dictionary file just to give an example:
-You can divide the dictionary file into smaller files and give a prefix like
af_xxx.txt
go_xxx.txt
jp_xxx.txt
and so on according to word list in each files...

So in this case you can make a very simple index like the word "Gideros" will in the file which is starting with "go" prefix.

Of course as i wrote have no idea what is the purpose of your big file, but if you will share more information maybe some other ideas can be invented

bigtunacan · April 2016

I won't have control over all of the files that are loaded. I create file sets that contain quiz type sets of questions that users the practice and compete with (this is an educational learning game).

Some of these sets of questions are controlled by me, but my users will also be able to create sets of questions/answers which can be used to compete against one another.

I could put some arbitrary limit and not allow files over a certain size (which I may end up doing for memory purposes), but in terms of locking the main application thread I think that will be hit before memory usage especially on lower end Android devices.

What I would like to do is spin up a background thread that does the file loading and present the user with a loading spinner or such.

If I were creating a native Android application I would do this with an IntentService

http://developer.android.com/training/run-background-service/create-service.html

I'm wondering if there is a way to do something similar with Gideros where I can spin up and control a background thread that can provide me with some type of callback mechanism once it has completed.

antix · April 2016

@bigtunacan, I haven't tackled anything like this yet but if you have a giant text file with question and answer pairs you could get your program to read one line of the file each frame (or on some other timed event) and the program can run as normal but just not access the data until it has all loaded. In theory it could be setup with a callback and everything. After thinking for a while I made an example (attached)..

First the class that does the actual loading..

 
FileLoader = Core.class(Timer)
 
function FileLoader:init(delay, repeatCount) -- STARTUP
  self:addEventListener(Event.TIMER, self.onTimer, self)
  self.callBack = nil
  self.file = nil
  self.output = {}
  self.began = 0
  self.elapsed = 0
end
 
function FileLoader:dispose() -- SHUTDOWN
  self:stop()
  self:removeEventListener(Event.TIMER, self.onTimer, self)
  self.callBack = nil
  if self.file then self.file.close() end
  self.file = nil
  self.output = nil
  self.began = nil
  self.elapsed = nil
end
 
function FileLoader:beginLoad(fileName, callBack) -- BEGIN LOADING A NEW FILE
  self:stop()
  self:reset()
  if self.file then self.file.close() end
  self.output = {}
  self.file = io.open(fileName, "r")
  self.callBack = callBack or nil
  self.began = os.timer()
  self:start()
end
 
function FileLoader:onTimer() -- PROCESS NEXT LINE IN FILE
  local input = self.file:read( "*l" ) -- READ ONE LINE FROM FILE
  if input then
    table.insert(self.output, input) -- INSERT LINE
  else
    self:stop()
    self.file:close() -- DISPOSE OF FILE
    self.file = nil
    self.elapsed = math.floor((os.timer() - self.began) * 1000)
    if self.callBack then
      self.callBack(self.output) -- CALL USER FUNCTION HERE
      self.callBack = nil
    end
  end
end

To use it we just do something like this..

local FileLoader = FileLoader.new(10, 0) -- NEW FILE LOADER THAT READS ONE LINE EACH n MS
 
local function onFileLoaded(data)
  print( #data.." lines loaded in "..FileLoader.elapsed.."ms")
end
 
FileLoader:beginLoad("data/somedata.txt", onFileLoaded)

So that should load the file in the background and call your callBack when the file is fully loaded.

This is just a very basic example and you could do more processing when each line is read, checking the first character for example to determine if the line is a question, or an answer, etc.

Does it help at all?

antix · April 2016

I thought about this further and decided to expand my example and make another function that generates a table of questions from the data that was read. You could implement it in the FileLoader class but it is probably just better to call this function from your callback.

So in the data file that gets loaded.. Any line beginning with * is a question. The first line after a question is the correct answer and subsequent lines are false answers. This gives a good base for a multi choice question system and it accommodates as many or as few choices per question automatically.

As you iterate through the lines of text you can use some simple rules to generate the list of questions like this..

local function generateQuestions(data) -- GENERATE QUESTIONS FROM DATA
  local ssub, insert = string.sub, table.insert
 
  local output = {}
  local question = {question = "", answers = {}} -- BLANK QUESTION
  for i = 1, #data do
    local line = data[i]
    if string.sub(line, 1, 1) == "*" then -- LINES BEGINNING WITH * DENOTES A QUESTION
      if i > 1 then -- SKIP 1ST INSERT BECAUSE IT WILL ALWAYS BE EMPTY
        table.insert(output, question) -- ADD NEW QUESTION
      end
      question = {question = string.sub(line ,2), answers = {}} -- NEW QUESTION WITH "*" STRIPPED OFF
    else
      table.insert(question.answers, line) -- ALL OTHER LINES ARE ANSWERS (FIRST ONE BEING THE REAL ANSWER)
    end
  end
  table.insert(output, question) -- ADD LAST QUESTION
  return output
end

Now you can display a question and just present the answers in random order.

Anyway, I better get back to what I'm supposed to be doing

bigtunacan · April 2016

@antix,

Thanks. As always you are very helpful. I haven't been logged in for a few days, but this is great stuff.

antix · April 2016

@bigtunacan, you're welcome. I'll be making use of my class in future projects. I think it could also be adapted for use with streaming content.

Howdy, Stranger!

Categories

In this Discussion

Top Posters

Loading large files on Android?

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Top Posters

Loading large files on Android?

Comments