Attempt on Localization

keszegh · August 2018

@MoKaLux
i did not read ealier comments, so maybe what i say is not new.
gideros can handle json files pretty well and there are converters from csv to json, google them. so these together should do the job for you. don't give up on your time invested into gideros yet, i'm almost sure your life with libgdx won't be easier.

hgy29 · August 2018

@MoKaLux, seriously ? CSV is most straightforward file format I know, it could be read with just a few lines of lua code without using an external lib. It makes me feel that either you don't know what a CSV is or you don't have basic lua knowledge. That said without wanting to offend of course.

MoKaLux · August 2018

@keszegh thank you for the support. Gideros is quite fun to use, the community is great. I will come back to gideros for another project.

For the project I am on I need some basic things done that is I have a .csv file with headers and I want to retrieve the value of one cell based on its row and column.

JAVA code:
tblarab = loadTable("_arab.csv", "header");
trow = tblarab.getRow(10); // 10th row is an example
String mystring = trow.getString("myheader");

voilà!

MoKaLux · August 2018

hello @hgy29, yep you've got a point, I am new to lua! As you say csv is the most basic file format but lua isn't straight forward. I have a header, I have a row, what value is in that cell? 25 lines of lua code to get it? I don't understand! I want to have some fun but I'm stuck.

hgy29 · August 2018

let me craft you a sample code, I will post it here in a few hours

keszegh · August 2018

@MokaLux, if you are that new to lua than you probably have many other things missing from your game anyway, by the time you have some practice, reaading a csv will be a breeze (especially that it means only adding one of the lua csv reader implementations and then a simple read call).

MoKaLux · August 2018

ok thank you very much for this. I'm having a quick try using libgdx to see! It is a simple project, nothing complicated, just to get some values from different rows and headers. Take your time!

SinisterSoft · August 2018

@MoKaLux Make sire you save your csv as a UTF-8 csv and not an MS-DOS csv - otherwise any special characters that are not in Ascii will go.

Download this and copy it into your source directory, include it in your project, right-click it and exclude from execution.

https://github.com/geoffleyland/lua-csv/blob/master/lua/csv.lua

This is a snip from my code. The csv file has columns that have the headers (in row 1), the first being my language 'en'. In the rows below that column I have the original text in my game - one per row. The other columns have 'fr', 'it', etc... and below each of the column headings I have the text for each language. If I add a language I simply add another column and the translated text below it.

function arabicProcessing(text)
	local Achar={0x002,0x024,0x062,0x084,0x0C4,0x104,0x144,0x184,0x1C2,0x1E2,0x202,0x222,
                               0x244,0x284,0x2C4,0x304,0x344,0x384,0x3C4,0x404,0x440,0x440,0x440,0x440,0x440,0x440,
                               0x444,0x484,0x4C4,0x504,0x544,0x584,0x5C4,0x602,0x622,0x644}
	local last=0	
	local rtext=""
	for p, cur in utf8.codes(text) do 
		if ((last>=0x627) and (last<=0x64A)) then
			local A=Achar[last-0x626]
			local unibase=0xFE8D+(A>>4)
			local of=0 --0=iso,1=fin,2=ini,3=med
			if ((cur>=0x627) and (cur<=0x64A)) then
				of=2+state
			else
				of=state
			end
			state=1
			if (of>=(A&0xF)) then of=0 end
			of=of+unibase
			rtext=rtext..utf8.char(of)
		else
			if last>0 then
				rtext=rtext..utf8.char(last)
			end
		state=0
		end
		last=cur
	end
	if last>0 then
		rtext=rtext..utf8.char(last)
	end
	text=""
	for p, cur in utf8.codes(rtext) do
		if cur==0x28 or cur==0x29 then cur=cur~1 end
		text=utf8.char(cur)..text
	end
	return text
end
 
local osLanguage=lower(sub(application:getLanguage(),1,2))
--osLanguage="fr"
local translated={}
if osLanguage~="en" then
	message("Language_"..osLanguage)
	if osLanguage=="pt" then osLanguage="pb" end
	local f=io.open("languages.csv","r")
	if f then
		local data=f:read("*a")
		f:close()
		local csv=require("csv")
		local f=csv.openstring(data,{header=true})
		if osLanguage=="ar" then osRightToLeft=true end
		for fields in f:lines() do
			local t=fields[osLanguage]
			if t then
				if t~="" then
					if osRightToLeft then
						t=arabicProcessing(t)
					else
						t=gsub(t,"ę","e")	-- replace letters our font doesn't have
						t=gsub(t,"Ő","Õ")
						t=gsub(t,"ő","õ")
						t=gsub(t,"Ű","Û")
						t=gsub(t,"ű","û")
						t=gsub(t,"ý","y")
						t=gsub(t,"’","'")
						t=gsub(t,"“","'")
						t=gsub(t,"”","'")
						t=gsub(t,"…","...")
					end
					translated[fields["en"]]=t
				end
				local n={"fr","it","se","hu","es","pb","ru","cs","ja"}
				for loop=1,#n do
					if osLanguage==n[loop] then
						narrow=0.8
						break
					end
				end
				if osLanguage=="ru" then narrow=0.7 end
			else
				print("*** language not found!")
				break
			end
		end
	else
		message("no languages file")
	end
end
 
local function translate(t)
	return translated[t] or t
end

In my code I then have things like this:

local t=TextField.new(smallFont,translate("Hello World!"))

So I always have the correct characters available I use this for setting up the fonts:

local chrs=""
fonts={{file="retro.ttf",sizeMult=1.0},{file="cyrillic.ttf",sizeMult=1.0},{file="japanese.ttf",sizeMult=0.8},{file="arabic.ttf",sizeMult=1.0}}
smallFont=TTFont.new(fonts,33,chrs,3)

Basically that will generate cached font data depending on which characters are used in the game - as they are used.

Armed with this information you should be ok.

MoKaLux · August 2018

wouaw! I would need some time to test! Ok, let's do it (libgdx can wait for godot :-)

Thank you all.

hgy29 · August 2018

As promised, a little sample code to show how it could be done without even using an external csv lib:

local function loadCSV(file)
	local function csvsplit(line)
		local t,w,l2={},nil,nil
		while #line>0 do
			w,l2=line:match("\"([^\"]*)\"[,\r\n]?(.*)") -- Check for quoted string
			if not w then w,line=line:match("([^,]*)[,\r\n]?(.*)") -- Non quoted 
			else line=l2 end
			if not w then break end --Nothing or issue, break
			t[#t+1]=w
		end
		return t
	end
	local header,csv=nil,{}
	for line in io.lines(file) do
		f=csvsplit(line)
		if not headers then -- Assume first line is headers
			headers={} for n,v in ipairs(f) do headers[v]=n end
		else
			csv[#csv+1]=f
		end
	end
	csv.getField=function (self,row,field) return self[row][headers[field]] end
	return csv
end
 
csv=loadCSV("135-bis-bt.csv")
print(csv:getField(4,"Value "))
print(csv:getField(7,"Time"))

SinisterSoft · August 2018

@MoKaLux I was stuck with the Arabic text so @Hgy29 sent me the arabicProcessing function in my example.

SinisterSoft · August 2018

You should also comment out or ignore the 'message' function in my code, that just sends out a print to any debug systems I use - like the console, flurry, facebook logs, and firebase.

The narrow variable I use to set an XScale for the text - to make it narrower for those languages.

I also set the textfield spacing to -1 for if the osRightToLeft has been set.

totebo · August 2018

A lot of activity in this thread! @MoKaLux, let us all know if you get it to work. I haven't used Arabic, but CSV files are easy and pretty much required for localisation .

MoKaLux · August 2018

I have got module csv not found error
no field package.preload['csv']

I did as you explained, included the code for csv.lua in my project then right clicked exclude from executing.
When I check the path for csv.lua it is in my project classes/csv.lua (so it is included).

I right clicked my lua file arabic_text.lua and checked csv.lua in file dependencies but still the error.

Can you help some more?

MoKaLux · August 2018

ok I need to check the other answers as I was busy on the first one. Let me try again please.

hgy29 · August 2018

MoKaLux said:

When I check the path for csv.lua it is in my project classes/csv.lua (so it is included).

If it is in classes/csv.lua, then require should be 'classes/csv' or 'classes.csv' instead of just 'csv'

MoKaLux · August 2018

thank you all + thank you @hgy29 your solution "worked" but the letters are not attached.

Now I need to mix your csv solution to your arabicProcessing code and get those letters linked together!

Thank you very much for your kind cooperation. I will post my progress and hopefully my success.

I am enjoying gideros a lot!

MoKaLux · August 2018

and by the way we should post the csv code in some more accessible place as well. That's a little gem!

antix · August 2018

@MoKaLux what were you having trouble with? As far as I am aware a CSV file is just a text file with comma separated values on each line of the file. To read and parse a csv file you can do something like this...

-- split text into substrings using separator char
function string:split(sep)
  local sep, fields = sep or ",", {}
  local pattern = string.format("([^%s]+)", sep)
  self:gsub(pattern, function(c) fields[#fields + 1] = c end)
  return fields
end   
 
-- read a text file line by line and print each line
local line
local file = io.open('contacts.csv', 'rb')
while true do
  local line = file:read("*l") -- read next line from file
  if line == nil then break end -- exit if no more lines
 
  local fields = line:split() -- split line into strings (comma separated)
 
  for i, v in ipairs(fields) do -- print all values in line
    print(i, v)
  end
end
file:close()

I tried LibGDX some time ago before I found Gideros. Looking at LibGDX now it seems kind of primative IMHO

MoKaLux · August 2018

hi antix, yes you are all familiar with gideros and lua, but I come from java where it is a little bit more to the point regarding csv files.

Gideros and the gideros community are really GREAT that's why I am interested in lua but I prefer java code to be honest. Thank you for your post. Peace. I'll get back soon to post here my progress.

SinisterSoft · August 2018

@antix csv files can get (slightly) complicated - sometimes the file has a BOM header to say it's UTF-8 (that's a relatively recent addiction thanks to Microsoft and it's UTF8 CVS export from Office 365). Also some of the data may use quotes and that has to be accounted for.

MoKaLux · August 2018

hello everybody, it is now working as "expected".

So the steps are:
1- a .csv file with "headers" UTF8 ex: "txtCode","txtAr","txtFr","gfx1","gfx2","txtEn"

2 - some data in the next rows ex: "","هٰذا كُرْسِىٌ.","Ceci est une chaise.","gchair","ghandy","This is a chair." *

* please note I have inserted arabic words with vowels on purpose to test but it does not work with vowels so your words should be without vowels.

3 - use the loadCSV.lua (it's a gem please see post above) from @hgy29

4 - use arabicProcessing.lua from the same nice author @hgy29

5 - use some appropriate fonts ex: DejaVuSans.ttf, DroidNaskh-Regular.ttf, ...

Et voilà!

Thank you all for this, I wouldn't have figure it out by myself, I don't like lua but I love Gideros!

One downside: the arabicProcessing doesn't take care of arabic vowels I don't know if that is possible like in java :-) that would be great.

Once again thank you ALL for the help.

hgy29 · August 2018

lua can be a little difficult to grasp at first when coming from Java or C/C++, but you'll eventually become used to the freedom lua brings: no static typing, inject variable or code into whatever class/instance you want (tables in lua), override 'system' code, etc. It reminds me of BASIC somehow, but it is way more powerful.

But yeah, I had hard time with it the first time I tried lua, I had to write a wireshark dissector at that time...

My favorite languages are Java (for large codebases) and Lua (for rapid prototyping or small projects), although I am comfortable with most languages: C/C++, PHP, assembly, VHDL, Pascal, Javascript,... I didn't like python, I am scared by perl, and hadn't a chance to touch ruby yet

SinisterSoft · August 2018

@MoKaLux Glad you managed to sort it in the end. If you figure out what's wrong with the vowels can you post the update here?

MoKaLux · August 2018

ok, but for that I need to search if someone did it in pure lua and then see if I can manage to transpose it to gideros. I may have a look at corona forum as well but I promise I won't use corona (I don't like it). Gideros rulez! It's no problem in java :-)

MoKaLux · August 2018

@hgy29 yes that's what I call a professional coder and you showed your talent in this post (as in many other posts as well). I started lua with love2D, it's great but no easy export to android! Gideros rulez! Godot is very powerful too I did some apps with it. I had a lot of fun, very easy, very fun, ... Why changed? the apk size it produces are a bit too big for me (40mb) for a basic project. Gideros is around 8mb for a basic project (I mean once installed on the device). Hope to make great apps using gideros in the near future with lua, oh lua!

keszegh · August 2018

@MokaLux, threatening us to move to another sdk was a good trick to get many responses, i shall try this trick too when i'm stuck.
joke aside i'm glad you found the solution with the help of the forum members.

about apk size, i think if you exclude some plugins then this can be decreased further. maybe you already do this.

Apollo14 · August 2018

MoKaLux said:
I give up! going back to java :-(

libgdx here I come!

why not C++ & Cocos2d-x?
it would be way better choice than libgdx

MoKaLux · August 2018

@keszegh yes that worked quite well! but I was not joking, I have been on a quest for a month testing any kind of engine I could find (@Apollo14 no I didn't try cocos2d-x, I don't know c/c++).

The apk size was between 40mb(godot, embarcadero) up to 100mb (haxe flixel) for a basic project so that was a no go further for me. So now I really want to get going coding and having fun.

With gideros getting the arabic part working would be awesome and that would mean the end of my quest. So now I need to do a little more searching how to do that in lua (or back to java?)

SinisterSoft · August 2018

I think you need to look at how utf8 characters are made up - so that the text is converted properly.

Howdy, Stranger!

Categories

In this Discussion

Top Posters

Attempt on Localization

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Top Posters

Attempt on Localization

Comments