Create a list of unique characters from localised string [SOLVED]

totebo · June 2016

Edit: Please scroll down to see accepted answer. I'll keep this here to provide a negative, before I provide you with a positive.

-- 8< --

I need to extract a list of unique characters from a string, so I can cache these when creating a font. This works, but not with special characters, that take up more than one byte each. I'm planning to use this for localisation.

Please see the code below. What is a safe way of accomplishing this?

--local str = "A string without special characters."
local str = "A string with special characters åäö."
 
local str_unique_chars = ""
 
for i=1, #str do
	local c1 = str:sub(i,i)
 
	local add_char = true
 
	for n=1, #str_unique_chars do
 
		local c2 = str_unique_chars:sub(n,n)
 
		if c1 == c2 then
			add_char = false
			break
		end
 
	end
 
	if add_char then
		str_unique_chars = str_unique_chars .. c1
	end
 
end
 
print( str_unique_chars ) -- Returns: A stringwhpecalå��.

totebo · June 2016

I noticed this upcoming feature. Would it help to use this instead?

http://docs.giderosmobile.com/reference/lua/utf8

pie · June 2016

@totebo I used this: it worked for german, italian and spanish
http://giderosmobile.com/forum/discussion/comment/46501#Comment_46501

totebo · June 2016

Fantastic! It works. Thanks! Here is the modified code:

--local str = "A string without special characters."
local str = "A string with special characters åäö."
 
local str_unique_chars = ""
 
for c1 in str:gmatch("[%z\1-\127\194-\244][\128-\191]*") do
 
	local add_char = true
	for c2 in str_unique_chars:gmatch("[%z\1-\127\194-\244][\128-\191]*") do
 
		if c1 == c2 then
			add_char = false
			break
		end
 
	end
 
	if add_char then
		str_unique_chars = str_unique_chars .. c1
	end
 
end
 
print( str_unique_chars ) -- Returns: A stringwhpecalåäö.

n1cke · June 2016

I added full UTF8 support (for example Lua 5.3 has only basic one) for upcoming Gideros release: https://github.com/gideros/gideros/pull/236
With it you can also cache all chars you need with the following code:

function newCharset(t)
	local chars = {}
	for i = 1, #t, 2 do
		for char = t[i], t[i+1] do
			table.insert(chars, utf8.char(char))
		end
	end
	return table.concat(chars)
end
 
local cache = newCharset{1, 65536} -- cache all
local font =  TTFont.new("BlissPro-Bold.otf", 30, cache, true)

`newCharset` accepts a table with integer pairs (ranges) i.e. you can use {32, 127} to get all Latin chars or {0x0400, 0x04FF} to get all Russian chars or add any charset ranges: {0x0590, 0x05FF; 0x0700, 0x074F; 0x0A00, 0x0A7F; ...}

Howdy, Stranger!

Categories

In this Discussion

Top Posters

Create a list of unique characters from localised string [SOLVED]

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Top Posters

Create a list of unique characters from localised string [SOLVED]

Comments