TTS (Speech synthesis) iOS7 Gideros plugin :D

ianchia · October 2013

Probably one of the most trivial plugins ever, but one of the most powerful, thanks to some incredible engineering inside iOS7.

This is just a quick play during my lunch. I'll gradually add all the other speech synthesizer options into the plugin my next lunch breaks. LOL. There will be a proper github repository. I'm just having fun and wanted to share...

Other options for the 2nd param are the following (Apple has made the major market voices the nicest, most money spent etc.)

th-TH
pt-BR
sk-SK
fr-CA
ro-RO
no-NO
fi-FI
pl-PL
de-DE
nl-NL
id-ID
tr-TR
it-IT
pt-PT
fr-FR
ru-RU
es-MX
zh-HK
sv-SE
hu-HU
zh-TW
es-ES
zh-CN
nl-BE
en-GB
ar-SA
ko-KR
cs-CZ
en-ZA
en-AU
da-DK
en-US
en-IE
hi-IN
el-GR
ja-JP

main.lua

require "BPiOSSpeech";
BPiOSSpeech.speak("Welcome to semiSirious. Roses are red, violets are blue. Please wait a sec while I grab some witticisms for you.","en-AU");

BPiOSSpeech.mm

#include "gideros.h"
#import <AVFoundation/AVFoundation.h>
 
static int speak(lua_State *L)
{
    NSString* utterance_string = [NSString stringWithUTF8String: lua_tostring(L, 1)];
    NSString* voice_locale = [NSString stringWithUTF8String: lua_tostring(L, 2)];
    AVSpeechSynthesizer *synthesizer = [[AVSpeechSynthesizer alloc] init];
    AVSpeechUtterance *utterance = [AVSpeechUtterance speechUtteranceWithString:utterance_string];
    utterance.rate = 0.2; // Tell it to me slowly
    utterance.voice = [AVSpeechSynthesisVoice voiceWithLanguage:voice_locale];
    [synthesizer speakUtterance:utterance];
 
    return 0;
}
 
static int loader(lua_State *L)
{
    //This is a list of functions that can be called from Lua
    const luaL_Reg functionlist[] = {
        {"speak", speak},
        {NULL, NULL},
    };
    luaL_register(L, "BPiOSSpeech", functionlist);
 
    //return the pointer to the plugin
    return 1;
}
 
 
static void g_initializePlugin(lua_State* L)
{
    lua_getglobal(L, "package");
    lua_getfield(L, -1, "preload");
 
    lua_pushcfunction(L, loader);
    lua_setfield(L, -2, "BPiOSSpeech");
 
    lua_pop(L, 2);
}
 
static void g_deinitializePlugin(lua_State *L) { }
 
REGISTER_PLUGIN("BPiOSSpeech", "1.0")

ar2rsawseen · October 2013

Wow, that sounds like one really interesting plugin
Especially for providing voice overs for in game texts

Mells · October 2013

@ar2rsawseen yes, or maybe children's book "click on a word to hear its pronunciation" feature?
Is that page on the forum still relevant today and up-to-date to learn how to test the plugin above?
I haven't started playing with plugins yet and have no idea where to put the files, etc...

@ianchia thanks for sharing

OZApps · October 2013

@ianchia, iOS7 has some amazing new API's but the hardware and the UI are a disappointment.

The other problem with Apple (yes, it is doing what Microsoft did earlier, turning away its developers) is making it difficult to develop for their platform. My iMac is still running Lion so I cannot update to xCode5. My iPad is running iOS 7 so cannot use it with the older version of xCode for development or adding apps onto it, I have to use alternatives like iTunes and synching. To develop for iOS7, I need atleast Mountain Lion andxCode5... this will break compatibility with several other older apps and utils that I rely on or use, the removal of rosetta has seen some good apps fade away that were not updated for intel platforms.

back to the point, that plugin is amazing

ianchia · October 2013

Here's a mp3 of what it says for those who are curious but haven't got into iOS7 or Gideros plugins. It's recorded using Voice Memos from my iPhone 5 talking into a 3GS, so it's a bit trebly. It sounds a lot better coming straight out of the phone. (-:

@ar2rsawseen @Mells @OZApps - I think the text-to-speech has some very interesting potentials beyond the usual "tap to say what's on the screen" user affordances.

Since iOS7 comes with *ALL* the voices built in, you could do audio localisation. Speak French text while the screen text is in English. Or have German accompanying text for something in Mandarin. The bonus is that all the text is just *plain text*. Not hefty audio assets compiled into your binary.

I think the world of foreign language learning has just been given a huge bump in ability for indie developers thanks to iOS7. I'm also *really* interested in what learning specialists for folk with language difficulties can suggest we can do with this technology. It's VERY VERY exciting. Basically, every indie iOS developer has been gifted with a high quality TTS SDK that would've been prohibitively expensive to deploy last month.

(And just think what kids could do what a DIY knock knock book! …. hahaha)

- Ian

ianchia · October 2013

@OZApps - if your iMac is capable of running Mountain Lion (and hopefully Maverick), why not just run Lion in a VM and use your older apps there, while moving ahead with ML/Xcode 5 development as your main host OS?

Respectfully, I'd have to disagree with you about Apple turning away developers. Xcode 5 and the newer iOS SDKs make life easier and better with each iteration overall. I remember the bad-ol days of provisioning hell with iOS3 and I'm never goin' back. (-X And Mountain Lion is really what Lion *should've* been. You'll enjoy ML much more than 10.7 … honest!

(And not sure what you mean by disappointing hardware. 64 bit processor! Oh the things I've been aching to do are going to be in reach for the main populace in 2 years ie. 6/5S/iPad5 etc! 64 bit processor! Whoo hoo. Seen this on the A7? http://www.anandtech.com/show/7335/the-iphone-5s-review/4 )

- Ian

ianchia · October 2013

My mind is still being blown away by the voice synthesiser in iOS7 but I've run into an edge-case that's got me a little puzzled.

Does anyone have any ideas on passing a UTF-8 string in Lua to Objective-C++?

For example, I have the UTF-8 mixed language string speaking using the Mandarin Chinese Voice "zh-CN" (which does an amazingly good job at speaking mixed English and Chinese phrases.)

"Welcome to semiSirious. 你好吗. All your base are belong to us."

The three Chinese glyphs in the middle are: ni hao ma (aka 'Hey, how are you?!')

I'm having trouble working out how to pass the Chinese in Lua to the plugin, since Lua strings don't support unicode. I could pass them as Unicode numerical character references (& #20320;& #22909;& #21527;) or Unicode entities (%u4F60 %u597D %u5417) (I added a space between the &# in this post as the web page was encoding it back to Chinese chars) but I don't know how to convert those entities back into the Chinese characters in the NSString. Using them as Unicode NCRs or Unicode entities within an NSString confuses the speech synth and it says the wrong thing. (A lot of the wrong thing actually - the three characters turn into long elaborate sentences … wacky!)

Has anyone come across passing Unicode strings from Lua to a plugin before?

Many thanks,

- Ian

ianchia · October 2013

Never mind, I've worked it out. (Long dry discussion on Unicode and NSStrings UTF-16's internals here: http://www.cocoabuilder.com/archive/cocoa/275083-nsstring-handling-of-unicode-extension-and-characters.html )

... NSString preferred format: \u4F60\u597D\u5417 turns it back to the Chinese characters that the speech synthesiser can pronounce correctly.

OZApps · October 2013

@ianchia,
I have been following Apple and have been wowed with the touchID and the 64-bit, It was indeed interesting and I was suggesting to people I knew that the Hardware in iPhone5S was the best thing one could get on the market, however when I said disappointing hardware, I was talking about this article http://www.gizmodo.com.au/2013/10/the-iphone-5s-motion-sensors-are-totally-screwed-up/

the iOS7 UI and software was a disappointment with the half baked icons and inconstancy in the UI colors, etc. amongst the plenty articles on this matter, https://medium.com/design-ux/ce87662270cf and the HTML bugs in iOS7 as documented here http://www.sencha.com/blog/the-html5-scorecard-the-good-the-bad-and-the-ugly-in-ios7/ and http://www.mobilexweb.com/blog/safari-ios7-html5-problems-apis-review

I have had a play with Mountain Lion as my MacBookPro is Mountain Lion and I am fine with Lion, it is similar to the choice between XP and Vista, though I was happy to move to 7, so if Mavericks is any good, I might move directly to that (I think it should be released in a couple of days now, any time soon) and though I have both parallels and VMFusion and even VirtualBox these are fine for running Windows, not tried running Mac OSX on this. Maybe I will try Mountain Lion on a VM just for xCode and keep the status quo and swap later if it is any good later.

ianchia · October 2013

@OZApps Hi Jayant, thanks for all the great links. (-:

I understand about the UI being half-baked but think about it this way … Forstall left midway through the cycle of iOS7 and Ives took over with 6 months to go. iOS8 will be the OS that unifies everything from a user-experience perspective. They just haven't had time to redevelop the OS with just half a year of redesigning the UI from the ground up, *plus* baking all the other advanced APIs into iOS7. Webkit and its integration into Mobile Safari is a prime example. There's a whole lot of really big advancements in Webkit and with the whole redesign of Mobile Safari, I'm pretty impressed it actually works the way it does.

It would absolutely be a horrible experience though if I was a mobile web-app developer. I do sympathise, but I've also learnt my lessons with living on the bleeding edge for years and being alpha/beta testers for big technologies. Not so interested in hurting myself anymore … (-;

The posts from Sencha and Mobilexweb were *really* interesting. I appreciate you sharing them - I do a lot of work using hybrid native/web technologies and it's always super useful to hear this stuff. I really must browse Sencha's blog more often … although I'm not a fan of their monolithic SDK. It takes a LOONG time to load a Sencha app. FYI, I found an iOS7 Webkit bug too (logged into Radar and open Radar) that wasn't listed in your two linked articles. If you're interested, it's here: http://openradar.appspot.com/15030460

That said, I recently released two Gideros apps that rely heavily on UIWebView for most of the viewports.

This one https://itunes.apple.com/app/open-studios/id388416851?mt=8 is a port of my magazine engine, and uses UIWebView almost exclusively, except for the UITabView which is actually Gideros sprites (no real reason to use OpenGL other than I had the code from an older codebase and I was in a hurry to ship.) I think it animates pretty well with CSS3 and UIKit transitions.

The core engine is MVC in Lua/Gideros and maintains rendering viewports using 4 UIWebViews and the minor UITabView Gideros OpenGL viewport. There's a templating manager using Lustache to generate the UIWebView content on the fly. It also uses Zurb Foundation 4 framework for a lot of the layout widgets. I'm pretty pleased with how it turned out considering that the development cycle was less than two week. Also the app submission/review process was a personal record for me. I submitted it a few days before iOS7's release and the app was approved in less than 24 hours! I suspect they approved *A LOT* of iOS7 apps that week! (-;

This type of MVC architecture where the model and controllers are in Lua and the views are a mix of WebKit and OpenGL works far better for me than an full webapp engine in JavaScript like the PhoneGap approach. For one thing, it's far easier to debug a Gideros app than a JavaScript app running on a device! Also, I don't have to worry about performance as much as an app fully living in webkit, since (from Lua) I tear down and reinstantiate webkit viewports on the fly, so memory consumption is kept under control. That said - the memory consumption of this app (around 70MB) is around twice that of the older pure Objective-C codebase. But the rapid development time is far worth the cost in running better on the newer devices. It runs fine down to an iPad 2 (it's a bit too slow on 3GS, but I can live with that.)

This enterprise app https://itunes.apple.com/app/photosentinel-viewer/id680404511?mt=8 uses a mix of UIWebViews and OpenGL (especially for the camera zooms on the time-lapse photos.) It was developed over a very long period due to personal illness, but despite the drawn out development cycle, again I'm happy with how it integrates into a cohesive user experience. Webkit provides a lot of bonuses (like all the UI widgets) that I don't have to rebuild in Gideros/OpenGL.

Overall, I'm pretty happy with the state of HTML development on iOS7 given my hybrid Gideros/custom plugins/Webkit approach. (So happy that I can whip up whatever I need in Gideros compared to Corona where I was hassling Carlos for many many months for needed functionality.) I'm excited to use the newer technologies like the advanced CSS layouts over the coming months. The new look&feel of the Safari Preview for debugging is also excellent and far easier to use than the Safari built into ML.

Best,

- Ian

Mells · October 2013

@ianchia
Thanks for sharing your experience with using Chinese with this plugin.
I'm going to need it soon for japanese texts.

ianchia · October 2013

Hi @Mells - could you post a paragraph of Kanji and/or Katakana into this thread? I'm curious how well the voice synth copes with it. I'll take what you post and attach back an MP3 for you to check? If you're able to do that, I'd be appreciative. Just got an idea for one of my existing apps to use Chinese and Japanese translations and wondering about the Japanese myself.

Many thanks (-;

- Ian

Mells · October 2013

Here it is

スタートアップウィークエンドとは？

スタートアップウィークエンドは世界の各地で行われ、高いモチベーションの開発者たちをはじめビジネスマネージャーたち、スタートアップに熱意を持つ人々、マーケティング専門家やグラフィックアーチストたちが集まりコミュニティ及び会社やプロジェクトを築き上げる三日間のイベントです。それは熱意とスキルを持っている人々とつながる為にもすばらしい機会ですし、あなたのアイデアを実現する為の創業の仲間たちが見つかるかもしれません。

ianchia · October 2013

Hi @Mells

Here it is attached. It sounds a bit choppy in places with phonemes being intercut with other sounds … since I don't speak Japanese, I can't tell how correct it is, but it doesn't sound "right" to me. I'll leave it up to your expert opinion. (This is my iPhone 5 speaking the text using iOS 7.0.2).

For what it's worth, I *think* it sounds better pronounced in Xcode's iOS Simulator, so perhaps Apple have swapped a better Japanese library in there that 'may' be what the iPhone 5S contains? I'm just speculating. I don't know.

Let me know what you think! Thank you for your help.

- Ian

Mells · October 2013

@ianchia
Unfortunately, overall this is not good.
However, from 00:17 to 00:30 and 00:40 to 00:43, it was really encouraging. Those parts are up to the level I would consider as acceptable (or better).

Howdy, Stranger!

Categories

In this Discussion

Top Posters

TTS (Speech synthesis) iOS7 Gideros plugin :D

Comments

Howdy, Stranger!

Quick Links

Categories

In this Discussion

Top Posters

TTS (Speech synthesis) iOS7 Gideros plugin :D

Comments