How do YOU do automated testing?

chipster123

Does anyone have techniques or tools they apply to regression testing their Gideros apps and libraries?
I'm looking for advice here.

Thanks!

jack0088

Don't know what advice exactly you're looking for, but I always test veeery much and very frequently. That's the case because I have a designer background and work better visually. It's easier for me to see issues that way. And I often work by what I would consider a try-and-error approach :P When I just don't get it I play with some parameters and make deep debugging. This obviously require a frequent testing.

So I set up TextMate (my fav editor) to work with Gideros. While coding I press CMD+R and everything opens in front of my eyes. So hitting this command is very fast and timeless.

Michal

If your business logic is separated from the view, you could use "standard" lua tools to test it from here: http://lua-users.org/wiki/UnitTesting

chipster123

@Michal, I understand what you're saying, but I'm looking to regression test the entire application, GUI and all. @jack0088, I'm looking to learn how other developers are testing their apps after making changes or adding features. Methods other than manually clicking through all the combinations. I realize now that this question is targeted more toward the utility application I'm working on than to game development, except for the gadgets and widgets people build for use in the games.

I'm more familiar with regression testing software that accepts input, crunches it and generates output. I then run comparisons against expected result and do this in a automated way. How does one recommend doing this for gideros based applications that require mouse interaction?

OZApps

@chipster, there used to be an app called Mercury this was used for GUI regression testing on Windows 3.1 and Windows 95 platforms. It would let you first set how you will test, it could be adding random entries and then performing a click on the button and would do that as many times as you want. It could also capture portions of the screen to compare or store as proof/results.

There were a few other such software that helped, the point is that if you want to test your logic as @Michal mentioned, you can test that using the command line (non GUI)

As for the GUI bit, you can record the same and then playback the sequence by using dispatch events. The only problem can be that since handlers are tied to specific objects, dispatching would require that you know those objects and it has to be part of the code (cannot be run as an external application)

Maybe you can also try the external method by using tools like Automator or keyboard maestro that can move the mouse and interact with the Gideros Player for GUI based taps, etc.

Magnusviri

http://sikuli.org/ (free)
http://www.testplant.com/products/eggplant/mobile/ (commercial)

My personal opinion is that setting up the GUI tests are so laborious that it isn't worth it, especially if your app changes a lot while you develop it. I don't even do unit tests but I guess I'm just lazy.

Michal

@Michal, I understand what you're saying, but I'm looking to regression test the entire application, GUI and all. @jack0088, I'm looking to learn how other developers are testing their apps after making changes or adding features. Methods other than manually clicking through all the combinations. I realize now that this question is targeted more toward the utility application I'm working on than to game development, except for the gadgets and widgets people build for use in the games.

I'm more familiar with regression testing software that accepts input, crunches it and generates output. I then run comparisons against expected result and do this in a automated way. How does one recommend doing this for gideros based applications that require mouse interaction?

I do unserstand what are you saying My experience is that it is better to put as many thing in your business logic as you can (so yes, even mouse clicks becomes mouseClick(x, y, button) method), because of two reasons: is is easier to test, and it is easier to port to another GUI.
The only way I see testing gideros application (apart from unit testing) is by comparing screenshots. But I don't belive it is a good way: some info on your screen may be random or time-dependent (like current time), yet still be correct.
For web thechnology, there are tools like selenium, but they can parse the page elements (skipping others). That would work for gideros if there was a set of GUI widgets, and you would use only those widgets in your app, and there was a way to parse this "view" of widgets. But for now, there is no such library, and I would stop and unit testing at this time (plus maybe for some screenshot comparisons for simple cases).
BTW, it would be easy to write a helper application, that "records" your actions (mouse clicks, finger touches and movement, button presses) and logs it in a file together with current timestamp, and then replaying it (regenerating the same set of events in the same order), so you could compare the results (screenshots for example).

ar2rsawseen

BTW, it would be easy to write a helper application, that "records" your actions (mouse clicks, finger touches and movement, button presses) and logs it in a file together with current timestamp, and then replaying it (regenerating the same set of events in the same order), so you could compare the results (screenshots for example).

@Michal, thats great idea
I think @bowerandy created something similar, called BhEventShield.

http://www.bowerhaus.eu/blog/files/better_app_videos.html

NReijmersdal

Here are some of my current idea's how to handle automated testing of a Gideros application.

Test-Driven Development:
When I start looking at a developing framework the first thing I do is check if there is a testing framework for Test-Driven Development.

Gideros does not include a testing framework (nor do any other mobile SDK's I know of), other LUA testing frameworks don't integrate well I think. Also I would like to run the tests from within the Gideros Studio.

I worked on something my self which hopefully suffices my basic needs. This resulted in gidTest which I published on github: https://github.com/nreijmersdal/gidTest

Figured it would be nice to share it here on the forums also.
Its still very hackish and basic, but I think its good enough for me to start working with it for a while.

GUI testing:
I successfully used the open source T-Plan Robot (http://sourceforge.net/projects/tplanrobot/) in the past. We did GUI testing of exports (Excel, PDF, etc) from a web-application with screenshots, this worked pretty good. T-Plan Robot has a recorder and its own internal scripting language. It lets you find parts of screenshot and interact with them. You can setup the Gideros player on a computer (maybe virtual) with a vnc-server and run your automated tests against that. These test could run on any VNC capable device, think jail-broken iPads or other mobile devices and are write once run anywhere, aslong the images stay the same. It has percentage matching, meaning the jpeg image compare could match 95%, this could help with scaling issues.

I don't think GUI testing is a good choice to start with while developing a game. Once your game is stable it might be a good idea, certainly in a bigger team which use continuous integration. Certainly if you would like to fix bugs later, like a year later and you forgot about how all the code works, its nice to have a series of automated tests.

Continuous Integration:
You can use the 'gdrbridge.exe' tool from Gideros to start the project against a player (which you can also start from the command line)
This means that it should be possible to run your tests from a CI server like Jenkins

ar2rsawseen

Wow @NReijmersdal that really sounds great. Will keep a close eye on your project and if you have any questions, you can definitely start a new thread and ask and keep us updated on your project

doridori

Excellent @NReijmersdal It seems you maybe the only one (I cant see any other threads on this for Gideros). I assume the best way to use is to import the gidTest.lua into my project and add a line to the top of main.lua which I would comment in/out as needed (as well as implementing the test_* funtions)?

Many thanks

ar2rsawseen

It seems not many are really interested in testing, like regression testing of the apps, or have some other solutions. But yeah, I'm actually also surprised that there are no more topics on this one on the forum

ianchia

Eric Wing wrote a wonderfully detailed five part post on how Coronalabs created their automated testing harness for successive builds of the Corona SDK. Highly recommended reading.

http://www.coronalabs.com/blog/author/ewing/

Note that these posts was written in 2011. Still a great and generous series of posts from Eric, but tweaks are most likely required for 2013 SDKs...

The third post covers Jenkins and automated on device testing

http://www.coronalabs.com/blog/2011/08/10/automated-testing-on-mobile-devices-part3/