Hi Everyone,<p>So I've been working on a mobile game engine for Saito (https://apps.saito.network/arcade) and am running into testing issues.<p>Right now I'm having to manually run the applications on whatever cellphones and tablets we can find. Is there a better way to do this? Asking as I figure if anyone has solved this problem with an automatic testing suite or something similar it will be someone who reads HN....<p>Goal is to test mobile browsers to see which ones work and eliminate avoidable failures. Most of the problems we are flagging to date are javascript-implementation issues or things like auto-transaction "features" on foreign-language browsers that strip events from the page and/or stop them from running very quickly.<p>Any suggestions? Anyone have a startup that tackles this sort of problem?
For local automated testing of mobile browsers on both iOS and Android you probably want to have a look at <a href="https://appium.io/" rel="nofollow">https://appium.io/</a>, which uses the Webdriver protocol.<p>If you want to run the same tests against both mobile and desktop browsers, I recommend <a href="https://webdriver.io/" rel="nofollow">https://webdriver.io/</a>, which uses Appium for mobile testing.<p>If you're interested in a Docker setup as a boilerplate to test against all major browsers (both mobile and desktop) with best practices for e.g. test video recordings, you might also wanna take a look at this repo of mine: <a href="https://github.com/blueimp/wdio" rel="nofollow">https://github.com/blueimp/wdio</a>
Are you manually testing, or programmatically/automated? If it’s the latter then Browserstack (<a href="https://www.browserstack" rel="nofollow">https://www.browserstack</a>) provides mobile devices for testing on mobile browsers in a cloud device farm, there are others too like SauceLabs and AWS Device farm. I’ve had good success with BrowserStack but you should check whether they support the browsers you need, they also provide an easy way to tunnel through to your staging/qa environment if you need to do that. For reporting, especially if it’s not just you and you’re in a team, consider Tesults (<a href="https://www.tesults.com" rel="nofollow">https://www.tesults.com</a>), I’m the founder so hit me up with a message and I’ll do what I can to get you setup.
Appium with Selenium (or XCUITest for iOS) will shorten your life expectancy and grow you some grey hairs, but there is no other better alternative for testing at user level. It is highly unreliable and each new phone OS version will tend to break your automation<p>As for finding devices there are a few cloud providers like xamarin or perfecto (never really worked with them in production) that gives you remote access to a lot of real devices and software versions<p>Having said that try to divert more testing towards lower levels- unit tests, testing on simulators, integration tests with a full phone and browser etc.<p>A good place to also ask is:
<a href="https://testersio.slack.com" rel="nofollow">https://testersio.slack.com</a>
The first task to write some sort of test script to have assertions for the core features of your game.<p>For testing specific UI features, I would recommend looking into Selenium/Appium. Selenium for desktop browser testing, Appium for mobile testing (using Appium you can do mobile specific things like touch actions for instance).<p>For testing JavaScript, you can take a look at libraries like <a href="https://jasmine.github.io/" rel="nofollow">https://jasmine.github.io/</a>.<p>Once you have a test script(s) ready, you can point it to cloud based testing services like BrowserStack (I work there).<p>We provide a large number of actual physical mobile devices to test on (no emulators), For complete list see: <a href="https://www.browserstack.com/list-of-browsers-and-platforms/automate" rel="nofollow">https://www.browserstack.com/list-of-browsers-and-platforms/...</a><p>Our service is completely free for open source projects !<p>If you want to go the manual way, you can also do that with BrowserStack, we provide the 'Live' product where you can control and test your website on actual mobile devices straight from your browser.
Let me add visual diffing constraint to the discussion, otherwise I need the same as OP. percy.io is a cool visual diffing tool I used and liked a lot.<p>Last time when I used it it supported only chrome/firefox, but both mobile and desktop.<p>Please let me ask how is visual testing supported in other solutions like: selenium, appium, browserstack, webdriver.io, etc aka what do you use for crossbrowser visual testing?
They let you test with real hardware. <a href="https://www.perfecto.io/" rel="nofollow">https://www.perfecto.io/</a> Used them for a long time and everytime amazed by the amount of hardware they have. The whole setup is kind of need.
HeadSpin <a href="https://www.headspin.io/" rel="nofollow">https://www.headspin.io/</a> provides a large set of real Android and iOS devices distributed around the globe. Testing on real hardware is always better than testing in an emulator, and testing in a real location provides additional data points that help you eliminate performance issues with your application.<p>Testing just mobile web browsers is of course supported, but if you ever have the need you can test native game engines and get video and network performance statistics at the same time. Automated testing via Appium is natively supported, or you can bring your own framework.
We mainly use Browserstack. It's a very powerful tool, especially with their very impressive Chrome dev-tools integration, though it can be a bit laggy at times. Quite unbeatable for the huge array of devices they have available though.<p>We also have a device table with some common devices that we can use whenever needed. A recent Samsung Galaxy phone (not sure which model), iPhone SE, Kindle Fire and an iPad etc.
Most devtools on browsers have "responsive tools" - letting you change your screen to mobile or adjusting pixels. I use it sometimes for my web apps.