[seanwilson]

> I think the workflow is pretty easy: buy a MacBook & an iPhone, and load your app onto the phone & test on it for things which the iOS simulator is not good at.

> Don't CI your apps, CI your libraries.

That's not scalable for complicated apps that you can write automated UI tests for though.

[falsedan]

Could you take the complicated bits and put them in a library? How complicated is your app?

To me, what you described is an app that was not designed to be easy to test component-by-component & will have a high overhead on maintenance. Apple’s restriction on virtualising MacOS seems unrelated to how the app was architected, so it feels unfair to expect Apple to alter their position to better support something they weren’t responsible for.

[seanwilson]

> To me, what you described is an app that was not designed to be easy to test component-by-component & will have a high overhead on maintenance. Apple’s restriction on virtualising MacOS seems unrelated to how the app was architected, so it feels unfair to expect Apple to alter their position to better support something they weren’t responsible for.

We have complex commercial apps with fully automated user interface tests (e.g. it'll test that you can enter your username + password, actually login and you'll be able to see content). You could test all the individual components as much as you want but you're still going to get bugs at the UI layer that you can automatically test for.

We do the same for Android and CI for that is so much simpler.

> Apple’s restriction on virtualising MacOS seems unrelated to how the app was architected, so it feels unfair to expect Apple to alter their position to better support something they weren’t responsible for.

You'd think they'd want to support workflows that led to better quality apps being created. Surely it's their responsibility to support developers?

[falsedan]

> you're still going to get bugs at the UI layer that you can automatically test for

Surely you can prove the correctness with integration tests of your modules, and then your acceptance tests can be simple 'did the thing not show an error when we clicked on the thing, and it went to the right page'? We have 'simulate the world' 'acceptance' tests at work and they're terribly flaky, and I look back at the layered approach to testing we used in a previous job where the go-live test was to hit the landing page & search results page & check that the status code was 200.

To be fair, the kind of tests I'm proposing won't catch layout issues. But, neither will the tests you're using.

[seanwilson]

I'm not saying it's always a good idea to write exhaustive UI tests but it's nice to have the option. There's apps that I've worked on where you simple must test several business critical UI interactions haven't broken (e.g. logging in, submitting data) before you deploy. Apple getting in the way of automating this without a decent reason is annoying.

What made your UI tests so flaky? As long as the code for them isn't much beyond "enter text into text input X, click this button, you should be able to see this text" they're not too bad to write but the initial setup can be tiring.

> To be fair, the kind of tests I'm proposing won't catch layout issues.

Screenshot comparison based tools can be amazing for this as long as the way your app looks is fairly stable.

Automated testing is always a tradeoff with effort to setup vs the time you save (I'm against e.g. TDD for everything) but for complex apps and big teams it eventually pays off.