Machine learning requires a problem that you can have partially correct, so that it can climb the gradient to optimize on. If you can build tests that have an analog instead of pass/fail output, you could, in theory, do it with machine learning.
Beware that machine learning in a single pass/fail is more like having an infinite number of monkeys trying to write the works of Isaac Asimov.
[Edit/Update] All of the tests could be individual values, so non-zero (but nowhere near all ones) might help. Thanks for making me reconsider this, sdenton4.
Not all ml is gradient based. Other options exist: Bayesian black box optimization (like vizier), or genetic algos. And Vizier is actually quite efficient for small problems.
With genetic algorithms you still need to be able to calculate a fitness. Usually a test is fail/success. There's no fitness in that.
I would guess the other optimizers also need such a signal?
Success/failure is just a binary classification signal: you can look at how it correlates with your target variables, how noisy it is for particular choices of variables over multiple trials, etc. The noisier it is, the harder it is to learn, but such is life.
I think it would be impractical for a naive fitness function that is 0 for failure and 1 for success. Wouldn't the signal be too difficult to find? GA would be brute force until you find code that passed a test.
I don't think tests for factorio are trivial.
Maybe you could move the goal/fitness function along the way. So start with something that compiles. Then having the desired input output. Etc.
Addendum. More tests. I see. Would you then lead the algorithm along a trajectory. If you can pass this simple test you would probably be able to pass this as well then this... Babysitting it along the way. Ideally you wouldn't need to, but maybe to make it possible..
Beware that machine learning in a single pass/fail is more like having an infinite number of monkeys trying to write the works of Isaac Asimov.
[Edit/Update] All of the tests could be individual values, so non-zero (but nowhere near all ones) might help. Thanks for making me reconsider this, sdenton4.