[sandoooo]

Let's say I have a hyperparameter optimization task where I have to tune a simulation to some spec by varying 2-4 input parameters, and the output is a single number. I have no analytical gradient, though it's probably OK to assume the domain is smooth. Each sim takes hours to run, the entire search could take days, and I would like something that works well in parallel so I can speed up the search. What's the state of art here? Are there anything close-to-state-of-art that's useable out of the box? I've read a few papers but they don't tend to come with software.

[mrdmnd]

I think Bayesian Optimization is what you're looking for here. There's an internal tool at google ("Vizier") for which a white paper has been published that solves this exact problem. I don't know if there are any public implementations of Vizier but you could probably reverse engineer some of it from the white paper: https://static.googleusercontent.com/media/research.google.c...

[unishark]

Powell's method (actually methods, with different solvers depending on details such as whether you have constraints) is very old and implemented practically everywhere.