[mandevil]

Optuna or skopt are open source and won't take any GPU time at all to do it.

[janalsncm]

Optuna requires exploring the hyperparameter space which means running the experiments with those hyperparameters.

For a fixed search space it will almost certainly be better though.