[_eric_z_lin]

This looks like a really useful tool for keeping AI systems optimized, especially as models and data evolve over time. I'm curious, have you considered how Nomadic might integrate into CI/CD pipelines? It seems like it could be valuable for automatically re-tuning parameters and ensuring performance doesn't degrade with new model versions or data updates. Any plans for features that would support this kind of continuous optimization workflow?

[simbasdad]

Thank you so much. Yes, we believe CI/CD pipelines are a treasure trove of data for continuous ML system optimization (these are non-deterministic systems run repeatedly with new evaluation results at each run), where you get to learn about your own ML systems. Nomadic integrates well here to continuously collect data, that it can then use to better identify optimal HP configs on the same systems. We envision this as: every time you run CI/CD pipeline, you get more data with which you can learn about your ML system better, and Nomadic is your engine to realize this.

[elizabethhu]

to add on-- if you're more interested in real-time optimization (where the best configs are automatically set and iterated on in your system), Nomadic can integrate directly at the application level within your production code. You can then make queries like nomadic.get_optimal_value(experiment_id="...", default=...) to fetch the most recent and optimal hyperparameters for your system. This approach lets you continuously refine and set the best versions of your production system using both your CI/CD pipeline and historical production data