[bigfoot675]

I think taking the approach of "help[ing] your team make informed, data-driven decisions" through generating reports is valuable here. In my opinion, it goes too far if we start continuously deploying ML code like it's a SWE project. To take an example in the case of autonomous vehicles, pushing continuous updates to perception modules without thoroughly exploring the ramifications of an update could be potentially catastrophic.

Obviously we can't predict every error by thinking hard, but datasets will never serve as a full representation of what models might experience in the real world. Continuous deployment to an ML model could affect undefined behavior in unpredictable ways.

[rhythmvertigo]

Definitely- automatic model deployment would only work for a subset of applications. We think a lot more about having models available as candidates for deployment, which can then be inspected by domain experts on a team.

One of our motivations for building visual reports that appear like comments in a pull request is giving teams metrics & info to discuss when deciding if merge is right. That way, the automated part is training and testing, but the decision making is human (i.e., data scientists whose skills are better used interpreting models & data than running repetitive training scripts).

[ishcheklein]

Agreed! I think, it's similar to software engineering. It's not that often that large websites are being deployed to prod without someone approving/QA-ing it manually first, right?

CI/CD systems in this case help automating this as much as possible, but do not completely replace decision making process, I would say.