[agibsonccc]

Disclaimer: I sell a product in this space.

Our model server just works with every format. We created our own DL framework and maintain that. Eg: We're the 2nd largest contributor to keras alongside having our own DL framework.

We've found it's easier to just have 1 runtime with PMML, TF file import, keras file import, onnx interop coupled with endpoints for reading and writing numpy, apache arrow's tensor format, and our own libraries binary format. We also have converters for r and python pickle to pmml as well. Happy to elaborate if interested.

For ingest we typically allow storage of transformations, accepting raw images, arrays etc. We have our own HDF5 bindings as well but mainly for reading keras files. Could you tell me a bit about what you might want for ETL from HDf5?

[jrumbut]

My work right now is more at the research/prototyping end (and often not deep learning), unfortunately. I will say though that your product is exactly the sort of thing I was referring to, and deeplearning4j is a great resource. It's cool to run into the author!

Would you be willing to characterise the sort of system that you end up replacing, what your customers were doing before using your service?

[agibsonccc]

Sure! So our model server usually replaces a bunch of docker containers with different standards with zero ways of inputting data.

An interesting example was a customer that bought a startup. The startup had a docker container that leaked memory (they were using an advanced TF library, google quit maintaining it). The parent company had tomcat infrastructure. They told the startup to get it to work in tomcat. They approached us.

Dl4j itself is built on top of a more fundamental building block that allows us to do some cool stuff in c: https://github.com/bytedeco/javacpp-presets

We maintain the above as well. We have our own tensorflow, caffe, opencv,onnx etc bindings. This gives you the same kind of pointer/zero copy interop you would in python but in java.

Another area we play in is RPA. You have windows consultants who don't know machine learning but deal with business process data all day. They want standardized APIs and to run on prem on windows: https://www.uipath.com/webinars/rpa-innovation-week/custom-b...

They want the ability to automatically maintain models after they are implemented with an integration that auto corrects the model. We do that using our platform's apis. On line retraining is another thing we do.

We also do automatically rollback based on feedback if you specify a test set. If we find the test is less accurate for a specified metric over time, we roll back the model. This prevents a problem in machine learning called concept drift which means the domain changes over time.

Lastly, you have customers generating their own models and you need to track/audit them all. Sometimes to do online retraining like the RPA use case. Some customers scale to millions of models. They need this kind of tracking.