Unfortunately almost exclusively via http rest apis. It's not great, but it's the lowest common denominator between the vast "ingestion" service we've built (connectors to web apis, local application for local file upload, raw api endpoints, etc.).
We've started exploring the apache arrow format as a compressible binary format with a dedicated wire format just to cut down on parsing processing costs.
We've started exploring the apache arrow format as a compressible binary format with a dedicated wire format just to cut down on parsing processing costs.