[bogomipz]

Thanks for the insights. I was curious about this however:

>"But operating with CRDs at scale is a different story and suggests careful testing with the specific applications involved."

Do you mean the number of different CRDs deployed here or just the number of custom resources created? Or is it the same concern with either? I'd be curious what you are defining as "scale" as well?

[psoundy]

The number of deployed CRDs is not likely to be an issue. The number and size of custom resources (CRs - instances of CRDs) is potentially an issue.

Scalability is relative, and depends on many factors including but not limited to:

- the resources available on the hosts running apiservers and etcd members

- the number and size of resources (custom and native) that controllers will maintain

Relatively speaking, a cluster of a given size might be perfectly capable of handling on the order of many thousands of resources . Push that an order of magnitude and the overhead of serving LIST calls - marshaling json from etcd to golang structs for apimachinery and back again for sending over the wire - could exhaust an apiserver’s memory allocation. And since the impact of resources is cumulative, any one application relying on lots of CRDs might not destabilize a cluster on its own but might well contribute to an unhealthy cluster when running alongside similarly CRD-heavy applications.

The key takeaway is that the kube api is best thought of as a specialized operational store rather than a general-purpose database. Anyone wanting to rely on CRDs at non-trivial scale would be well-advised to test carefully.