[locuscoeruleus]

I don't know what type of ML you do, but my experience is that it is hard to pin point poor model evaluation as being the reason for poor performance in production (especially if you have an interest in protecting your ego). It can be hard enough to figure out if you even have poor performance in production as you often don't know how well you could/should be doing. I have recently started thinking about what part of the performance issues I have are due to poor evaluation vs under-specification. I'm not sure how I'm supposed to tease the two apart.