I mean I agree with you on this, but the idea that you can gauge a person's skillset through conversation (gasp!) and talking about your craft and details about past projects, etc seems to be totally lost on people today.
I mean, it goes both ways. I feel that I am a fairly proficient engineer (built something with 10M+ downloads) but for a long period of my life I would do fine in every part of the interview except the part where you just chit chat a bit. I'd try to tell interviewers about myself and the smiles would just fall off their faces.
It took me a long time to understand that these sorts of conversations had their own "rules" to them, which I had to follow if I wanted to do well. These rules, of course, have virtually nothing to do with programming aptitude or ability, and seem to me, somewhat cynically, to be another way by which interviewers can allow their own biases to enter into the interview process. For instance, one time I got rejected because I seemed "too excited" about my personal side projects; it was deemed that I wouldn't be as excited about the work that I'd do at the company. Of course this is nonsense; I'm now happily employed and pretty excited about my work. I have plenty other examples of me saying reasonable things in interviews and being rejected for that reason.
There's really no silver bullet here. Getting rejected is always going to piss people off.
See, I take a bit of a different view in the example you've given. Like, if I got rejected because I seemed "too excited about my personal side projects" I'd come away from that thinking "if that's really their take away, I'm kinda glad I don't work with them!"
You're right that the conversational interviews (just like any social gathering, really) have their own rules. But I think the most important thing you can do during those interviews is to just be yourself. After all, you want them to get to know you, just as you want to get to know them, right? How else can you each be sure that you're a good fit for each other? If they reject you for something you said that is true but that they just didn't like (e.g. a difference of opinion on something), or they nitpicked some little thing you said even though the rest of the conversation went smoothly ... well, in my opinion, you're better off.
Normally I'd agree, but I got passed up by a large number of fairly reasonable-seeming companies for arbitrary reasons like this. If it's just one or two, sure, maybe I'm better off. But after that it starts to have a real impact; it becomes harder to negotiate, maybe one of those companies would have been just fine anyways, etc...
It took me a long time to understand that these sorts of conversations had their own "rules" to them, which I had to follow if I wanted to do well. These rules, of course, have virtually nothing to do with programming aptitude or ability, and seem to me, somewhat cynically, to be another way by which interviewers can allow their own biases to enter into the interview process. For instance, one time I got rejected because I seemed "too excited" about my personal side projects; it was deemed that I wouldn't be as excited about the work that I'd do at the company. Of course this is nonsense; I'm now happily employed and pretty excited about my work. I have plenty other examples of me saying reasonable things in interviews and being rejected for that reason.
There's really no silver bullet here. Getting rejected is always going to piss people off.