I was working on a similar idea for a bit. I wanted to support a tree-like structure for branching conversation but never figured out a good UI for it. I really like your approach!
Thanks! I think it still has a ways to go, my bigger concern now is how to make the non visual experience feel/sound great. Auto playing, notifying the transition to a new comment, etc.