Why are people even allowed to dress up ChatGPT... an autoregressive transformer... as a faith-based holistic doctor? Anybody could see this going wrong from a hundred miles away.
A few days ago a family member insisted to be brought to the ER because Claude told him. Result: he was immediately hospitalised and received surgery the following day to remove a tumour that was compressing his spine. Claude was very clear that he had to act as quickly as possible to lower the chances of permanent damage.