Yeah, that's absurdly scary. That's a huge social engineering/social control risk of there's a non trivial number of people using it like that...I mean imagine influencing the LLM to subtly pump and dump assets.
Yeah I mean I just asked it how to make a million bucks off of memecoins and it gave me an extensive list of memecoin investing tips to make sure I didn't end up as a bag holder.
The only thing I'd trust it for would be explanations about well documented concepts and strategies, but even that is better to get from a well respected Youtube channel or financial advice outlet.
I suspect a decent open source llm would have a lot more transparency than the current methods of coagulating collective information