I was trying to coerce gpt-4o to talk about the gcd and lcm in terms of sets of the prime factors where the product is the union of the sets, gcd is the intersection, and the lcm is the union less the intersection and it kept telling me I was incorrect and being "non standard".
To an extent; they can give hints and suggest directions, but you need to treat them as an unreliable narrator: think of them as entities that can help or deceive you at random.
That being said, we are researching tailored LLMs and other architectures to assist mathematical research that are more geared towards accuracy at the expense of freedom ("imagination"). The Lean FRO has some related information and links.
It has a long, long way to go.