[chii]

> How do you expect it to produce useful programs

why not ask it to produce said program, and then evaluate it on that output, rather than proxy it via asking a logic puzzle?

It's like doing a coding interview when hiring an employee with these brain teaser puzzles, and when they fail you disqualify them, rather than asking them to do a real task that would be something they'd encounter on the job.