[bellowsgulch]

Certainly, but deconstructing the problem, none of the models seem to appreciate the staggering difference between a ball valve and a button release.

Of course, there's also no super soaker engineer jobs to take, so I'm sure training sophisticated models to do well in that area is not a high priority for any firms.

[cyanydeez]

I assume you prep them with a proper manual of smaller part combos so they atleast have some chase of stumbling into the correct configurations.

I wonder if a more generic lego-manual like task would be more representative. It kind seems like you're testing for AGI.

[bellowsgulch]

Yeah, I start with smaller tasks, like matching dimensional parts. Tasks like these you can't one shot, or they end up producing diagrams of plausible looking Super Soakers, but maybe ones that look more like an enthusiast making their own designs from scratch. Fair approach! But replication is different.

For me, it's not knowing whether or not it understands there's a big difference between a ball valve and a button release, or that once you start talking about depressing mechanisms for pressure release, you're activating some sort of signals that are too close to triggers (which, is what you want, after all!) and "triggers" are embedded with a very short distance to "guns" in any well-trained model.