[catlover76]

Is there a "cutting edge"? The space seems pretty pseudo-sciency

I'm reading some papers on arxiv right now, and trying to implement them in our codebase at work. Those papers usually involve doing some common sense thing and measuring the results. Anyone could have come up with it, but they did the data science and showed some evidence it worked.

If there is a better way, I would love to know lol

[danielmarkbruce]

two cents: any situation involving billions/trillions of variables looks pretty pseudo-sciency because you can't reduce it down or isolate components very well. People can do studies, add things and take things out, and sort of hint at things and explain things sort of. It is what it is.

[janalsncm]

Real science is reproducible and provides testable, falsifiable hypotheses.

In practice, some models (ChatGPT in particular) are not deterministic. This makes reproducing things harder. Not impossible, but harder.

[muzani]

I'd expect science in this field to look more like economics than physics. You're probably not looking at lab results or anything, but at experiments already done "on the field" with controlled variables. Like an economist assessing minimum wage laws might compare employment in two neighbouring states over a period of time.

[danielmarkbruce]

Certainly not physics - but shouldn't it be more like biology than economics? You can do experiments in biology (and with LLMs), the problem is you can't quite get to a cause-effect. The situation is just so complex. In both cases we are desperate to get an "explanation" that fits in a human brain and makes us feel like we understand, but it's out of reach.

[muzani]

Yeah, biology does seem like a better comparison, especially with "strands" of AI performing differently than what they used to after patches.

[BOOSTERHIDROGEN]

It's kind of dream, is hard to to deterministically where the path is.