[sfink]

The cat's out of the bag, if someone stops then someone else would start.

I would entreat people to consider the net effect of anything they create. Let it at least sway your decisions somewhat. It probably won't be enough to not do it, but I think of it more as the ratio between net positive :: net negative, and paying attention to that ratio should help swing it at least somewhat -- certainly more than giving up and ignoring the benefits :: harms.

[martin-t]

The whole idea of developing AGI (even if LLMs are probably the wrong approach) is so strange when you think about it.

The smartest people in the world are working very hard in order to make themselves completely redundant and cheaply replaceable. If they succeed, they will turn their main skill and defining characteristic into a meaningless curiosity.

And life will not be better, the manual work of today will still need to be done and robots are not up to the task. Even with better programming for cheap, the hardware (and I mean even simply the metal) is too expensive compared to human hands.

[munksbeer]

Do you mean you think we'd all be superfluous for any thinking jobs, and would only be useful for manual labour?

Is there a chance this doom scenario is wrong and we end up with something approaching the idealists benign AGI post scarcity world?

[martin-t]

The trend is that once ML models are able to compete with humans on a task, they perform is much faster and cheaply (although very often with a much higher error rate for now).

IMO thinking will be useful to some extent because the bottleneck will be getting instructions from the models into people to perform them. So if the model has to spend less time communicating with you (limited by human cognition, not the model), you will be more productive.

All work that can be done remotely can also be done by AI. Currently there are limitations like you having more context from communicating with your colleagues.

I suspect at some point, companies will try to mandate that all work-related communication (even face to face) has to be made accessible to AI. They will no doubt try to go two steps forward and then back off one step as a compromise so it will be a local model in the name of privacy or something like that.

At that point, AI will have the same work-related knowledge you have. In fact it will be able to cross reference knowledge from all workers in the company. So why would you work from home if it can do everything you could but faster and cheaper?

[munksbeer]

What is your conclusion? Will it be a benign benefit, bringing us closer to post-scarcity, or will we be doomed to a dystopian existence? Can you elaborate on how you see it?

[martin-t]

Post-scarcity can work if everyone can say "here are me needs, they are now fulfilled, I have a happy and satisfying life". The issue is some people have a deep need to have more than others so they will always strive to have more, including power. AI, if it's ever created will be just another tool they'll use to get it.