[wcarss]

I don't disagree with you, but your example of nails and their cost reductions made me wonder whether we reached a meaningful limit in say, some fundamental material terms, or whether we just reached a limit in terms of return on investment.

Return on investment can be too low because the investment required is really high, but it can also be too low because the returns are just limited. If prices had dropped 90%, surely nails became even more ubiquitous, but at that stage there's only so much more money to dig out of the cost reduction hole. It feels plausible that there may have been ideas about more digging that could be done, but the reward just wasn't there in the market, especially versus just selling what worked.

I bring it up because the distinction in one specimen may speak to a larger trend: do new sigmoid developments tend to fail to materialize more often because of serious physical limits / lack of good ideas, or because of limitations to ROI? (Or, other things?)

In the arena of AI, the ROI on more intelligence/unit-cost seems pretty high right now. So, it seems like the difficulty of applying any potential innovations would have to be staggering for none to be pursued. Or, there'd have to just not be any good ideas to try.

Overall, I think there's ideas to try. So in my opinion, that shapes out to justify a bullish sentiment on sigmoids continuing to stack until the perceived potential gains from more intelligence/unit-cost somehow fall off.

Like I said, I don't disagree, we really don't know. But I feel it's a good bet that there's more coming.

[notahacker]

There are more ideas to try, but this doesn't necessarily mean they're better ones or that we're bound to come up with them. Training transformers on corpuses of human text has worked extremely well at enabling AIs to generate continuations which are consistent with human text including really useful human text like code, but the limit might be the human text rather than the transformer architecture...

The ROI for cornering the nail market seems like it could have been big. The ROI for making something significantly more efficient than a ICE would have been very high for most of the last century and technology that is better in many respects than ICEs does now exist, but it took us roughly a century to get there. The ROI for coming up with something that's better than the ~1% annual efficiency improvement on turbofans would be extremely high, but we don't know what that is (probably some sort of propfan, but that idea's over 50 years old...)