| Marcus has a good point when he says that the scaling will only benefit in a logarithmic way, not in a linear way. I watched the Sora video as well and in the Q&A session, they admit that they will need a lot of data. Marcus' contention is that there isn't enough data on the internet and synthetic data will just result in errors. Where Marcus gets it wrong is that he defines "right" as producing an algorithm that an AI will follow to get a deterministic result. So, every time Sora 2 (Gary's Version) produces a video of a glass shattering, it is the same shatter pattern being produced; and it must be a precise duplicate of a glass shattering in nature. That's Marcus' win situation, which Sora will unlikely ever reach. Maybe transformer-based AI will never be capable of perfectly simulating reality in order to unlock its secrets. It seems to me that, according to the transformer-denialists, in order to create an AI that understands reality, we must fully understand reality first and then program the AI with that understanding. In my mind, I imagine neural networks as drift-car drivers (I think of them as Ken_Block, Paul Walker would also be acceptable). Sure, your average drift-car driver has no idea how to solve a three-body equation algorithmically, but a great drift car driver can maneuver four spinning tires in a state of critical oversteer around a race track curve without the use of a calculator and get it right most every time. And yes, race-car drivers have short lifespans, it's true. That's what terrifies Marcus so much about neural networks as well, and why he is so adamant that we listen to him when he says that there are dangerous curves ahead. I personally would rather live in a world where there are Ken Blocks and Paul Walkers, and that's how I live my life (not in auto-racing, though) but I understand why that frightens people. |