| I had a similar moment of existential career crisis as OP. I'm at the 20 year mark as well, in terms of developing software professionally. I've always felt like with new technology, I could grok at a high level how things worked. But LLMs like GPT seem like magic and I went through stages of initial astonishment -> despair realizing the potential impact it would have on the industry -> acceptance. While I still feel uncertainty and fear about the future, as others have echoed, I'm realizing it's a tool for developers to use. We can either choose to accept it and understand how to work with it, or reject it. The things GPT can generate amazes me, but I'm finding that it's a good starting point or reference to build on... not a final solution. It will generate things that are sometimes completely wrong, and it's your own experience and judgement that has to be used to determine that. GPT cannot do that... at least not yet. I think back 20 years ago and remember reading through a lot of physical books with occasional web searches landing on experts-exchange or random forums. Then came Stack Overflow and that became in invaluable tool, along with the ubiquity of free tutorials on YouTube and elsewhere. And now we have GPT which I'll ask if I really get stuck on something and it gives me new ideas to try. Perhaps in the near future, GPT is the tool that I'll use first. I found this podcast episode helpful for me to process what I’ve felt: [Lex Fridman Podcast #376 – Stephen Wolfram: ChatGPT and the Nature of Truth, Reality & Computation][1]. It's an unsettling feeling (in general) to feel like a foundation you've built and live on could potentially be made quickly irrelevant. I'd like to say I have words of wisdom to get rid of that feeling but I don't. What has helped me is to acknowledge these feelings as valid, and then try to get clarity in what direction to move. It's not the foundation itself that's important per se, but it's the skills you've acquired in building the foundation that's more important. [1]: https://lexfridman.com/stephen-wolfram-4/ |
This is exactly how I feel. I felt so out of my depth looking at the ML architectures and I could not make any sense of it. I thought perhaps, they get inspired by neuroscience for the layers etc.
But a friend who works on LLMs mentioned, the architecture of large ML models, are mostly experimentally discovered, not designed. If that's the case, that's even worse... it means an entire field which perhaps could replace me in future, doesn't even have a knowledge foundation for its breakthroughs, but just goes by experiment... I thought it was only the weights inside the model that evolves, not the architecture itself.
Which body of knowledge do I study then, and is it even engineering anymore? That's something else, which I am not sure if my programming experience applies.
The amount of GPU/Capital it takes to evolve such architectures, run such experiments has to be prohibitively expensive.