| How about this distinction. There are some things where the "lean startup" applies. For instance if you made an Ebay or AirBNB or Reddit or Substack kind of a site you could get a rough prototype running quickly. The software is maybe 20% of the effort, but 80% of the effort is in business development (recruiting people to kickstart the market) Some products on the other hand take years of development and may or may not work. Golden Rice, Falcon 9, the LLVM-based C compiler are all examples. I worked on a system which was uncomfortably in between these models. On one hand were developing LLM-like systems before LLMs as we know them were available so we could have spent a few years on development. However we could sell projects to customers which caused us to zig and zag a lot to meet their needs. That was a good thing because we learned a lot about what was possible (it contributes to the research) but we wasted a lot of time with spoiled work in progress, etc. In our case I think the investors believed in our vision but were skeptical about our ability to execute (rightly so: I couldn't even get the data scientists to use a standard version of Python even though that was what I got hired for) and would bring in consultants that were often counterproductive (zigging and zagging to meet customer needs meets customer needs but spending weeks writing up OKRs is busywork.) I believed in the story more than anyone but the C-levels because I had been working on a similar thing on my own account. I'd tell people when it was tough that if our product was sufficiently realized it would be worth it for one of our customers to buy us. I thought it would be a Big 5 accounting firm or an airplane manufacturer but it turned out to be a major consumer brand. That's honorable and probably paid the VCs back what they put in, but had we had the funding to develop technology for a few years and enough contact with applications to know what direction to go in, switched to transformer models the moment BERT came out, and if we were more disciplined about our streaming engine so it always gave the right answers (wrote down what the algebra was for it rather than argue about whether we should call it an algebra) we could have changed the world. |
Other actual tech examples seem to fall into 1 of 2 camps: obvious but hard to do (better search engine, better rockets, electric cars, etc), or cool but non-obvious customer end uses (maybe LLMs, VR/AR, curved or flexible high def screens, etc). The latter category has more risk but probably lower hanging fruit to get started, because the market needs are less obvious.
In your example, do you think the customer focus lead to pre-mature optimization and kind of tunnel-visioned the team away from further LLM development? That's another type of trade-off that's probably impossible to predict at the time. I mean who doesn't want customers.
I'm not entirely surprised that OpenAI was able to achieve so much given their structure - they had the mandate of a trendy new research lab, top talent, with 100M+ funding and no need to cater to any early customers. Seems like a great (though typically impractical) way to build big new things. Then they had the right top-level guidance when the tech was getting ready, to pivot and raise more money (unlike XEROX PARC for example).