Why LLMs Can't Really Build Software
Ironically, the model of software development laid out here may end up helping LLMs build software better:
- Build a mental model of the requirements
- Write code that (hopefully?!) does that
- Build a mental model of what the code actually does
- Identify the differences, and update the code (or the requirements).
In particular, Kiro from Amazon is taking steps to apply this approach to LLM based software development. I see the future of software LLMs as less Bolt.new or Codex, where you one shot an app, and more Kiro, where you iteratively step through features one at a time.
Many of the LLMs out there and LLM coding approaches (think vibe coding) are effectively sped up waterfall development. The trouble with waterfall is that we do not have perfect knowledge about what our customers want. We never will have that perfect knowledge because prediction is hard and humans are no good at it^1.
We use agile and "lean" because we need to test software bit by bit to figure out which bits work and which bits don't. The other day I heard about someone using "daily" sprints to get software out faster than ever. There are a few edge cases where this can work (adding enterprise support for a consumer grade app being one). But for the most part, it just means you are building software you don't know people want.
For LLMs to build software well, they ironically need to adopt more of the iterative development principles from software development.
That said, there is progress being made here. Hallucinations, according to OpenAI, are "solved." Honestly, I need to read that paper alongside Hallucination is Inevitable: An Innate Limitation of Large Language Models to see who is bullshitting.
The space is moving so fast it is hard to keep up! But for now, one thing I can say with certainty is that "vibe coding" is another word for waterfall and we've proven that doesn't work (outside of certain specialized cases).
1. For more on this, check out Superforecasting and The Black Swan