The contents of the most important library in the world are unreadable. Not because you lost your library card three years ago. They’re locked in ash and carbon. Nineteen hundred years ago, Mount Vesuvius erupted. We all know what it did to Pompeii, but what it did in Herculaneum is as important. The mud and ash spewing from the volcano buried one of the largest libraries in the ancient world. Thousands of scrolls, containing the history and wisdom of the ancient world were lost. Or so we thought until technology and AI opened a path to recovering the lost text. Right now, teams from all over the world are working to unlock the writing in these texts.
Rediscovering the ancient world is only one of the near-infinite uses of AI. The capabilities grow daily. Few people can keep up. But I found a couple who might be able to and tried to learn everything I could from them.
Daniel Gross and Nat Friedman are two of the most prolific investors in AI. Before they started investing, they were builders in the AI space. Gross built a company that he sold to Apple at age 23. He then ran Apple search and machine learning for several years. Nat Friedman was the CEO of GitHub, where he led the development of Copilot. They also happen to be sponsoring the competition to read the scrolls buried by Vesuvius. If anyone can help us understand what AI can do and where it is going, it is these two.
The duo are six-time repeat guests on Ben Thompson's podcast, Stratechery. He has done the hard work of grilling them on all things AI. I've spent the past few weeks bingeing and re-bingeing the episodes to learn all I can about AI. (The episodes are available on the paid plan. Paying for a monthly subscription to listen to them all for yourself is worth the cost.) You could write volumes of text (or record hours of audio) based on their thoughts. For now, I thought I'd share a few of their less non-obvious insights.
Opportunity in Low-Status Areas
Today, AI is characterized by flashy demos and seemingly impossible capabilities. Everyone wants to build the next ChatGPT. But many of the winners of this AI cycle will come from unsexy opportunities. Think of things like gathering data and cleaning it. It's difficult work, but every AI company needs data. If you can slog through the data prep work, and produce something of quality, you'll have a line of customers out the door.
Maybe data isn't your thing? Another low-status place to look is in creating benchmarks. Everyone wants to show off how well they score on benchmarks. And how they can outperform their competitors. But no one wants to spend the time creating the benchmarks. One of the most popular benchmarks is MMLU. It is used by all the big-name organizations. The not-so-secret thing about it, is that it was created by a student as part of his undergrad education. There is nothing wrong with it. However, there is plenty of space for the other benchmarks and evaluation tools. If you are good at prompting LLMs and finding ways to break them, you could be the person to innovate in this space.
"I think probably one way to look for arb in any industry is to ask what’s low status." - Nat Friedman
Great Design Will Win Out
The release of ChatGPT launched the chatbot and text interface revolution. Every app needed its own AI-powered bot. While a text-first experience worked well for ChatGPT, imitators found it was not a one-size-fits-all solution. Text is great because it is simple and flexible. But it is also dangerous because it makes it hard for users to understand the limits of the system. For every spark of discovery offered by these tools, there are 10x as many paths where users get lost. There is tons of room for design innovation in the AI space. The best tools will figure out how to blend great UIs with all the capabilities of LLMs and other models.
Great design in the AI space is not limited to UIs. Winners in the space will also incorporate great design and taste into their models. Mistral is a great example of someone doing this now. When talking about them, Friedman referenced their deep focus on data quality and cleanliness. Gross talked about them as having the chance to be the "Pixar" of model makers. Someone with a distinct voice and style. While Mistral is on that path now, there is room for plenty of other unique approaches to model design and style.
"The issue with text is one observation we always had from Apple is unlike a GUI, the customer does not understand the boundaries of the system. So unless... you have AGI and it’s smarter than a human, great. Up until that point, you need something that has this feature that the GUI has, which is amazing. The GUI only shows you buttons you can press on, it doesn’t have buttons that don’t work, usually." - Daniel Gross
We Need More Products
On the surface, it might sound crazy to say, but we need more AI products. Hundreds seem to launch each day, but there is so much room for more. The rate of innovation and breakthroughs in research is speeding up. This means that there are always new capabilities to build on. Keeping your ear to the ground through channels like arXiv, podcasts, and forums. And be ready to spring on the latest and greatest model developments.
And remember that not everything you build has to be the next ChatGPT. A lot can be discovered through building fun projects and "toys" for yourself. Gross and Friedman shared an example of a teleprompter they built to learn more about LLMs. I've built a daily newsletter and a text-to-speech tool. None of these are groundbreaking or innovative but they helped me learn a ton. And remember that many of the best products start out looking like toys. So that thing you build for yourself may turn into the next big thing.
"That said, I think even if the researchers stopped right here and they didn’t produce any more capabilities, it would take us something like five or ten years to digest just what GPT-4 can do and all the other state-of-the-art models can do, into products. There are so many variations and variants and workflows and user experiences that need to be invented and reinvented and permutations that need to be tried, and we’ve just started to scratch the surface" - Nat Friedman
One thread that connects everything here, is that the impossible will become possible. Back to those scrolls. It was recently announced that text from the first scroll has been read. Work is underway to scale the process. Eventually, all the scrolls will be read. The amount we know about the ancient world will greatly multiply. There are millions of other "scroll projects" out there, it's time to put AI to work and do the impossible.