Ian Mulvany

April 21, 2023

Some recent reads about how to think about large language models

I have been doing a lot of reading about large language models in the last few weeks. Here are some of the more thought provoking pieces that I’ve read. I’m pulling out quotes from these pieces, what I think is important about them, and some reflections on them. I don’t fully agree with everything in these posts, but they have all been helpful in evolving my own thinking about large language models.

The aim of this post is to summarise what others are saying about these models. I’ll followup with my own thoughts, and reflections on my own experiences, in another post.


If you choose to only read one thing, this is the most useful thing to read. This paper talks about what we know, and don’t know about these models, and what we can predict about how they are going to evolve. The most important point from this paper is that even the current abilities of the models that we have were not know ahead of time by the people that built them, and indeed some of those were not discovered until some time had passed in using them. This level of unpredictability is going to continue, even as the models become more powerful. That means that no matter how disruptive they might be now, our ability to predict how much more disruptive they will be in the future us quite low. 

The key points this paper makes are:

1 LLMs predictably get more capable with increasing investment, even without targeted innovation. 
2 Many important LLM behaviours emerge unpredictably as a byproduct of increasing investment. 
3 LLMs often appear to learn and use representations of the outside world. 
4 There are no reliable techniques for steering the behaviour of LLMs. 
5 Experts are not yet able to interpret the inner workings of LLMs. 
6 Human performance on a task isn’t an upper bound on LLM performance. 
7 LLMs need not express the values of their creators nor the values encoded in web text. 
8 Brief interactions with LLMs are often misleading. 

recent methods can dramatically reduce explicit bias and toxicity in models’ output, largely by exploiting the fact that models can often recognize these bad behaviors when asked (Dinan et al., 2019; Bai et al., 2022b; Ganguli et al., 2023). While these mitigations are unlikely to be entirely robust, the prevalence and prominence of these bad behaviors will likely wane as these techniques are refined.

though, these encouraging signs do not mean that we can reliably control these models, and the issues noted in Section 4 still apply

Ok, so the takeaway from this paper is that we have not reached the limits of what these models can do, we can’t predict what they will be able to do in the future but they are likely to become more powerful. What we think of today as their weaknesses and limits are not hard limits, but equally we don’t have any guarantees that we will be able to overcome those weaknesses, as future capability and limitations are so unpredictable.

What we can infer from this is that utility is not going to go down, so we need to learn how to work with these tools with the right level of caution right now, and continue to apply those approaches as these models evolve.


This is a really nice article giving a perspective on how to think about large language models. Think about them as resigning engines, not knowledge databases. The key takeaway for me from this piece is that it is going to be critically important to match these models with highly trusted sources of information, and not to depend on the text that is directly generated from the model. Thinking about them as “search engine” is the wrong pattern. They are tools that can act like they understand text, and can be directed to do lots of tasks on the basis of that understanding. The training data gives us a tool we can talk to, but what we need to do then is point that tool at the data we want to work with, and get that data into the best format for the tool to interact with. 

Here are some good quotes from this article: 

Even though our AI models were trained by reading the whole internet, that training mostly enhances their reasoning abilities not how much they know. And so, the performance of today’s AI models is constrained by their lack of knowledge.

This is crucial to understand because it predicts that advances in the usefulness of AI will come from advances in its ability to access the right knowledge at the right time—not just from advances in its reasoning powers.

So, what does this mean for the future? I think there are at least two interesting conclusions:
1 Knowledge databases are as important to AI progress as foundational models
2 People who organize, store, and catalog their own thinking and reading will have a leg up in an AI-driven world. They can make those resources available to the model and use it to enhance the intelligence and relevance of its responses.  

We’re already seeing newer alternatives springing up that wrap some  [business logic](https://en.wikipedia.org/wiki/Business_logic)  around the database layer to make it easier for AI developers to do common tasks. Some of these are developer libraries like Langchain and LlamaIndex. And some seem to be more fully featured developer tools like  [Metal](https://www.getmetal.io/?utm_source=every)  and  [Baseplate](https://www.baseplate.ai/?utm_source=every) . Just like Pinecone, are also likely to raise a lot of money or already have! AI’s advancement is a raindance that calls forth capital from Patagonia vest wearing angels.

Another post that is good on how we might think about these tools comes from Simon Willison, who is definitely worth following on this topic. He talks about them being a “calculator for words”


Using language models effectively is deceptively difficult So many of the challenges involving language models come down to this: they look much, much easier to use than they actually are. To get the most value out of them—and to avoid the many traps that they set for the unwary user—you need to spend time with them, and work to build an accurate mental model of how they work, what they are capable of and where they are most likely to go wrong.

I hope this “calculator for words” framing can help.

A flaw in this analogy: calculators are repeatable [Andy Baio](https://waxy.org/)  pointed out a flaw in this particular analogy: calculators always give you the same answer for a given input. Language models don’t—if you run the same prompt through a LLM several times you’ll get a slightly different reply every time. This is a very good point! You should definitely keep this in mind. All analogies are imperfect, but some are more imperfect that others.

OK, we have seen that the future of these models is going to continue to be powerful, that they provide an almost thinking capability in our interactions with our systems, what has Bill Gates got to say about this? Well he thinks it’s going to be pretty transformative:

The Age of AI has begun - Bill Gates

Although humans are still better than GPT at a lot of things, there are many jobs where these capabilities are not used much. For example, many of the tasks done by a person in sales (digital or phone), service, or document handling (like payables, accounting, or insurance claim disputes) require decision-making but not the ability to learn continuously. Corporations have training programs for these activities and in most cases, they have a lot of examples of good and bad work. Humans are trained using these data sets, and soon these data sets will also be used to train the AIs that will empower people to do this work more efficiently.

Company-wide agents will empower employees in new ways. An agent that understands a particular company will be available for its employees to consult directly and should be part of every meeting so it can answer questions. It can be told to be passive or encouraged to speak up if it has some insight. It will need access to the sales, support, finance, product schedules, and text related to the company. It should read news related to the industry the company is in. I believe that the result will be that employees will become more productive

Global health and education are two areas where there’s great need and not enough workers to meet those needs. These are areas where AI can help reduce inequity if it is properly targeted. These should be a key focus of AI work, so I will turn to them now.

First, we should try to balance fears about the downsides of AI—which are understandable and valid—with its ability to improve people’s lives. To make the most of this remarkable new technology, we’ll need to both guard against the risks and spread the benefits to as many people as possible.

Second, market forces won’t naturally produce AI products and services that help the poorest. The opposite is more likely. With reliable funding and the right policies, governments and philanthropy can ensure that AIs are used to reduce inequity. Just as the world needs its brightest people focused on its biggest problems, we will need to focus the world’s best AIs on its biggest problems. 

Finally, we should keep in mind that we’re only at the beginning of what AI can accomplish. Whatever limitations it has today will be gone before we know it.

I work on the tech side, and I’ve seen first hand how these tools can enhance and speed up the creation of software. Simon Wardley has one interesting perspective on this:


As a rule of thumb, around 95% of the code we need to build has already been written. Unfortunately, we often don’t have an effective way of finding that code that represents the same function as the code you were about to write. This is why IT shops endlessly build the same things over and over again.
I don’t see a long-term career in software anymore. Any dreams I had of earning decent money as a software engineer are slowly fading. Lex Fridman in his podcast said “if you’re anxious about GPT4 its probably because you’re a shitty programmer”. I mean, I’m not the smartest in the room but I have generated value with the software I’ve made. And was convinced that I’ll make decent money as long as put in the work. I’m just not that sure anymore.

I don’t fully agree with that point, but it depends on where we land. We are going to go through a hard transition point over the next two to three years. In one scenario these tools tip the economics of how we create software, but don’t change how we create software. Another perspective is that these tools lead to a Gutenberg moment for software creation and unlock a huge pent-up demand for more software to be created. I rather think this latter scenario is likely. Geoffrey Litt’s post is really worth reading on this:


In general, GPT-4 feels like a junior developer who is very fast at typing and knows about a lot of libraries, but is careless and easily confused.


We have now looked a little at the utility of these tools, and how to think about them from that perspective, but how should we think about how they think? This post asks us to think about the question of whether these models can be said to think!! It’s a really nice read, and cuts to some of the history of AI research. My takeaway from this is that whether these models think or not, our interactions with them are so persuasive that they give sufficient feeling to us of being able to think, that maybe the question starts to become less important! 

I’d not encounter the idea of the “manifest image” of the world before, but I really like it, and I do think that these tools encode a manifest image of the world, and I’ll expand on that a bit more in another post. 

Here are some key quotes from this article: 

Wilfrid Sellars, a philosopher of language, termed this world of human things the “manifest image” to contrast it against the world of atoms bumping around in the void (the “scientific image”), which is better understood.

Knowledge: a database of facts; language: a series of rules; meaning: logical propositions. All this is clumsy and excessive literalism to Skinner, without any scientific merit. In short, he thinks the bridge between the manifest image and the scientific image is not sturdy at all, and talk of mental software can only ever be a metaphor.

Indeed, anyone who has tried and failed to pin down something as simple as what a government or a fight or a game is in precise terms will know that things in the manifest image are strangely elusive, even those as seemingly concrete and exact as words or physical objects. 

It is able not just to respond to questions but to respond in the way you’d expect if it did indeed understand what was being asked. And, to take the viewpoint of Ryle, it genuinely does understand — perhaps not as adroitly as a person, but with exactly the kind of true intelligence we attribute to one.

If we continue down this path, we have to start to look at questions around AGI, and what these models mean for that. I have very specific thoughts about AGI that I’m not going to go into here, but the following very long post starts to dig into this question:


It’s really long, and I don’t recommend reading it. My high high level summary is that what this long post is saying is that: 

* Reinforcement learning has the concept of agents, and these have been a focus of AI safety 
* Large language models are not RL, but better described as self supervised predictive training. 
* However they act like agents better than the best RL agents that have been made to date. 
* This poses a slight existential question to the field of RL and AGI and AI alignment 
* The author offers the idea that these models can be described as “Simulators”, and with that be brought into the fold of thinking about AI alignment. 

Here are some quotes from the piece: 

But this behavior would be very unexpected in GPT, whose training doesn’t incentivize instrumental behavior that optimizes prediction accuracy! GPT does not generate rollouts during training. Its output is never sampled to yield “actions” whose consequences are evaluated, so there is no reason to expect that GPT will form preferences over the consequences of its output related to the text prediction objective._14_

What it does is more like animation than divination, executing the dynamical laws of its rendering engine to recreate the flows of history found in its training data

it was not optimized to be correct but rather realistic, and being realistic means predicting humans faithfully even when they are likely to be wrong.

GPT is behavior cloning. But it is the behavior of a universe that is cloned, not of a single demonstrator, and the result isn’t a static copy of the universe, but a compression of the universe into a generative rule. This resulting policy is capable of animating anything that evolves according to that rule: a far larger set than the sampled trajectories included in the training data, just as there are many more possible configurations that evolve according to our laws of physics than instantiated in our particular time and place and Everett branch.

There is a trend in AGI research to hope that an “AGI” will have almost limitless power, and will accelerate human capability, and solve all of our problems for us. For me my read on these LLMs is that they will not give you truth, will not see into the future. They are a new and incredibly powerful tool that we have created which we may work with and use to build on top of. They will alas not solve cancer for us, will not solve the climate crisis for us, but it may aid us in helping us solve these problems ourselves. We cannot therefore abscond from our own responsibility to be agents and to take ownership of the consequences of our actions.

A post inspired by that former one attempts to define a “Semiotic Physics”


I have a lot of problems with this framing, but I think it is going to be influential, and there are some ideas in the post that I find useful. The post hopes to create an analytical framework that can be used to interpret and analysis large language models by using the tools of dynamical physics on how the models generate or simulate text. 

The term “semiotic physics” here refers to the study of the fundamental forces and laws that govern the behavior of signs and symbols. Similar to how the study of physics helps us understand and make use of the laws that govern the physical universe, semiotic physics studies the fundamental forces that govern the symbolic universe of GPT, a universe that reflects and intersects with the universe of our own cognition. We transfer concepts from dynamical systems theory, such as attractors and basins of attraction, to the semiotic universe and spell out examples and implications of the proposed perspective.

In particular the post talks about “attractor sequences” I can kind of get along with the idea of attractor sequences where the writer describes them as ones in which 

small changes in the initial conditions do not lead to substantially different continuations.

A language model might create one particular trajectory rather than another because of the shape of the attractor landscape.

While I think the main idea behind this post is fundamentally flawed, I do like thinking about these systems from a landscape point of view. 

OK that’s a wide round up of posts that talk about these models, how to think about them, what kinds of effects they might have in our world in the coming years. 

About Ian Mulvany

Hi, I'm Ian - I work on academic publishing systems. You can find out more about me at mulvany.net. I'm always interested in engaging with folk on these topics, if you have made your way here don't hesitate to reach out if there is anything you want to share, discuss, or ask for help with!