Ian Mulvany

July 20, 2023

LLMs evolve, they are not predictable, and that's challenging for product development

This tweet thread is worth looking at:

What this tells us, and what is worth paying attention to, is that when building on top of LLMs that are outside of your control, the fine-tuning, or model, can move underneath you. If you have strategies for making the LLM work for you through prompt engineering, then your strategy could stop working. Suddenly. 

What would be great is if LLMs made available a clear indication of when key components or the model or the fine tuning change, but it's not in our gift to make that happen. 

So we need to:

- continue how to think about running local version of models. 
- build observability, and be ready to change our strategies

About Ian Mulvany

Hi, I'm Ian - I work on academic publishing systems. You can find out more about me at mulvany.net. I'm always interested in engaging with folk on these topics, if you have made your way here don't hesitate to reach out if there is anything you want to share, discuss, or ask for help with!