tags: #llms #ui #local-models #tool-review
After following this post - https://simonwillison.net/2024/Dec/27/open-webui/#atom-everything) from Simon Willison, I installed Open WebUI - https://github.com/open-webui/open-webui) using this command:
uvx —python 3.11 open-webui serve
(As an aside, uv has just totally taken away all of the pain for me in running Python applications and managing their dependencies.)
It opened a web browser running this tool locally with the gemma2:latest. I’m running on a 16GB Apple Air M1 laptop. It defaults to using sentence-transformers/all-MiniLM-L6-v2.
I created a local knowledge base (you create a named workspace, and you can drop files onto that workspace, and it creates a local RAG interface for you).
I dropped docx files and pdf files containing the text of my CTO updates to my department over the 2024.
With this setup, I found:
- It’s slow.
- The responses are not great in terms of perceived intelligence.
- Figuring out how to get this to work was not immediately obvious, but not too hard either (if that makes any sense?).
- This setup didn’t do a good job of extracting information from PDFs, and I got a null result for what should have been an obvious query. When I converted that one PDF to txt, I got more or less the answer that I was expecting.
- It makes it easy to set up different knowledge bases and to pull them into a chat query - I think this is a real novelty in terms of interaction design.
- The whole app just worked, and it is trying to cover a lot of different use cases - chat, RAG, tools, pipelines, managing models locally, managing interactions with hosted LLMs.
- There are a lot of systems settings, from external search engines to default prompt settings to API keys to Auth, this is a big application, and I think it could struggle under the weight of what it is trying to support.
- I use a number of different models on a regular basis, something like this could become a place to bring those different interactions into one pane of glass.
At the moment, given the hardware I’m running, I don’t see myself spending any more time with this tool, but if I get better hardware or if local models improve in efficiency, then this is a really compelling tool for managing interactions with models, especially being able to create different sets of knowledge bases.
Tags from OpenAI:
software review, technology, tech tools, machine learning, model management, knowledge bases, python applications, data science - 004.65