Cross posting this blog - https://towardsdatascience.com/classifying-source-code-using-llms-what-and-how-f04c7dbcba9b
It’s chock full of great advice on using LLMs. The task they worked on was determining if some given code was malicious.
Some key advice:
- running an LLM is expensive, check if another method might work for you.
- LLMs can often pay more attention to instructions at the start and end of an input (they phase out on the content in the middle).
- be consistent in the terms you use to prompt the LLM.
- for classification tasks you can ask both for the classification, and the reason why the LLM applied the classification. Helpful for debugging.
- you can ask the LLM to take on different kinds of personas and this gives you coverage of different points of view. These can be combined with some kind of consensus algorithm to get better results.
- remember, another more powerful LLM is likely to come along at some point.
The whole post is worth reading.
It’s chock full of great advice on using LLMs. The task they worked on was determining if some given code was malicious.
Some key advice:
- running an LLM is expensive, check if another method might work for you.
- LLMs can often pay more attention to instructions at the start and end of an input (they phase out on the content in the middle).
- be consistent in the terms you use to prompt the LLM.
- for classification tasks you can ask both for the classification, and the reason why the LLM applied the classification. Helpful for debugging.
- you can ask the LLM to take on different kinds of personas and this gives you coverage of different points of view. These can be combined with some kind of consensus algorithm to get better results.
- remember, another more powerful LLM is likely to come along at some point.
The whole post is worth reading.