Recommended Course: AI Evals For Engineers & PMs by Hamel and Shreya
A month ago, I noticed- AI products looked great in demos. They worked fine in toy examples.
But under real pressure? They quietly cracked.
No stack traces. No crashes. Just wrong answers delivered with confidence. So I dove into an AI evals course. What started as "let's understand how to judge AI systems" became something bigger:
A month ago, I noticed- AI products looked great in demos. They worked fine in toy examples.
But under real pressure? They quietly cracked.
No stack traces. No crashes. Just wrong answers delivered with confidence. So I dove into an AI evals course. What started as "let's understand how to judge AI systems" became something bigger:
Every durable AI product runs on one thing—a ruthless evaluation engine. And most teams are flying without one.
LLMs as Static Judges
The first red flag: most teams treat their LLM judge like a frozen oracle.
They pick GPT-4 or Claude, slap on a rubric, and ship. No continual calibration. No drift checks. No feedback loop to see if the judge itself is failing in production.
Over time:
- The judge's behavior shifts (model updates, prompt changes).
- Product usage changes.
- User expectations evolve.
But the eval setup stays stuck.
This creates a double failure:
- Misaligned judge → wrong product decisions. You think you're improving because scores go up, but your judge is drifting.
- Silent failures in production. You don't see degradation until a customer escalates.
An AI product without living evals is like a bank without risk management. Things seem fine—right up until they aren't.
When Data Becomes the Enemy: most AI teams don't have a model problem. They have a data problem.Two villains kept appearing:
Data drift. The world changes, new edge cases appear. Teams have no shared definition of what "drift" means, so they can't detect it.
Data leakage and broken splits. Test data bleeds into train. Bugs creep into pipelines. The system looks great because it accidentally saw the exam beforehand.Most evaluation failures are engineering failures—not AI failures. Wrong splits. Leaky pipelines. Mis-specified metrics.
Eval code nobody treats like production code.Evals should be treated like unit tests and regression tests, not quarterly analytics reports.If bugs in your eval pipeline survive for weeks, you don't have evals—you have vibes.
For some products recall is more important than percision- for some others percision is more important than recall. One phrase kept echoing:
Data first, retrieval later- Most teams do the reverse. They pick a retrieval method, hack a pipeline, throw questions at it. Then wonder why evals feel messy.
Data drift. The world changes, new edge cases appear. Teams have no shared definition of what "drift" means, so they can't detect it.
Data leakage and broken splits. Test data bleeds into train. Bugs creep into pipelines. The system looks great because it accidentally saw the exam beforehand.Most evaluation failures are engineering failures—not AI failures. Wrong splits. Leaky pipelines. Mis-specified metrics.
Eval code nobody treats like production code.Evals should be treated like unit tests and regression tests, not quarterly analytics reports.If bugs in your eval pipeline survive for weeks, you don't have evals—you have vibes.
For some products recall is more important than percision- for some others percision is more important than recall. One phrase kept echoing:
Data first, retrieval later- Most teams do the reverse. They pick a retrieval method, hack a pipeline, throw questions at it. Then wonder why evals feel messy.
As I dove deeper in the course, I realized all of the exercise of an AI product is actually an AI Finetuning business- the better you are fine tuned for a specific task- the better augmentation the product gets.
Everyone's obsessed with Agentic RAG—LLMs looping with retrievers and tools. Yet most teams never measure the retriever separately from the LLM. They judge the answer and forget the ingredients. The retriever does a poor job. The LLM overcompensates with hallucination. The final answer "looks okay." Evals pass.
Everyone's obsessed with Agentic RAG—LLMs looping with retrievers and tools. Yet most teams never measure the retriever separately from the LLM. They judge the answer and forget the ingredients. The retriever does a poor job. The LLM overcompensates with hallucination. The final answer "looks okay." Evals pass.
Two dimensions teams ignore:
Taste and open-endedness. AI products aren't just judged on correctness—also on voice, verbosity, appropriate caution. These are eval dimensions too.
Beyond Software 2.0 regression tests. Regression-style evals are crucial, but AI needs more: qualitative sampling, adversarial tests, taste checks, "does this feel on-brand?" assessments.You need hard metrics for grounding and soft evals for product feel. Both live in your eval framework.
We had an eye-opening exercise: agentic applications with 12 tools. Two failure patterns emerged:
System prompts without tool reality. The prompt described tools the application didn't expose at runtime. The design looked brilliant on paper. In practice, the agent never had access. Evals here check strategy: Does the agent attempt the right tool? Does it fall back appropriately? Does prompt design match runtime reality? Structurally valid, semantically wrong. The agent produced perfect JSON with coherent arguments—while hallucinating facts. JSON validates. Chain-of-thought looks clean. Arguments are coherent. Content is fiction. The course pushed hard on detecting silent failures: agents that fail to call databases, tools never invoked due to prompt misalignment, systems that pass simple evals but crumble under realistic workflows. Evals stress-test not just the model, but the prompting strategy and the engineer's mental model. one takeaway overshadowed the rest:
Evals are not a checkbox. They are the skeleton of a serious AI product.
A misaligned LLM judge drives bad roadmaps, bad trade-offs, bad business decisions.A leaky eval pipeline convinces teams they're improving when they're drifting.But when evals are treated like unit tests, when retrieval is evaluated apart from generation, when agents are stress-tested for silent failures, when data is designed before retrieval is chosen—the product stops being a demo. It becomes a system.If AI is the new software, then evals are the new testing discipline.The teams that understand this early will quietly build the most resilient, trustworthy AI products of the next decade.
Finally we had a great session on custom interfaces- why build that- what is the future of GenAI interfaces etc.
All in all- great course with loads of knowledge of an ML engineer alongside testing engineer. Ultimately the course resonated deeply the sense:
everything in life is easy, except if you are tricked into that thing being hard
remember that all people who are really good at something have huge incentives to tell you how hard it is, as that is the only thing that makes them special
It was awesome to be a permission less apprentice.
Image courtesy: my fave artist- the butcher!