We use AI inside CoachBase to turn coaching-call transcripts into summaries, notes, and action items. The goal of this is to allow coaches to be more fully present on the call, not needing to worry about taking notes or specific timestamps, and not requiring them to have another subscription to a third-party note-taking app like Granola. 

As AI models get smarter, faster, and cheaper, I wanted to check in on how our original choice of model stacked up, and how things have changed in the last few months.

So buckle up, buttercups. It's about to get a little nerdy.

When a model reads a meeting transcript and produces a summary, the result is remarkably easy to accept at face value. The writing sounds plausible. The bullet points resemble the conversation. Nothing has caught fire. You could nod approvingly and move on with your life. But that isn't much of a quality standard.

I wanted to know whether another model could do the job better for roughly the same cost. There are plenty of public leaderboards, but none of them answer our particular question: can a given model listen to a coaching conversation, preserve key moments, identify the commitments that were really made, and avoid confidently hallucinating things or inventing goals for a client?

So in order to answer that question, I built a small evaluator, which is essentially a repeatable test for AI output. You give several models the same input, compare each response against criteria you decided ahead of time, and then calculate a score based on their output. It's less like asking which model writes the nicest answers and more like giving each model the same exam with the same answer key.

It turns out that scoping that exam was harder than building it. I could have tried to measure every possible quality of a transcript summary, but the resulting score would have meant very little. Instead, I started with the decisions CoachBase makes from a transcript: What happened? What did the client commit to? Is this a new task or an update to an existing one? Was a deadline clearly stated? Did the model invent anything?

I then separated the parts that a computer can check from the parts that still needed human judgment. The evaluator can compare actions, deadlines, and task references consistently, but a person still has to decide whether the summary is faithful, useful, and worth reading before the next coaching session. That boundary became the scope of the evaluator.

The first version compared a model's output against expected phrases and fields. It checked whether the summary mentioned the right concepts, whether the expected action items appeared, and whether task details such as deadlines matched.

To my surprise, one model scored 38.9. My initial reaction was that a score of only 38.9 sounded dreadful. The output itself seemed pretty good, though, which left two possibilities: either I was being fooled that the output (which was pretty decent) wasn't actually very good, or the test was grading the wrong things.

It was mostly the test.

The evaluator expected particular phrases, while models quite reasonably described the same commitment in different words. For example, "Interview three customers about the positioning" and "speak with current clients to refine the message" can mean the same thing to a person while sharing very little literal text. So it turns out I had built a spelling bee for a job that required comprehension. Neat.

The takeaway for me was that the initial "low" score of 38.9 was useful. It forced me to test the test.

So, back to the drawing board. The revised evaluator still starts with the transcript. Then, for every potential action item, we determine whether it was required, optional, or unsupported by the conversation. That last one is key, because a polished fabrication is still a fabrication, even if it arrives with impeccable grammar.

I also separated ordinary coaching calls from things like business meetings. Our platform can be used for all kinds of contexts, and not every call that takes place is specifically a coaching call. A one-on-one coaching conversation and a six-person status call require different contexts. Lumping them together hides that difference, and causes the AI model to repeatedly miss. 

Our evaluator's final score now weights 40 percent to a blind human review of accuracy, usefulness, and action-item quality. The automated portion gives 30 percent to finding the right commitments without adding extras, 15 percent to avoiding specifically tempting fabrications, 10 percent to handling existing tasks correctly, and 5 percent to deadlines the transcript states clearly enough to calculate.

When running the evaluator, we hide the model names during the human review step. That way a reviewer can be as fair as possible, without their opinions about OpenAI, Google, Anthropic (or whichever company annoyed the internet most recently) skewing their judgement.

I also made the action matching semantic, which is a fancy way of saying the evaluator compares meanings instead of trying to compare against the same exact phrase. It still checks specific details, like whether the model created a new task or updated an existing one correctly.

To make the comparison fair, I took the exact same model outputs and ran them through both versions of the evaluator. Nothing about the models’ answers changed, but the grading did.

Across the two coaching runs, the scores moved like this:

Gemini 3.1 Flash Lite
100.0 before, 86.4 after

Grok 4.3
55.6 before, 72.7 after

Claude Haiku 4.5
100.0 before, 54.5 after

DeepSeek V3.2
95.8 before, 50.0 after

GPT-4o mini
63.9 before, 47.8 after

Mistral Small 2603
97.2 before, 27.3 after

Hawk-eyed readers will notice that the new evaluator completely rearranged the field. The old phrase matcher loved Claude, DeepSeek, and Mistral. The semantic version moved Grok from last place to second and became much harsher on almost everyone else. It was the comparison I wanted, but it was not the conclusion I expected.

When I went back and manually inspected the outputs, I noticed that the revised evaluator was still missing some obvious matches. For example, Mistral returned “Conduct customer interviews to refine positioning statement” for an expected action labelled “Interview three customers about the positioning.” A human person would recognize those as the same commitment without much trouble, but the matcher did not, and thus Mistral’s score collapsed from 97.2 to 27.3.

In short, the original evaluator was too generous when a model repeated the expected words. The revised evaluator can be too harsh when its embedding model or similarity threshold fails to recognize when something has been paraphrases.

So... I can't honestly say the revised evaluator is more accurate (yet). It does measure more of the behavior we care about, including hallucinated commitments and the like, but its semantic matching still needs calibration against human judgment.

That's why I made human review part of the evaluation process rather than a ceremonial check-of-the-box. Before trusting either scoreboard, we need people to grade the same outputs blindly, which then enables us to see which automatic score agrees with them more often.

The most interesting part of this process for me wasn't the model ranking. It was how often a reasonable-looking scoring rubric produced an unreasonable result.

At any point, a model could improve its score by simply emitting more junk, because the penalty for fabrication was divided by the number of outputs. Outputting an optional action could steal the match from a required action (not good). Similarly, not updating an existing task could score better than updating it with the wrong verb (also not good).

Each rule looked defensible on its own, but together, they occasionally created nonsense. Sweet. 

The fix was to keep tracing concrete examples to answer questions like: If Model A makes a worse mistake than Model B, can the formula ever rank A higher? If simply outputting nothing avoids a penalty, have we then rewarded omission? If an action is optional, can including it make the score worse?

The evaluator improved when I started treating it like a product rather than a spreadsheet with an air of authority.

Based on these results alone, I'm not planning switch CoachBase's model just yet. We have a baseline, and several models have earned their place in the contender's circle. There's still more evaluating to be done. 

Gemini led the pack with a score of 93.2 but costs almost triple our current model. Grok scored 86.3 but cost roughly twelve times as much as our current model. Those numbers helped identify finalists, but they're not necessarily strong enough to justify switching the production model.

I already know that the next run needs a larger set of redacted real coaching calls, as well as repeated samples from each model and blind human ratings. I care less about whether a model wins by a few points than whether it consistently provides outputs that are genuinely useful and gives our coaches something valuable before their next sessions.

That's a slower answer than glancing at a leaderboard, but it's an answer I can trust. And if the next model scores 38.9, at least I'll know to interrogate both the student and the exam.