LLM as a Judge: How to Write a Judge Prompt for SQL and Chart Agents
Learn to write an LLM-as-a-judge prompt that checks another agent’s SQL, data and charts: what evidence to give, which checks to ask, and how to test the judge.
Lesson summary
Give the judge the evidence, ask it narrow questions, make it reason before it rules, and check it against people.
- A vague judge grades what it can see, and a wrong chart looks as tidy as a right one.
- Hand the judge the question, the SQL, the schema notes and sample rows, not just the output.
- Split “good” into pass/fail checks tied to the request, and let code do the exact ones.
- Lay the prompt out in a fixed order and make the judge reason before its verdict.
- A judge prompt is a classifier: fixed labels, skeptical, and blind to the agent’s self-praise.
- Test the judge against people’s labels and read every disagreement.
Why did the one-line judge give a wrong chart 9 out of 10?
It only saw the chart and was never told what good means, so it graded how tidy the chart looked.
The judge has the SQL but still passes the double-counted chart. What is missing?
Something that shows an order has many items: the schema notes or sample rows where the same order repeats.
Do the bar heights match the query result? Who should check that?
Code. It has one exact answer, so a comparison in code is cheaper and never wrong. Save the judge for questions that need judgement.
Why write the reasoning before the verdict?
The model writes one word at a time and never goes back, so reasoning written after the verdict cannot change it.
The agent ends with “Verified: totals match.” Should the judge see that?
No. It is a claim, not evidence, and judges lean toward confident text. Give the judge the artifacts instead.
Your judge agrees with people 78% of the time. Is it good?
Not necessarily. Look at how many of the bad outputs it catches. The one-line judge agreed 31 of 40 times but caught only 4 of 12 failures.
Stage 1 of 8
Who should have caught the wrong chart?
Neha asks her team’s data assistant for revenue by region, last quarter. One agent writes SQL, the database runs it, and a second agent draws the chart. North towers over the rest at ₹50.0k, so Neha tells her team North nearly doubled.
It did not. North made ₹26.0k. Both agents did exactly what they were asked, and nothing checked their work. The usual fix is a third model that reads the other agents’ output and rules on it. That is an LLM as a judge.
Here is the route. We will see why the obvious judge prompt fails, what the judge needs to see, what it should check, how to lay out its prompt, how that prompt differs from an agent’s, and how to know the judge is right. We start with the prompt almost everyone writes first.
Stage 2 of 8
Why doesn’t “rate this 1 to 10” work?
The first judge prompt most people write is one line: is this chart good, rate it from 1 to 10. We hand the judge the chart and that line. The North bar is off by almost double.
What score will the judge give this chart?
Nine out of ten: clear labels, a sensible bar chart. The judge graded exactly what it could see, and everything it could see looks fine. A wrong chart and a right chart look equally tidy.
Run the same prompt three more times on the same chart. It says 7, then 9, then 8. Nothing changed but the model’s sampling. A 1 to 10 scale gives that randomness room to show, because nothing says what a 7 means.
That one line has two holes. The judge never saw how the chart was made, and it was never told what good means. We fix the first hole first: what the judge gets to see.
Stage 3 of 8
What does the judge need to see?
Here is everything the pipeline made along the way: the question, notes on the tables, the SQL, the rows it read, and the chart. Dark cards are the ones the judge was not given. Watch it try to trace the tall North bar back to a cause.
Hand it the SQL. The thread now reaches the line that joins orders to order_items. But joins are normal SQL. Without knowing how the tables are shaped, the judge has no reason to object, so it still passes.
Add the schema notes. order_items has one row per item, and total_amount is the whole order’s amount. Now the join means something: every order is added once for each item in it. The judge fails the chart and says why.
Add a few sample rows and the proof is on the table. Order 1041 had two items, so its ₹12.0k appears twice, and 1043 appears three times. North sums to ₹50.0k. Counted once each, it is ₹26.0k. A judge can only catch a mistake it can trace, and it can only trace through what you hand it. A random sample can miss the repeats, so send row counts too.
A judge can only catch a mistake it can trace, and it can only trace through the evidence you hand it.
Your turn, with a fresh judge. Tap a card to hand it over, then press Run judge. This query hides a second mistake too. Give the judge enough to catch both, and keep it inside its 200k token window.
Stage 4 of 8
What exactly should the judge check?
The judge can now see enough. But “is this good” still leaves it to guess what good means. So we break good into checks: yes or no questions that tie a word in Neha’s request to something in the output.
“Revenue” should be SUM(total_amount). “By region” should be GROUP BY region. “Last quarter”, asked on 10 July, means 1 April up to 1 July. The WHERE line has a start but no end, so order 1048 from 4 July slips into South. That is the second mistake.
Each check gets a written pass rule and fail rule. For example, fail “each order counted once” if any order can appear more than once before the SUM. Written rules mean the judge applies your line, not its own.
Some checks need no model at all. Does the SQL run? Is the result empty? Do the bar heights equal the result values? Code answers those exactly, every time, for free. It can even catch our double counting: after the join, compare COUNT(*) with COUNT(DISTINCT order id). Run code checks first, and give the judge only the questions that need judgement.
Switch the Report control between Score 1–10 and Checks. The score says 6 out of 10 and blames nobody. The checks say the SQL agent failed two and the chart agent passed all of its own. The chart drew the wrong numbers faithfully, so the fix belongs upstream.
Checks turn one fuzzy question into several with one right answer each, and they point at the agent that went wrong. Next, how to lay the evidence and the checks out in one prompt.
Stage 5 of 8
How do you lay out a judge prompt?
Everything now goes into one prompt, in a fixed order, top to bottom. It opens with the job in one sentence: you are reviewing SQL written by another agent, and you report problems but never fix them.
Next comes the evidence, each piece in its own labelled block, like <sql> and </sql>. The tags keep the agent’s SQL apart from your instructions, and let each check name the block it should read.
Then the checks, each with its pass rule and fail rule, plus one short example of a failing case. An example shows the judge where your line sits better than an adjective like “correct” can.
Last comes the output format. Suppose the judge writes a verdict field first and a reasoning field after it. The reasoning notices the join repeats orders.
Can that reasoning change the verdict it wrote first?
Watch it write. “pass” comes out first. Then the reasoning spots the repeated orders, and it is too late. Words written after the verdict cannot change it, so the output contradicts itself.
Switch the Output order control to Reasoning first. Now the judge quotes the JOIN line and the schema note, reasons about them, and only then writes the verdict: fail. The case is made before the ruling.
Two lines finish the page. Allow “cannot_judge” when evidence is missing, so the judge never guesses pass. And ask for JSON with one entry per check, so your code can read it. Next, why this page looks so unlike your agents’ prompts.
Stage 6 of 8
How is a judge prompt different from an agent prompt?
Your SQL agent has a prompt too. The same model can sit under both prompts, but they ask it for opposite jobs. On one side it works as the agent. On the other it works as the judge.
As the agent, it makes something, and many different drafts could be right. As the judge, it sorts one thing into one of a few fixed bins: pass, fail or cannot_judge. Only one bin is right.
The agent is told to be helpful, fill gaps with sensible guesses and fix what is wrong. The judge is told to be skeptical: it wants the evidence on its desk before it rules, and it never fixes the SQL. A judge that repairs the query and then grades its repair tells you nothing.
Agents often end with a comment on their own work. Ours wrote: “Verified: totals match the database.” Turn on the Agent’s note control and watch what the judge does with it.
With the note, the judge passes the chart. Judges lean toward confident, authoritative text, and a claim is not evidence. Hand the judge the SQL, rows and chart, never the agent’s opinion of them.
Judges have other known leanings. They tend to prefer the longer answer, prefer text from their own model, and when comparing two outputs, favour one position. Keep length out of the checks, use a different model for the judge when you can, and run comparisons twice with the order swapped. Keep temperature low where the API allows it, and rerun a few cases to see how much verdicts still vary.
So a judge prompt is a classifier: a few fixed bins, written rules, a skeptical stance, and no view of the agent’s self-praise. Which raises the last question. How do you know this classifier is any good?
Stage 7 of 8
How do you know the judge is right?
A judge sorts outputs into pass and fail, so you test it like any classifier: against answers you trust. Here are 40 outputs from the pipeline. Someone who knows the data marked each one by hand: 12 are wrong and 28 are right.
Run the one-line judge from the start of the lesson on all 40 and sort the results. It agrees with the person on 31 of 40. That sounds decent.
Now look only at the 12 wrong outputs. It flagged 4. Most outputs are fine, so agreeing is easy: a judge that passes everything would agree 28 times. Catching the bad ones is the whole job, and this one misses two in three.
Run the judge from this lesson instead: evidence, checks, reasoning before verdict. It catches 10 of the 12 wrong outputs, and raises 3 false alarms.
Read every disagreement. One miss showed amounts in the wrong currency, and no check covered units. Two false alarms used a view where the join was already safe. Add a check and one example for each, and run again.
Switch the Judge prompt control between v1, v2 and v3 and watch the misses shrink. Version 3 catches 11 of 12 with 1 false alarm. Keep a few labelled cases you never tune on, and rerun them whenever an agent or the prompt changes.
A judge is only as trustworthy as its last test against people. The numbers here are made up for the lesson, but the method is the real one. Now let us put the whole judge together.
Stage 8 of 8
What does a good judge look like?
We started with one line: rate this chart 1 to 10. The judge saw a tidy chart and gave it 9. A vague judge grades what it can see.
Then we handed it evidence: the question, the SQL, schema notes and sample rows. The thread from the North bar reached order 1041, counted twice.
We split good into pass/fail checks tied to Neha’s words, and let code do the exact ones. The checks blamed the SQL agent, not the chart agent.
We laid the prompt out in a fixed order and put the reasoning before the verdict, because words after a verdict cannot change it.
We saw that a judge prompt is a classifier: fixed labels, skeptical, never fixing, and blind to the agent’s own note about its work.
And we tested the judge against a person’s labels, looking at the failures it caught rather than overall agreement.
For your pipeline: one judge after each agent, handed the request and that agent’s evidence, asked narrow checks, reasoning first, answering in JSON, and tested on labelled cases.
A verdict needs somewhere to go. When a check fails, send its reasons back to that agent for a retry or two. Send cannot_judge and repeat failures to a person, and log every verdict. That is how North gets its real ₹26.0k. A judge on every request adds time and cost, so you may run it on a sample instead. Next time a chart looks perfect, you will ask what the judge was allowed to see.