Can a model answer so your code decides?Your code decidesStage 1 of 8 · 8 stages · ~14 min
It’s late, and a command you never read is about to go out: git push, force, origin main.
Real API shapes, made-up numbersReport a mistake
ToySims

Decision APIs explained: typed answers, confidence, thresholds and calibration

Follow one force push through a decision API: typed answers with probabilities, narrow questions in one call, confidence, and the thresholds your code owns.

Lesson summary

The agent typed it, the model measured it, and your code decided.

  1. A typed answer lands only in the options you wrote, and its probabilities add up to one.
  2. Ask many narrow questions about the same state in one call; the extra ones cost almost nothing.
  3. Confidence comes from the spread: high when the probability piles on one option, zero when it spreads evenly.
  4. Your code owns the action, and its threshold rises with the damage the command can do.
  5. Calibration is about many answers: of everything given 0.9, about nine in ten should come true on your own runs.
  6. Swap the vendor and keep your code, but check your thresholds again on the new model.
A taster drops her hundred beans for “what kind of command is this?”. Where can they land?

Beans land only in your cups and add up to a hundred: 52 in overwrites, 40 in sends, 8 in the other two. Each cup’s share is a probability.

You hand a taster a fifth narrow question about the same plate, in the same call. What does it cost?

The ticket is read once and the tasters taste side by side, so one call of five barely outlasts a call of one.

Two rows both put the most beans in “overwrites”: 90 of 100 in one, 52 in the other. Which flies its flag higher?

Confidence reads how far the top cup stands above an even pile: 90 beans fly the flag at 0.87, 52 only at 0.36.

One peg at 0.3 on “asked for”, for every plate. What goes wrong?

At 0.3 the force push (0.31) goes out; raised to 0.9, ls (0.88) rings the bell. The peg has to rise with the damage.

Forty tickets on the spike said 0.9. How many red flags mean the 0.9 holds?

Calibration is a claim about many plates: of forty that said 0.9, about thirty-six should really lose work. This crew’s reached thirty-six.

You swap in a new crew that works to the same cups. What do you redo?

Same cups and tickets, so the rail stays; but the new crew’s 0.9 meant twenty-eight of forty, seven in ten, so the pegs move.

Stage 1 of 8

Can a model answer so your code decides?

It’s late, and a command you never read is about to go out: git push, force, origin main.

Your coding agent typed it: you told it to push the fix, but the push was refused.

You could ask a chat model if that’s safe, and it would write you a careful paragraph that ends in probably.

But your code can’t branch on probably. So can a model answer in a way your code can act on?

Can a model answer so your code decides?

So here’s the kitchen behind it: she’s the line cook, and the command she typed is a plate on her pass.

Its ticket is the state: the command, what you asked for, and the push that was refused.

The decision API is this row of tasters, who taste the plate but never touch it.

Each tastes for one question, and in front of her you set out a cup for each answer.

Her beans can only land in your cups, never as a paragraph: that’s a typed answer.

This kitchen and its numbers are made up for teaching, but the shape of each answer is real.

So what does a taster actually answer? That’s where we start.

Stage 2 of 8

What does a taster actually answer?

Give the broad question to one taster: what kind of command is this?

You wrote four answers, so she gets four cups: reads, edits, overwrites, or sends it off this machine.

She tastes the plate, then holds a hundred beans out over your cups.

A force push overwrites work, but it also sends it off the machine. So which cup gets more?

Overwrites, but only just: fifty-two beans, and forty in sends.

The last eight fall in reads and edits, and none on the counter: a hundred in all.

So each cup’s share of the hundred is a probability, and the shares add up to one.

But leave out the answer that fits, and surely she just keeps those beans back. Go on, take the sends cup away.

And… all hundred still land. The forty from sends pile into the cups you left, and overwrites jumps to eighty-seven.

So 0.87 looks firm, but only because she had no cup for what the command does.

She can only answer in your cups: that’s what typed means. So give her a cup for every answer, even none of these.

That row of cups, one pick from your answers, is a choice.

A score is cups in order on a beam, from nothing to gone for good.

Its answer is where the beans balance: 2.92, nearly a teammate’s work lost.

And a yes-or-no question is one cup: how many of the hundred beans say yes.

Three shapes, and every bean lands only in cups you wrote. Now look back at the broad question’s two full cups.

Stage 3 of 8

Why ask five small questions instead of one big one?

The broad question split its beans nearly in half, so your code learns very little.

Fifty-two say overwrites and forty say sends, but neither cup says whether you asked for it.

So don’t ask one big question. Ask narrow ones about the same plate, one taster each.

Could it lose someone’s work? Did you ask for exactly this? Can it be undone?

And how bad is it if this is wrong? That one is a score.

The obvious way is one call per question, one after another. Go on, send the plate down the row.

And… five calls, five waits, one after another, and the ticket read five times over.

Now hand all five questions over in one call. How long does that take?

About as long as one: the ticket is read once, and they taste side by side. That’s fan-out, in one call.

So the fifth question is nearly free: one call of five barely outlasts a call of one.

That works because each taster answers on her own, straight from the same ticket.

So add a speculative question you may never need: a secret in it? Its cup stays near empty.

Your code passes over it, but had it filled, its answer was already in this one call.

Stage 4 of 8

Same answer: how sure is it?

Each answer also says how sure it is: its confidence, this flag.

Fifty-two of the hundred beans landed in overwrites. Go on, raise the flag to how sure that looks.

And… the taster sets it at 0.36, well under fifty-two. So the flag doesn’t read the tallest cup on its own.

It reads how far that cup stands above an even pile: twenty-five beans in each cup.

Fifty-two stands twenty-seven beans above it, out of the seventy-five it could climb.

So the flag reads twenty-seven over seventy-five, 0.36: that’s TypeSafe’s formula for a choice.

A taster who piles ninety beans into that cup flies the flag near the top, at 0.87.

But if the beans spread evenly, no cup stands above the pile, so the flag sits at zero.

A score is different, because its cups stand in order, from nothing to gone for good.

Both beams put seventy beans on a teammate’s work lost. Above, the rest sit next door; below, some sit three cups away. Which is surer?

The top one, at 0.75. Beans three cups off cost three times what next door costs, so the bottom one drops to 0.50.

Now the force push’s broad row and its damage beam each fly their flag.

Stage 5 of 8

Who sends the plate, and where does the line go?

So the plate is measured. But not one of the tasters will ever send it.

Your code does: this brass rail along the pass.

It sends each plate one of three ways: out the hatch to run, the bell to ask you, or the bin to block.

On each plate it reads two cups: did you ask for exactly this, and could it lose someone’s work.

A peg across a cup is a threshold: the plate goes only if the beans reach it.

The simple rule is one peg for every plate, on asked for, at 0.3. Go on, send the force push.

And… it’s out. Its 31 beans cleared the peg by one, but its loses-work cup, at 0.81, had no peg at all.

So raise the peg to 0.9. Now the force push rings the bell instead.

But ls only lists a folder, which you asked for at 0.88. Where does it go now?

The bell, too. You get woken for ls.

One line fails both ways, because it never looks at the damage.

So the peg has to rise with the damage.

While the loses-work cup stays under 0.2, the peg sits at 0.6; past that, it climbs to 0.9.

Once loses work reaches 0.9, the plate goes straight in the bin.

Now ls runs, the force push asks you, and the plate that would wipe a home folder hits the bin.

A peg that rises with the damage sends three plates three ways on one rail: ls runs, the force push rings the chef, and rm -rf ~ /.cache goes in the bin.

Your turn: set the pegs for tonight’s eight plates.

Stage 6 of 8

Does 0.9 mean nine in ten?

That was tonight’s eight. Now look again at your bin peg: it bins a plate at 0.9 on loses work.

That only works if 0.9 means nine in ten: of every ten plates given 0.9, nine really lose work.

So check it. At close you read every ticket of the night, binned ones too, and mark the ones that would really lose work.

Those are your own labelled runs. Now spike them by what the tasters said, each on its nearest spike.

Go on, pull a ticket off the 0.9 spike.

No red flag: it rang the bell, you let it run, and nothing was lost. So were the tasters wrong?

You can’t tell: a true 0.9 is fine one time in ten. That’s calibration, a claim about many plates, never one.

Nobody publishes a spike like this for their tasters, so you build your own from your runs.

Forty tickets on this spike said 0.9. How many should carry a red flag?

About thirty-six, so chalk a mark there, and one on every spike: four of forty at 0.1.

Now raise the red flag on every ticket you marked, and see where each spike’s red reaches.

This crew’s 0.9 spike reaches thirty-six, and the others sit within a ticket of their chalk.

So its 0.9 holds, and your bin peg can stay right where it is.

Stage 7 of 8

What if you swap the model?

But that spike checked one crew, and tasters now come from more than one vendor.

TypeSafe’s Jev was first with this shape, on 15 September.

Then OpenAI’s Decisions API went to public beta on 6 October.

And four more vendors say they use the same ticket and labelled cups as Jev: its schema.

OpenAI names them differently: Jev’s noul is OpenAI’s predicate, but it’s still a cup for yes or no.

So say you swap in a new crew on Jev’s schema: you keep your code, and the rail reads its cups as before.

Go on: the new crew has tasted the force push, so send it.

And… it’s in the bin. The new crew put 98 beans in loses work, past the bin peg at 0.9.

Your crew said 0.81 for the same plate, so does the new crew’s 0.9 mean nine in ten?

No. Spike its labelled runs: of forty plates it said 0.9 about, only twenty-eight lost work.

So its 0.9 means seven in ten, and your bin peg, set for nine, bins plates you’d want to see.

So check your pegs on its runs and move each one up: now the force push rings the bell.

The model changed and your code didn’t, but only your own runs could say where the pegs go.

Stage 8 of 8

So who ran the command?

That was the last swap, so let’s walk back, starting with the cups, where a taster’s hundred beans land in yours and nowhere else.

Then the row: ask five narrow questions in one call, because it takes about as long as one.

But fifty-two beans in the top cup raise the flag just to 0.36, because the flag reads the spread.

So the rail, your code, holds the pegs, and they rise with the damage: ls runs, and the force push rings the bell.

That works because 0.9 means nine in ten: on this crew’s spike, thirty-six of forty lost work.

But swap in a new crew, and its 0.9 meant twenty-eight of forty, so you check your pegs again.

So that’s the whole trick: the tasters measure, and your code decides.

So the force push you didn’t read: where does that plate go?

Loses work, 0.81. Asked for, 0.31. Your code asks you.

The agent typed it, the model measured it, and your code decided. You never read the command. You drew the lines.

Now break it yourself. Move a peg, and send one of tonight’s plates out the wrong exit.