The OpenAI Decisions API Hype Met a Jar of Marbles

A developer ran a thousand trials against OpenAI's brand new Decisions API. Loaded coin, biased to 70% heads. The API nailed it, sampled heads almost exactly 70% of the time.

Then he tried a jar of marbles. Five red, three blue, two white.

True odds of drawing red, 50%. The API picked red 86% of the time.

He reordered the answer options, moved red from first to last. The number dropped to 73%.

Same question, same jar, same model. Just a different order on the list.

i could not stop thinking about that.

This is the API everyone spent a week losing their minds over. OpenAI announced it at DevDay, powered by a specialized gpt-6-luna, aimed squarely at the fast decision market TypeSafe's Jev had just made famous.

The public beta landed days later. The takes were loud.

Jev killer. The end of a whole startup category.

And buried under all of it was this quiet experiment where a probability moved because someone rearranged a list.

If you're going to hand a model your routing logic, you should know about the marble jar first.

What the Decisions API actually is

The idea is simple and i wish more launches were this honest. Instead of generating text token by token, the model returns one of three typed answers:

• predicates: yes or no, with a probability
• choices: pick one from a fixed list you define
• scores: rate against a rubric

No essays. No filler. Just a judgment, fast.

OpenAI claims up to 10x speedups because there's no long-form generation. In theory that's around 150ms per call instead of 1.6 seconds of text. For high-volume routing, that math is attractive.

The catch is what you give up. The model can't explain itself. It can't show its work.

You get a verdict and a number instead.

The Jev fight

here's a question people always ask about this launch: was it a copy?

TypeSafe's Jev went from viral curiosity to a $7.5B valuation at over $100M ARR in what felt like weeks. It defined the category.

Then OpenAI shipped a direct answer almost immediately. The r/singularity thread summed up the mood: "Takes away a massive chunk of the market TypeSafe was eyeing."

The top Reddit reply was less impressed.

Wait, it's a copy of Jev, and essentially just a super light Luna?

The benchmark numbers landed somewhere in between. LangWatch ran both on LLM-as-a-judge tasks:

OpenAI faster and more accurate. Jev less than half the price. The stat nerds called it a statistical tie with a price gap.

The slow part nobody tweeted

The interesting complaint came a few days later. A thread on r/LLMDevs titled Jev alternatives called the Decisions API "disappointingly slow." That is not what you expect after a launch built on a 10x speed claim.

Same story as always with speed launches. The benchmark is a toy. Real traffic is not.

People trying to route support tickets found the median latency fine and the tail annoying, enough seconds to feel broken.

i think the deeper issue is what OpenAI measured against. Compared to its own generation endpoints, this is fast. Compared to a small model doing the same classification job, the gap shrinks fast.

One more thing nobody puts in the launch post: both models fall apart when the answer needs actual math.

Adding up invoice line items? Hard fail on both. The model judges, it does not calculate.

Microsoft showed up three days later

And then the weirdest thing happened. Microsoft dropped Microsoft-Decision-1, its own decision-scoring model, three days after OpenAI's beta.

The twist: they built it by post-training Alibaba's open-weight Qwen3.5-9B. Not an OpenAI base. Microsoft, OpenAI's biggest partner, used a Chinese open model for its fast judgment layer.

It's cheaper too. $0.042 per million input tokens, no output charges. Text-only, 32k context.

Microsoft claims it wins a 36-benchmark suite of nearly 150,000 blind questions, running 35x faster than flagship models like GPT-6 Sol. They're already planning to rebase it on their own MAI models.

i read that as a statement. Microsoft doesn't want to depend on OpenAI for the cheap, boring, high-volume judgment layer.

So now three players are fighting over the same job: judge a thing, fast, cheap. The winner becomes the default brain inside every routing system.

A break from the benchmarks

i have a folder on my desktop called "launch week." Every time a company announces the fastest model ever, i screenshot the announcement and drop it in there.

There are forty files now. None of them is the fastest model ever anymore.

it's a weird hobby but it keeps me honest.

gpt-6-luna, Jev, Decision-1, Dots. Every name was the future for exactly one week.

i told a friend about the folder once. She said the real pattern was companies running out of nouns. i think about that every launch.

i remember reading a Jev alternatives thread and watching people argue about whether some open model called Strom "looks OK." Not great. OK.

That's where the category already is. A new thing appears and within a month it's a commodity with a spreadsheet of tradeoffs.

And the naming. That's the part i can't get past.

Your infrastructure now depends on models named like indie bands. Jev sounds like a fantasy novel character. Luna sounds like a perfume.

Decision-1 sounds like a Star Trek android.

The honest part

Most people don't need this. Let's be clear about that.

If you're routing a few thousand decisions a day, a normal model with a JSON schema does the same job. Slower, sure, but free, because you already pay for the model.

This category only matters at volume. Content moderation at millions of items. Ticket routing across an enterprise.

Agent loops that would otherwise burn thousands of tokens deciding between two options.

And if you do use it, don't trust the raw probabilities. The marble experiment is right there.

A model whose confidence shifts 13 points because you reordered a list is giving you a vibe, not a measurement.

Treat the score as a hint, build your own thresholds, and test the order of your answer options before you ship.

One more honest note: the 10x claim only applies when you were generating text you didn't need. If your workflow writes a paragraph, this API is useless. It's a switch, not a writer.

---

i keep coming back to that marble jar.

The coin was perfect. The jar was a mess.

And that's the whole story of this API in one experiment. Great when the question is clean. Weird when the world shows up with mess and order and bias.

i'll use it for triage. i won't ask it for truth.

maybe the real lesson is simpler. The hype cycle moved faster than the calibration. Everyone wanted to argue about who killed Jev, and almost nobody asked whether the probabilities held up when you shuffled the answers.

that question is worth more than all the launch takes. and i still haven't found a good answer to it.