Gut Feel Benchmark

Gut Score averages skill over seven deadlines from 0.25 to 16 seconds, with every late answer counted as the base rate. Skill is the share of the base rate’s squared error that a forecast removes, with no deadline. Both run to 100, and both average the poker, base-rate and Hacker News packs.

With 400 headline items per contestant, against 75 in the trial run of 25 items per pack that came first, the intervals shrank to between a third and a half of their earlier width. Each contestant beats the next one down in at least 99% of bootstrap resamples, except Microsoft-Decision-1 against Opus with reasoning off, at 61%.

Gemini 3.8 Flash answered only the trial run’s 100 items, because it ran past the 120-second limit on 23 of 25 poker hands. On those items it scored 0.3, against 59.0 for Luna.

The decision models keep nearly all their skill from half a second on, while the chat models keep none of it before two seconds. OpenAI Luna scores 45.7 at the quarter-second deadline and 63.3 at half a second, against a skill of 64.7. Opus with reasoning off scores nothing before 2 seconds, 55.3 at 4 seconds and its full 69.3 only at 16.

Each cell below is skill at one deadline, shaded more strongly the more skill it holds.

High reasoning bought Opus one point of skill and cost it nearly five points of Gut Score. It used 132 reasoning tokens per answer instead of 27, scored 70.3 against 69.3 and slowed from 1.8 to 3.1 seconds, so its Gut Score fell from 30.8 to 26.2. The skill gain sits inside both intervals, and it widens only on heart disease, outside the headline, where high reasoning scored 70.3 against 62.4.

Reasoning off held for neither chat model. Opus used 27 reasoning tokens per answer and Sol 118, with no warning from the SDK. Sol took a median 22.3 seconds per poker hand, so a poker skill of 97.1 kept a Gut Score of 4.0.

The exact packs separate the field and the outcome packs barely do. The chat models scored 100.0 on base rates and 97.1 to 98.9 on poker, against 99.9 and 93.4 for Luna, 80.3 and 76.8 for Jev, and 67.3 and 52.6 for Microsoft-Decision-1. On Hacker News no contestant passed 12.1, and the three decision models scored between −2.0 and 0.9.

Heart disease sits outside the headline, in grey, because its records have been public since 1988 and a model may remember them. The chat models scored 62.4 to 70.3 there, and the decision models 40.2 to 46.4.

The full run used $5.84 of AI Gateway credit, and the benchmark has used $9.35 so far. A thousand answers from Jev or Luna cost about 2 cents, and a thousand from Opus with high reasoning cost $4.99.

In the trial run, Gateway appears to have billed Gemini’s 25 timed-out calls although they returned nothing, and that cost is why Gemini sat out the full run.

Each contestant answered all 703 items of four packs. Poker hands are scored against exact equity over all 1,712,304 boards, and screening-test problems against the exact posterior by Bayes’ rule. Cleveland Clinic patients are scored against their angiograms, and Hacker News stories posted from 2 to 9 October 2026 against whether they reached 100 points.

Every item went through AI SDK 7’s experimental_decide on Vercel AI Gateway, natively for the decision models and through a structured-output adapter for the chat models. Calls ran four at a time with a 120-second limit and no retries, after one untimed warm-up call per contestant. Two of 4,218 calls failed: Luna refused one Hacker News item, and Sol ran past the limit on one poker hand. Both count as base-rate answers. Gateway answered a request that does no model work in 40 to 92 milliseconds, so network time is a small share of every latency.

No deadline scores above a contestant’s skill with no deadline, so a late answer can lower a score but never raise it. The intervals come from 1,000 paired bootstrap resamples of the items, each scored against the pack’s own base rate.