OracleLadder

You have probably done this yourself: you work out every piece of a hard problem and still get the answer wrong, because one piece never makes it into the final step. Language models show the same pattern, and for all six models we tested it is the most common way they fail.

If you train, compare, or benchmark reasoning models, a wrong answer alone tells you very little. Did the model lack a plan, fumble a step, or hold every piece and fail to put them together? Each calls for a different fix.

OracleLadder separates these cases. It holds each problem fixed and changes only what the model is told: nothing, a roadmap of the steps, or the roadmap plus the answer to every step. When a model solves every step alone and still fails with every answer in hand, we call it the composition gap.

One problem, four levels of help

Qwen3-8B (a 2025 model, thinking off) answers each question 8 times. Pick a level of help to see every answer it gave.

Through the point (2, 3) on the line p: y = 2x − 1, a line q perpendicular to p is drawn. Find the area of the convex quadrilateral bounded by p, q, and the coordinate axes.

M1Line q has slope −1/2 and passes through (2, 3). Find its equation.answer given: y = −x/2 + 4

M2Substitute y = 0 into y = 2x − 1 to set up the x-intercept of p.answer given: 0 = 2x − 1

The last step, assembling the area, stays hidden because its answer is the solution.

“But this point isn’t on the x-axis or y-axis, so it is not part of the convex quadrilateral bounded by the four lines.”Attempt 3 with both step answers in hand, about the corner (2, 3).

True region, area 19/4 Shape most answers measure, area 65/4

Figure 1. Every attempt at each step alone is correct. Every attempt at the full problem is wrong, at every level of help. With both step answers given, all 8 attempts find the four axis intercepts and leave out the corner (2, 3). The most common answer, 65/4, is the area of the dashed shape. Answers are copied from the saved outputs.

How OracleLadder works

A teacher model (GPT-5.4 Thinking) reads each problem and its reference solution and writes one fixed roadmap of milestones: small standalone problems, each with a gold answer that a program can check. Automatic checks drop milestones that give the final answer away. The teacher never sees the models we test, so every model gets the same roadmap.

We then run two tests on each model. The milestone test asks whether the model solves each milestone alone. The ladder of help gives the full problem with no help (C1), with the roadmap (C2), and with the roadmap plus every milestone answer (C3). A deterministic symbolic verifier grades every answer, with no LLM judge.

Protocol: the teacher builds a milestone family; Track A tests each milestone alone; Track B climbs C1, C2, C3 on the parent problem. — **Figure 2.** The protocol, from the paper. Track A is the milestone test. Track B is the ladder of help. The lowest rung that solves the problem gives the outcome.

Two tests, five gaps

Crossing the milestone test with the lowest rung that works sorts every failure into one of five gaps. Each gap points to a different fix. Pick a model to see its counts on 354 problems, and pick a cell to read what it means.

Solved with no helpNeeds the roadmapNeeds the answersNever solved Passes the milestone test Fails the milestone test

Directcounted above

Figure 3. Counts at a 16K-token budget with 8 attempts per condition. A problem counts as solved at a rung if at least one attempt is correct. Direct solves are counted once, whatever the milestone test says.

Composition is the largest gap for every model

Six open models from 2024 and 2025, from 8B to 671B parameters. For every one of them, the composition gap is the largest of the five gaps: 33–48% of the 354 problems.

Our verifier is strict, so some of these problems are grading errors. An LLM review flags 22–29% of each model's composition-gap problems as verifier or format errors. After removing them, the gap still covers 24–37%.

Remove problems flagged as grading errors

Figure 4. Share of the 354 problems in each gap. The number on the right is the composition gap. The flags come from an automated GPT-5.5 review, and each family's label is carried from the 4K to the 16K milestone test.

One accuracy number hides the difference

DeepSeek-V3.1 is right more often per attempt than gpt-oss-20b, 28.4% against 25.7%. Yet gpt-oss-20b solves more problems once milestone help is allowed: 214 against 183. It leaves more of its failures recoverable. In 1,000 bootstrap resamples of the problems, gpt-oss-20b solves the most problems in every one.

Figure 5. Per-attempt accuracy with no help against the share of problems that any rung of help solves, on the same 354 problems.

The help is problem-specific, and it carries over

Is the roadmap real information, or only a longer prompt? We replace it with a roadmap from another problem, with a generic "break it into steps" prompt, or keep it and pair it with wrong answers. Only the matched roadmap lifts recovery. The same contrast holds on MATH500 and AIME 2024/25, where Qwen3-8B (thinking off) is both the screen and the student.

Figure 6. Share of problems solved under each condition. Main set: MilestoneRL-2K. MATH500 uses at least 3 correct of 8 attempts, because 4 screening attempts let easy problems through. AIME uses at least 1 of 8. The pattern also holds with a second teacher (83–87% same per-problem outcome) and in a 10-problem code pilot (6.5% → 41% → 89% of attempts pass).

Two RL runs with the same gain move different problems

Two RLVR runs start from Qwen3-8B-Base. They gain almost the same on MATH, +20 and +18 points. One turns 30 more problems into direct solves, the other 20. The runs differ in more than the reward, so we read them as two readouts of training and do not rank the methods.

Figure 7. Change in each outcome against Qwen3-8B-Base on the 354 problems. Intermediate checkpoints show both runs dropping below the starting point before they recover, which a single accuracy curve does not show.

What the composition gap contains

We report three numbers separately, because each points to a different fix.

33–48%of problems fall in the composition gap, depending on the model.

22–29%of each model's gap is flagged as grading errors: the verifier or the reference answer. That fix belongs to the benchmark.

16 of 18audited families recovered by Qwen3-8B with every answer given. The rest of the gap mixes roadmaps that miss a step with real failures to combine steps.

Figure 8. Raw-output retest of the 18 families that two authors labeled as genuine composition failures, with the roadmap and every milestone answer given, 8 attempts each, graded by hand. Most failures trace to the reference answer or its format. On ff548f26 (Figure 1), Qwen3-8B solves both steps alone and still fails all 8 attempts, while gpt-oss-20b solves it in 6. Bars show correct attempts out of 8 for Qwen3-8B (thinking off) and gpt-oss-20b.

Talk and poster

A five-minute explainer with captions. A 2-minute version is here.

Poster (PDF) and slides (PDF). Poster Session 4 in Sydney, Wed, Dec 9, 17:00–20:00 AEDT.

Data, code, and citation

The 354-problem diagnostic set with every roadmap and gold milestone answer.
Milestone-test and ladder results for all six models, the symbolic verifier, all prompts, and the audit labels.
Replication data for MATH500, AIME 2024/25, and the code pilot. Any new model can be run on the released roadmaps directly.

Everything is at github.com/slark-prime/OracleLadder. Code is MIT-licensed and data is CC BY 4.0.

@inproceedings{wang2026oracleladder,
  title     = {Solving Every Step Is Not Enough: Milestone Oracles Reveal
               a Composition Gap in {LLM} Math Reasoning},
  author    = {Wang, Zhuohan and Ma, Haoran and Wu, Tianyu and
               Duan, Yuanlin and Liao, Zichun and Yu, Jieming},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  eprint    = {2609.32235},
  archivePrefix = {arXiv}
}