Oct 22, 2026 · 9 min read
ThinkingCap: Qwen3.8-Flash-Next — up to 51% and on average 39% less thinking, comparable benchmark accuracy
TL;DR
This is the fourth model in the ThinkingCap series. We reduced how much Qwen's Qwen3.8-Flash-Next thinks, without compromising the quality of its answers.
- Up to 51.0% reduction in thinking tokens, on average 38.5% measured across six benchmarks, for 1.03pp of accuracy
- The largest model in the ThinkingCap series so far, and the compression holds at that size
- Same sampling settings (thinking on, temperature 1.0, top_p 0.95), same serving stack (vLLM), drop-in replacement
ThinkingCap PRO. Our enterprise edition cuts thinking tokens by 60% on average and up to 75.8%, so your team gets faster answers and lower inference costs at scale. Contact us at enterprise@bottlecapai.com.
Our fourth ThinkingCap
We have published three token-efficient models so far: ThinkingCap-Qwen3.6-27B, ThinkingCap-Qwen3.8-27B and ThinkingCap-NVIDIA-Nemotron-3.5-Lightning-30B-A3B.
Qwen3.8-Flash-Next is a sparse mixture of experts whose bf16 weights take about 360 GB, which needs four H200s to serve. It is a bigger model than the other ThinkingCap releases, and this one shows the same thinking-token efficiency carries over at that size: thinking comes down 38.5% on average, in line with our Qwen3.8-27B result.
Video comparison of the original model and the ThinkingCap version, prompted with the same input.
Both models run on the same hardware and use the same inference settings.
The training objective
We start with Qwen3.8-Flash-Next. The goal is to spend fewer tokens getting to the answer, and change nothing else: preserve the knowledge, reasoning ability, answer quality, instruction following and safety behaviour.
Results
Accuracy per task family, Qwen3.8-Flash-Next against the ThinkingCap checkpoint
Accuracy tracks the original model closely in every family. Agentic work costs least, 0.17pp; maths and code 1.06pp; long-context and multimodal 1.41pp. Averaged over all six benchmarks the cost is 1.03pp.
Mean thinking tokens per task family, Qwen3.8-Flash-Next against the ThinkingCap checkpoint
The thinking is where the change is, and every family gets shorter. Maths and code reduces the most, 43.2%, then long-context and multimodal at 39.5%, with agentic last at 22.5%. Averaged over the six benchmarks it is 38.5%.
| Benchmark | Accuracy (base) | Accuracy (ours) | Δ accuracy | Mean thinking tokens | Δ mean thinking | Median thinking tokens | Δ median thinking | Mean answer tokens | Δ mean answer | Trunc. % | Seeds |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Math & code | |||||||||||
aime26 | 97.92% ±0.85 | 97.19% ±1.15 | -0.73pp | 16694 → 9281 | -44.4% | 9494 → 4371 | -54.0% | 9 → 9 | -0.8% | 0.0 → 0.0% | 32 |
hmmt_nov25 | 99.17% ±1.03 | 97.71% ±1.55 | -1.46pp | 15975 → 8874 | -44.5% | 9584 → 4866 | -49.2% | 12 → 11 | -1.4% | 0.0 → 0.0% | 16 |
livecodebench | 91.86% ±2.71 | 90.86% ±1.05 | -1.00pp | 28653 → 16945 | -40.9% | 16565 → 6860 | -58.6% | 492 → 421 | -14.5% | 0.3 → 0.0% | 4 |
| Long-context & multimodal | |||||||||||
realworldqa | 85.62% ±0.50 | 84.67% ±0.43 | -0.95pp | 1102 → 540 | -51.0% | 161 → 115 | -28.3% | 4 → 4 | -0.8% | 0.0 → 0.0% | 8 |
aa_lcr | 80.62% ±1.54 | 78.75% ±2.60 | -1.88pp | 1510 → 1088 | -28.0% | 934 → 706 | -24.4% | 242 → 198 | -17.8% | 0.0 → 0.0% | 8 |
| Agentic | |||||||||||
tau2 | 83.57% ±1.33 | 83.39% ±1.86 | -0.17pp | 2111 → 1637 | -22.5% | 1698 → 1333 | -21.5% | 1176 → 1054 | -10.4% | 0.0 → 0.0% | 8 |
| Average (6 benchmarks) | 89.79% | 88.76% | -1.03pp | 11007 → 6394 | -38.5% | 6406 → 3042 | -39.3% | 322 → 283 | -7.6% | 0.05 → 0.00% | |
Every row is the original BF16 model against the ThinkingCap checkpoint, run on the same harness with the same settings on the same questions. Accuracy is the mean over seeds, ± the 95% confidence interval from the spread between them. Every cell in the last row is an average across the six benchmarks, each counting once — so Δ mean thinking there is the mean of the six reductions, not the ratio of the two token figures beside it, which is 41.9%. On τ²-bench the thinking tokens are per task, summed across the rollout's turns, rather than per response. Column definitions are in Evaluation methodology.
Every one of the six gets shorter. The deepest mean cuts are RealWorldQA (−51.0%), HMMT Nov 25 (−44.5%) and AIME 2026 (−44.4%); the shallowest τ²-bench (−22.5%) and AA-LCR (−28.0%). The two shallow rows are the two where the base model was already brief — 2,111 thinking tokens on τ²-bench and 1,510 on AA-LCR, against 28,653 on LiveCodeBench. There is less to cut when a model is not overthinking in the first place.
On accuracy, most of the results are within noise: on five of the six the gap is smaller than the run-to-run spread, and τ²-bench is flat to within two tenths of a point. RealWorldQA is the exception — at 765 questions its intervals are tight enough, ±0.50 and ±0.43, that a 0.95pp gap is a real one. The largest gap is AA-LCR's 1.88pp, which its own ±1.54 and ±2.60 do not resolve. Averaged over all six the cost is 1.03pp.
Truncation — responses whose thinking never closes, which score zero — goes from 0.05% to 0.00%, equal-weight across the six benchmarks.
Accuracy against the thinking-token budget, both models
Read left to right, this is the question a deployment actually asks: for a given token budget per question, how much accuracy do you get? ThinkingCap's curve sits above the original model's through the whole middle of the range — the same accuracy for a smaller budget — and the two converge once the budget is wide enough that neither model is constrained by it.
Evaluation methodology
Reasoning models are noisy to evaluate. At the recommended temperature of 1.0 both the content and the length of a response move run to run, so a single seed tells you very little. Everything below is multiple seeds per benchmark, with accuracy reported as the mean and a 95% confidence interval across seeds.
Both sides are the same run. The original model and ours were evaluated by the same harness, on the same hardware, with identical sampling and serving settings. The original model was measured by us, not quoted from published figures. The configuration:
- Harness — our internal evaluation suite, running both models on vLLM 0.31.0 without speculative decoding.
- Hardware — 4× NVIDIA H200, tensor parallel 4. The bf16 weights are about 360 GB.
- Context — generation capped at 253,952 tokens per response; 131,072 on AA-LCR and 65,536 per turn on τ²-bench, because their documents and multi-turn transcripts occupy the rest of the window.
- Sampling — thinking on at
reasoning_effort=xhigh, the chat template's own default; temperature 1.0, top_p 0.95, top_k 20, min_p 0.0. Ours inherits Qwen's recommended preset unchanged, so no part of the comparison is a decoding change. - Accuracy — the mean across seeds with a 95% t-interval, which says how far the answer moves when only the sampling seed changes.
- Seeds — 32 on AIME 2026; 16 on HMMT Nov 2025; 8 on AA-LCR, RealWorldQA and τ²-bench; 4 on LiveCodeBench v6.
- Question sets — the complete set on every benchmark: AIME 2026 (30 problems), HMMT Nov 2025 (30), LiveCodeBench v6 (175), AA-LCR (100), RealWorldQA (765) and τ²-bench (278 tasks across airline, retail and telecom).
- Scoring — exact match on the extracted answer, except LiveCodeBench, where the generated code is run against the problem's tests, and τ²-bench and AA-LCR, which are Gemini-judged. τ²-bench also runs against a Gemini-simulated user and is reported as the unweighted mean over its three domains.
What the columns mean
- Seeds — independent sampling runs of the benchmark.
- Accuracy (base) / Accuracy (ours) — as above; Δ accuracy is the difference between them, in percentage points.
- Δ mean thinking — the headline: the change in thinking tokens, totalled across the benchmark and divided by its question count, so a long question weighs more than a short one.
- Thinking tokens are the tokens before the reasoning closes, answer tokens those after it. Each is written
base → ours, and the Δ beside one is that benchmark's own percentage change. The median columns say the same thing about the typical question rather than the average one. - Truncated % — the share of responses whose thinking never closed, which score zero.
Get the model
You can find the model here: bottlecapai/ThinkingCap-Qwen3.8-Flash-Next
It is released under the base model's own licence, the Qwen Community License 1.0, whose conditions carry over — including the separate licence Qwen requires for Model-as-a-Service and AI work-assistant businesses.
Contact us
Have any questions, or a workload you want this run against?
Send us a mail at hey@bottlecapai.com
ThinkingCap PRO. Our enterprise edition cuts thinking tokens by 60% on average and up to 75.8%, so your team gets faster answers and lower inference costs at scale. Contact us at enterprise@bottlecapai.com.
