Sep 25, 2026 · 3 min read
Halve the thinking trace length, quadruple the work
Shorter reasoning is cheaper and faster, but on a server that is limited by KV cache it gets even better. Cut the length of each answer by 2.3× and the same GPU can finish 4.7× as many tasks within the same time window.
TL;DR
Same GPU, same prompt, both servers have 20 minutes to finish as many tasks as they can. Both models answer an AIME geometry problem.
- ThinkingCap-Qwen3.8-27B: 112 tasks finished
- Qwen3.8-27B: 24 tasks finished
That is 4.7× more finished work from reasoning traces only 2.3× shorter.
Want this on your own traffic? enterprise@bottlecapai.com
The bottleneck is memory, not compute
A server that is under load rarely runs out of FLOPs first. It runs out of KV cache. Every running sequence keeps a cache that grows with its length, and the server can only run as many sequences at once as fit in that memory.
A model that reasons at half the length therefore gives two separate gains. Each task needs fewer tokens, so it finishes sooner. Each sequence also takes up less cache, so more of them fit on the GPU at the same time. The server runs more tasks side by side, and each one finishes faster.
The numbers
These steps build on each other. Over the full 20 minutes, traces 2.3× shorter raise the average task parallelization 1.6× (24.9 tasks running at once against 15.1), which gives 1.6× the token throughput (511 against 319 tok/s). Each task also needs 2.3× fewer tokens, so that throughput turns into far more finished tasks.
The 20-minute average understates the gap, because the effect needs a few minutes to build. At the start every trace is still short, so both servers fit a similar number of them. The difference appears once the traces grow and ThinkingCap starts finishing requests and freeing their cache. From 300 s to the end of the run, ThinkingCap has on average 24.0 tasks generating at once against the base model’s 11.8 (2.0×), for 495 against 253 tok/s (2.0×).
What this means if you serve models
Per-token pricing understates what shorter reasoning is worth to whoever pays the GPU bill. On a saturated server a shorter trace saves its own tokens and frees memory for the next request. In our run, that meant the same GPU finished 4.7× as many tasks in the same 20 minutes, at the same cost.
Setup
- Hardware: 1× RTX PRO 6000, 8 GB KV cache
- Serving: vLLM 0.29.0, with prefix caching disabled so that no request reuses another request's cache
- Load: 64 concurrent requests, all with the same AIME geometry prompt
- Task: “Find the sum of all real numbers such that there is at least one point where the circle with radius centered at is tangent to the parabola with equation .”
- Window: 20 minutes; a task counts as finished once its full response has been returned
Contact us
If you run inference at volume, this is where shorter reasoning pays off most. We fine-tune variants for your workload and effort tier and test them on your own evals before you commit: enterprise@bottlecapai.com
