RTX 5060 Ti 16 GB as an LLM server: 69 to 407 tokens per second, and the thermal wall
A consumer GeForce card in an enterprise chassis is an awkward combination that plenty of people are quietly running. The card has no telemetry the server's management controller understands, so the fans stay blind to it. We measured what that costs, and what the card actually delivers, across two sessions in August 2026.
The short version: a single user gets 69 tokens per second from a 9B model, thirty-two concurrent users get 407 tokens per second in aggregate, and if you run it hard on the chassis default cooling it hits 83 °C and engages hardware thermal slowdown. That last number is the one that changes how you deploy.
That last part has a mechanism, and it is easier to see than to describe. There is an interactive airflow model at the end of this post: pull the fan duty down and watch the card start breathing its own exhaust. The numbers between here and there are what it is calibrated against.
What was measured, and on what
An NVIDIA GeForce RTX 5060 Ti (GB206, 16 GB) in slot 1 of a Dell PowerEdge R740, dual-socket, Debian 13. The card is natively a PCIe x8 part. The model under test throughout is Qwen3.5-9B; the sub-2B ladder used separate weights.
All prompts were synthetic. GPU telemetry was recorded at 500 ms intervals — 33,391 samples in the first session — because a coarser cadence turned out to miss the thing that mattered. Percentiles are reported only where the sample count reached 20. Quality was guarded on every profile with a fixed golden-output set plus near-context-limit sentinels; a profile that produced garbage was recorded as failed regardless of how fast it ran.
Engine comparison: the primary question
Four serving stacks, same base model. The llama.cpp and Ollama rows share a byte-identical Q6_K GGUF, so that pair isolates wrapper overhead exactly. The Q4_K_M-versus-AWQ comparison is the cross-engine bridge: different bit widths, both quality-guarded.
| Engine and profile | Quant | Single-stream decode | TTFT p50 | Concurrent aggregate |
|---|---|---|---|---|
| llama.cpp b10318 | Q4_K_M | 69.4 t/s | ~430 ms | — |
| llama.cpp b10318 | Q6_K | 55.6 t/s | 451 ms | 105 t/s @ 4 |
| Ollama 0.32.6 (same weights) | Q6_K | 48.1 t/s | 1399 ms | — |
| vLLM 0.26.0, eager | AWQ 4-bit | 23.1 t/s | 214 ms | — |
| vLLM 0.26.0, CUDA graphs | AWQ 4-bit | 65.2 t/s | 203 ms | 407 t/s @ 32 |
Three things fall out of this table.
For one user, llama.cpp at Q4_K_M wins outright. 69.4 t/s with clean quality guards and 5.3 GiB of weights. Q6_K trades about 20 % of that speed for higher-precision weights and remains the quality reference.
Ollama costs about 13 % for convenience. On byte-identical weights it returned 48.1 t/s against raw llama.cpp's 55.6, and its time-to-first-token was roughly three times longer. That is wrapper plus an older embedded runner, not a model difference — the weights were the same file.
vLLM is a concurrency engine, not a latency engine. Single-stream it is slower than llama.cpp. Under load it is roughly four times faster in aggregate. Choosing between them is really a question about how many people are talking to the thing at once.
The concurrency curve, and where it stops paying
| Streams | Aggregate output | Gain over previous | TTFT p50 |
|---|---|---|---|
| 1 | 59.4 t/s | — | 203 ms |
| 2 | 105.9 t/s | +78 % | 365 ms |
| 4 | 181.0 t/s | +71 % | 726 ms |
| 8 | 287.8 t/s | +59 % | 1180 ms |
| 16 | 384.9 t/s | +34 % | 1623 ms |
| 32 | 407.1 t/s | +6 % | 2077 ms |
vLLM reported a 77.5k-token KV cache at startup, holding 2.36× max concurrency at the full 32k per request, with weights at 8.42 GiB and KV at 2.62 GiB. A GPU memory utilisation setting of 0.84 kept the reserve intact with no out-of-memory events.
One inconsistency worth naming rather than smoothing over: the engine table above reports vLLM single-stream at 65.2 t/s, while the first point of this curve reads 59.4. They come from two different runs — a dedicated single-stream profile and the first rung of the concurrency sweep — with the same engine, model and context. The two agree on time-to-first-token to the millisecond and differ by about 9 % on decode. We have no explanation for that gap in the record, so treat vLLM's single-stream figure as "around 60–65 t/s" rather than either number precisely.
For comparison, llama.cpp's own continuous batching went 46.5 t/s at one stream, 74.8 at two, 105.1 at four — a 2.26× scale-up and roughly four times below vLLM's ceiling. That gap is the entire argument for running two different engines.
The other half: reading is cheap, writing is not
Everything above is output throughput — decode, the tokens the model writes. The input side behaves completely differently, and if you size a deployment on decode numbers alone you will get it wrong in both directions.
| Phase | Throughput | Context |
|---|---|---|
| Prefill (reading the prompt) | 1,853.8 t/s | 32k profile |
| Prefill | 1,441.3 t/s | 64k profile |
| Decode, short context | 55.6 t/s | near-empty cache |
| Decode, 28.5k tokens in cache | 45.1 t/s | n = 22 |
| Decode, 57k tokens in cache | 38.1 t/s | 64k profile |
Prefill runs about 33× faster than decode. That ratio is not a quirk of this card — it is the difference between a phase that multiplies whole matrices and a phase that emits one token at a time, re-reading the entire attention cache to do it. Reading is compute-bound and the GPU is good at it; writing is memory-bound and the GPU mostly waits.
The practical consequence is that prompt length is nearly free and output length is what costs you. An 8,000-token RAG prompt adds roughly four seconds of prefill; asking for 2,000 tokens of output costs around forty. If you are trying to make something feel faster, cut what you ask the model to write, not what you give it to read.
And decode gets slower as the conversation grows
This is the caveat on every headline decode figure in this post, including ours: 69 t/s is a number from a short prompt. The same model answering the twentieth turn of a long conversation, or reasoning over a large retrieved context, is meaningfully slower — attention reads scale with what is already in the cache. Quote the empty-cache number if you like, but size capacity on the loaded one.
Optimisations: mostly disappointing, and one that mattered
| Axis | Result | Verdict |
|---|---|---|
| FlashAttention on | Throughput normal at 56 t/s and the pre-checks passed, but all six near-limit sentinels failed and the following golden emitted garbage — state corruption at long context. Still corrupting on a newer engine build. | Quarantined |
| Quantised KV cache (q8_0) | The build exits at startup when asked for quantised KV with FlashAttention off, and the KV buffer never engages. It depends on the option that is quarantined. | Blocked |
| Speculative decode (0.6B draft) | Engaged correctly, drafts accepted, decode unchanged at 55.6 t/s. Draft weights cost an extra 610 MiB for nothing. | No gain |
| Q4_K_M against Q6_K | 69.4 versus 55.6 t/s, 4,861 versus 6,307 MiB of weights, quality guards clean on both. | Adopted |
| CUDA graphs (vLLM) | 65.2 versus 23.1 t/s eager — 2.8× for a configuration flag. | Always on |
| Constrained JSON decoding | Schema-constrained output ran at 72.3 t/s against 72.4 unconstrained. Both produced 100 % valid JSON on this task. | Free |
Three of the six went the wrong way, which is the normal ratio and the reason for measuring rather than reading forum advice. FlashAttention is the cautionary one: it looked fine on throughput and on the first quality check. Only the near-context-limit sentinels caught it, and they caught it six times out of six. A benchmark that stopped at "tokens per second and one sample output" would have shipped a config that quietly corrupts long conversations.
The thermal wall
This is the finding that changed a deployment decision rather than a configuration flag.
Under a sustained 20–30 minute soak at roughly 180 W and 100 % utilisation, on the chassis's default cooling, the GPU climbed to a plateau and reached 83 °C with hardware thermal slowdown engaged. Out of 33,391 samples, 29 carried a thermal flag — 27 reporting hardware thermal slowdown and 2 a software thermal limit.
The reason this matters more than the raw number: a 20-second live polling cadence saw only 80 °C. The 83 °C spikes and the slowdown flags happened between samples. The stop condition existed and was never triggered in real time — it was found afterwards, in the 500 ms recording. Any thermal gate that depends on someone watching a terminal is not a gate.
The management controller cannot see this coming. It has no telemetry from a GeForce card, so it runs the fans on a profile that assumes the slot is doing nothing interesting.
The mechanism underneath is recirculation: the card's exhaust leaves at the long edges, and if the chassis crossflow is too weak to carry it away it loops back into the intake a few centimetres below. Intake temperature climbs, core follows. The model at the end of this post lets you drive that directly — hold 180 W, drag the duty to 20 %, and watch the re-ingested figure climb.
The intervention is straightforward once you know to make it. Raising the slot's airflow target to 500 LFM held the card at 73 °C at 145 W under later load. A subsequent full session run with a protective 620 LFM floor produced 2 thermal-flag samples for the entire session, against 29 on default cooling, with a maximum of 81 °C and no driver errors.
So: an unattended sustained-serving profile on a consumer card in this chassis should not run on default cooling. Either a static airflow floor or something that adjusts it in response to GPU temperature is a precondition, not a refinement.
One firmware detail worth writing down
Setting the slot airflow through the management controller's Redfish API has
an ordering trap. Writing the custom airflow value while the mode is still
Automatic returns HTTP 400 — the firmware
refuses a value it considers inapplicable. Writing mode and value in a
single atomic request returns 200 and applies immediately.
The intuitive order — set the value first so the mode flip cannot leave the slot in a custom mode with a zero target — simply does not work here. The atomic write removes that hazard anyway, because there is no intermediate state to be caught in.
Energy: measured, but not yet publishable
We did aggregate energy per thousand output tokens, and the numbers are omitted here on purpose. They were computed net of the ready-idle baseline — so they describe the extra energy a request costs above an idle card, not board power — and, more importantly, the single-stream and concurrent figures were integrated differently: per-request for one stream, whole-block for many. Those two methods do not produce comparable numbers, and the headline ratio we could have quoted would have folded a real batching effect together with an artefact of how the integral was taken.
The physics still points one way — amortising a largely fixed draw across concurrent streams should lower energy per token — but "should" is not a measurement, and this post is not the place to put a number we cannot defend. It will be re-derived with one consistent method and published then.
Two 30B-class models on a 16 GB card
Both of these were expected to be impossible and both ran.
A 30B dense model at IQ3_XXS quantisation loaded entirely onto the GPU — all 53 layers, 11,457 MiB of weights plus 1,024 MiB of KV cache, leaving 3,580 MiB free and clearing the memory reserve. First inference was coherent. It needs a newer engine build than the production one, and that newer build regresses a different feature, so it stays an experiment rather than a profile.
A 30B mixture-of-experts model ran by keeping its experts in host RAM and only attention on the card: roughly 2.4 GiB of VRAM against 18,644 MiB in system memory, decoding at 23–26 t/s with a 3.5-second time to first token and clean quality checks. For a machine with plenty of RAM and one modest GPU, expert offload turns "does not fit" into "fits, slowly".
What went wrong
Recorded because a benchmark that only reports its successes is marketing.
- The harness needed several rounds of fixes before it produced trustworthy numbers: a user-id failure under a non-root account, the model's own reasoning mode silently consuming the token budget, one engine's OpenAI-compatible endpoint streaming empty responses, and a readiness race during CUDA graph capture. The broken runs were kept rather than deleted.
- The 83 °C event should have stopped the soak in real time and did not, because the live polling cadence was too coarse. That is a process failure, not a hardware surprise.
- Two owner-requested models could not be tested at all in the first session: one used an architecture the pinned engine did not implement, and the other had no artefact that could be pinned to a specific version. One of them ran in the second session, on the newer build.
If you are considering this card for local inference
For a single user on a 9B model, expect around 69 tokens per second with a 4-bit quantisation and roughly 5 GiB of weights. That is comfortably faster than reading speed and it fits on a card that is not expensive.
For a handful of concurrent users, switch engines rather than tuning the one you have: about 385 tokens per second aggregate at 16 streams, and stop there, because past the knee you are buying latency with no throughput.
For anything sustained and unattended, solve cooling before you tune anything else. The card will silently throttle itself, the management controller will not tell you, and the only reason we know the exact temperature is that something was sampling twice a second.
An airflow toy for the thing that actually bit us
The thermal result above is a number. This is the shape behind it: a top-down and a side section through the chassis, with the card in riser 1, backplate to the lid and fans facing the board. Drag the fan duty down while holding 180 W and watch the exhaust linger at the card's long edges and get pulled back into its own intake.
It is an advection toy, not CFD. Geometry is scaled from the chassis and the readouts are calibrated so that the baseline reproduces the measured numbers — 23 °C inlet, 31 °C exhaust, 80 °C core at 180 W with the fans near 30 %. What it is good for is showing which direction the knobs move things, and why a slot airflow floor changes the outcome.
Try: hold 180 W and pull fan duty to ~20 % — exhaust lingers at the card's long edges and gets re-ingested, intake temperature climbs. Push duty past 70 % and the crossflow strips it into the vented riser-2 bay and the bracket line. Geometry is scaled from the chassis (front → rear: drives, 6-fan wall, CPU/DIMM banks, riser field); preheat and ΔT figures are order-of-magnitude. The section slices vertically through riser 1: ⊙ rings mark lateral exhaust leaving the plane toward the riser-2 bay; below ~40 % duty part of it wraps around the tail and re-enters the intake lane.
About the simulator
It is a two-dimensional advection toy, not computational fluid dynamics. Several hundred particles carry a scalar "heat" value through a velocity field assembled by hand from the chassis layout: the fan wall accelerates them, the DIMM banks and CPU heatsinks slow them and add heat, the card's fans pull from whatever is nearby, and the rear wall either vents them or steers them to the nearest opening. There is no pressure solver, no turbulence model, and no conservation of anything.
What makes it worth showing anyway is that the readouts are calibrated against the measurements in this post. At the default 30 % fan duty and 180 W it settles at roughly 80 °C core and 31 °C exhaust against a 23 °C inlet — the numbers we actually recorded. Not just at start-up: left running, it stays there. So the baseline is real, and the sliders show you which direction things move from it, at roughly the right magnitude.
The behaviour it exists to make visible is the recirculation loop: below about 40 % duty, a fraction of the card's own hot exhaust wraps around the tail and re-enters its intake, so the intake temperature climbs and the core follows. That is the mechanism behind the thermal event above, and it is exactly what raising the slot airflow target breaks.
What it cannot tell you: absolute temperatures for a different chassis, a different card, or a different riser; anything about acoustics; or what happens with the lid off. Treat a reading as "this direction, about this much", never as a prediction.
It runs entirely in this page — no libraries, no external code, nothing fetched. If you have asked your system for reduced motion it renders a single static frame instead of animating, and the Run button is yours.