AI labs guard their inference stacks the way software companies used to guard their data centers. On October 20, DeepSeek broke that habit. The lab published DeepInfer, the serving framework that runs its production API, on GitHub under Apache 2.0, and attached something more valuable than code: numbers nobody outside the company has ever seen. A single cluster averaging 155 trillion tokens per day, a 96 percent input cache hit rate, roughly 400 milliseconds to first token, and, per a technical report covered by BigGo Finance, about $38 in revenue per GPU per hour at off-peak electricity rates.
The code itself is a strong entry in the inference-engine race. The disclosures are the event. For the first time, a frontier-scale lab has published the revenue side of its serving economics, which means anyone can now check whether DeepSeek's famous prices are a subsidy or a business.
What landed in the repository
DeepInfer is not another research toy. The README describes it as "the high-performance inference framework that powers DeepSeek's large-scale production services," and DeepSeek's own X post on October 20 called it their in-house framework running DeepSeek-V4.1-Flash at scale. The repository had passed 8,000 stars within hours of publication, per The Next Web, and by the time of writing the GitHub API shows more than 16,000.
The feature list reads like a survey of everything DeepSeek has open-sourced over the past two years, assembled into one engine:
- Kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, plus custom TileLang kernels
- Parallelism: Attention data parallelism with balanced context parallelism, MoE expert parallelism with expert load balancing (EPLB)
- Speculative decoding: DSpark, DeepSeek's semi-autoregressive draft model, with confidence-scheduled verification; the max draft length defaults to 5 tokens
- SWA bounded replay: decoder prefill replays only a bounded suffix (default: keep 140 tokens), matching the sliding-window design of V4.1-Flash
- Asynchronous scheduling: CPU batch scheduling overlapped with device execution, on by default
- Batch-invariant inference and deterministic sampling: identical outputs across different batch compositions, NVIDIA only, controlled by
DEEPINFER_BATCH_INVARIANT=1 - Structured outputs constrained to a regex, JSON schema, or EBNF grammar
Hardware support is the geopolitically interesting line: NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, first-class, in the same framework, requiring CUDA 13 on the NVIDIA side and CANN 9.2 on the Ascend side. The build needs Linux, GCC 13 or newer, and Python 3.12. The default checkpoint is deepseek-ai/DeepSeek-V4.1-Flash-1017, the October 17 refresh on Hugging Face.
DeepSeek also shipped deepseek-recipe, a set of Rust libraries with Python bindings that reproduce the exact prompt encoding of its official API. That solves a real migration pain: when a self-hosted tokenizer disagrees with the hosted one, outputs drift in subtle ways that are miserable to debug.
The parts deliberately left out
The README is unusually honest about scope. Three components required for a complete inference service are not in the release: a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas. DeepInfer is an engine plus offline inference, not a turnkey server.
That omission is a design statement. Disaggregated serving, where prompt processing and token generation run on separate GPU pools, is the architecture every hyperscaler converged on, and the routing, cache-aware scheduling, and failure handling around it are deployment-specific. DeepSeek is handing operators the part that took them years to tune and leaving them the part that is mostly integration work. The acknowledgment section also credits where the engine came from: DeepInfer was "originally built primarily on vLLM," with additional inspiration from SGLang and TensorRT-LLM. The vLLM project publicly thanked DeepSeek on X and said it would fold ideas back into its own V4.1-Flash support, per The Next Web.
One caution before anyone cites the throughput charts: the published Pareto curves were measured by DeepSeek, on 4 SM100 GPUs for prefill and 64 for decode, each within one scale-up domain. Bloomberg's Saritha Rai noted that no outside party has benchmarked the claimed gains yet. Treat every number below as a vendor claim until reproductions land.
Why This Matters: the $38 arithmetic
Here is the section the rest of the industry will actually quote. DeepSeek disclosed two production figures at once: 155 trillion tokens per day through a single cluster, and roughly $38 in revenue per GPU-hour. Neither number alone is very informative. Together they constrain the blended price per million tokens, and that is a falsifiable claim about DeepSeek's business.
The arithmetic is simple. Let the cluster hold N GPUs. Daily revenue is $38 x 24 x N. Daily tokens are 155 trillion. The blended price per million tokens is therefore:
blended $/M = (38 x 24 x N) / 155,000,000 = 38 x 24 x N / 1.55e13
Because DeepSeek did not publish the cluster size, we can invert the question and tabulate what the two disclosures imply at plausible sizes. These are our labeled calculations, not DeepSeek figures:
| Assumed cluster size | Revenue per GPU-day | Blended $/M tokens |
|---|---|---|
| 25,000 GPUs | $912 | about $0.15 |
| 50,000 GPUs | $912 | about $0.29 |
| 100,000 GPUs | $912 | about $0.59 |
Every row uses the same two disclosed numbers; only the assumed fleet size changes. The 50,000-GPU row lands at roughly $0.29 per million tokens blended across input and output, cache hits and misses. That is the row worth staring at, because DeepSeek's public API prices for its Flash-class models have long sat in the tens of cents per million input tokens, and a 96 percent input cache hit rate means the average request genuinely does look like that. The disclosures are internally consistent with the price sheet, which is the whole point: this was not a marketing number that contradicts the billing engine.
Now the engineer's follow-up question. Is $38 per GPU-hour profitable? An SM100-class GPU rented on the merchant cloud market clears in the low double-digit dollars per hour, and owned hardware amortized over three years plus power lands lower, which is why DeepSeek framed the figure at off-peak electricity rates. If your serving cost per GPU-hour is meaningfully under $38, a 96 percent cache hit rate turns serving tokens into a high-margin commodity rather than a loss leader. That is the same conclusion we reached from the other direction in our DeepSeek V4.1-Flash architecture analysis: the model exists to make KV cache bytes cheap (890 bytes per token, roughly a quarter of V4-Flash), and cheap cache makes cheap serving possible. DeepInfer is the other half of that plan, the scheduler that keeps those cached bytes hot across a fleet.
The 96 percent hit rate deserves a data scientist's skepticism. It implies almost every incoming prompt is a small delta on something already in cache, which is exactly what agentic loops look like: a stable system prompt, a growing tool history, and one new tool result per turn. It is a workload-shaped statistic, not a physics constant. A company whose traffic is 200 first-queries-per-day from a search box will not reproduce it, and per our guide to reading AI benchmark claims like a skeptic, the number to watch for is the miss-path latency, not the hit-rate average.
Who should actually adopt this
Three audiences, in descending order of fit.
First, GPU clouds and hosts that already serve DeepSeek models. They can now run the same kernels, the same DSpark draft-verify schedule, and the same bounded-replay trick DeepSeek uses, and compete on cost-per-token with the API itself. Bloomberg's framing, echoed across coverage, is exactly this: enterprises running the models on their own hardware may cut compute costs.
Second, anyone serving MoE models on domestic Chinese accelerators. A frontier lab shipping day-one Ascend 950 support inside a production framework is a stronger signal to Ascend's buyers than any roadmap slide.
Third, self-hosters comparing local inference against API pricing. The honest read: DeepInfer raises the self-hosting ceiling but not the floor. Batch-invariant and deterministic sampling are NVIDIA-only, the missing load balancer is real work, and the engine currently supports exactly one model, V4.1-Flash. Unless your volume approaches cluster scale, a hosted endpoint stays the rational choice; what changed is that the gap now has published numbers on both sides.
Outlook
The strategic bet is legible. DeepSeek's models are already the price floor for open weights; open-sourcing the serving layer pushes that floor into infrastructure, where every cloud that adopts DeepInfer becomes a distribution channel for DeepSeek architectures. It also follows a week of pressure: the US ordered civilian federal agencies to remove DeepSeek products from their systems the day before this release, per The Next Web, and the lab published Huawei chip tooling earlier this month as an explicit CUDA alternative. Software that runs on Ascend as well as Blackwell is a hedge with a marketing budget.
For builders, the near-term signal is reproduction. The README ships a run_sweep.sh benchmark harness, so within weeks we should see whether independent operators on SM100 hardware approach DeepSeek's Pareto curves, and whether anyone ports the engine to consumer-proximate silicon, which third-party repositories circling the launch are already attempting for older architectures. If the curves hold under outside testing, the $38 GPU-hour stops being a press release and becomes a public cost model the whole industry gets priced against. If they do not, we will learn something just as useful: how much of DeepSeek's advantage was the framework, and how much was the 96 percent cache that only its own traffic pattern produces.
Either way, the direction of travel is set. Serving stacks are becoming as open as weights, and the interesting competitive line has moved from "which model" to "which dollars per million tokens, honestly stated." Power budgets tell the same story from the utility side, as we covered in our analysis of Microsoft's tokens-per-megawatt framing for Maia. The labs that publish their serving math first are the ones confident the math works. DeepSeek just published the clearest numbers anyone has seen. That confidence, more than any kernel in the repository, is what competitors now have to answer.