DeepSeek Open-Sources the DeepInfer Inference Framework It Runs in Production

On October 20, 2026, DeepSeek published the DeepInfer inference framework that powers its large-scale production services on GitHub under the Apache 2.0 license. This is the actual serving stack behind DeepSeek-V4.1-Flash, not a cleaned-up demo. And it landed one day after the US Department of Homeland Security ordered federal civilian agencies to remove DeepSeek products from their systems, which tells you something about how little the open-source community cares about geopolitics when there is good code on the table.

An inference engine is the software that serves an already-trained model to users. It splits the model across chips, batches incoming requests, and decides how fast answers come back. It is also a huge part of what an AI company pays per answer, which is why most labs treat their serving stacks as crown jewels. DeepSeek has now put its crown jewels on GitHub.

What the DeepInfer inference framework actually does

DeepInfer covers offline inference: optimized kernels, a model and sampler runtime, and the plumbing to spread a model across dozens of chips. The kernel list reads like a greatest-hits of DeepSeek's earlier open systems work: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, and custom TileLang kernels. For parallelism it uses attention data parallelism, balanced context parallelism, and mixture-of-experts expert parallelism with expert-parallel load balancing.

Decoding runs through a system called DSpark, which drafts tokens ahead and uses confidence estimates to decide when to verify them. By default it drafts up to five tokens. Outputs can also be constrained to a regex, JSON schema, or EBNF grammar, which matters a lot for developers building agents that need machine-readable answers. Two features that keep results identical across runs, batch-invariant inference and deterministic sampling, currently work only on Nvidia hardware.

DeepSeek shipped a companion called deepseek-recipe alongside it, an MIT-licensed component that converts incoming API requests into model inputs and converts outputs back the same way the official API does. So developers building a DeepSeek-compatible service now have both halves of the pipeline: the engine that runs the model and the glue code that makes it speak the same request format as production.

Two chip families, one framework

The headline for hardware people: DeepInfer supports Nvidia SM100 Blackwell GPUs and Huawei Ascend 950 NPUs on equal footing, and DeepSeek's own production figures were measured on both. Older Nvidia architectures like Hopper are not supported. Given the export-control climate, a frontier lab shipping first-class support for Chinese and American silicon in the same release is the detail the industry will remember longest.

The README is honest about the lineage: DeepInfer was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. This is not a from-scratch reinvention. It is DeepSeek's production-hardened layer on top of community foundations, which is exactly the kind of release that tends to actually get used instead of starred and forgotten.

What's in the box

• Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, custom TileLang kernels
• Parallelism: attention data parallelism, balanced context parallelism, MoE expert parallelism with EPLB
• Decoding: DSpark speculative decoding with confidence-scheduled verification
• Determinism: batch-invariant inference and deterministic sampling (Nvidia only)
• Structure: outputs constrained by regex, JSON schema, or EBNF grammar
• Companion: deepseek-recipe (MIT) for DeepSeek-compatible API request and response handling

What's missing from this release

DeepSeek is clear that this is offline inference only. There is no KV cache system, no prefill and decode disaggregation, and no load balancing across replicas. Operators who want a full online service still have to build or borrow those pieces. The benchmarks DeepSeek published were all run on Nvidia hardware, and the 155-trillion-tokens-per-day production figure is DeepSeek's own number, not independently tested. It is also worth remembering that in April 2025 DeepSeek said this engine was too tied to its internal systems to release at all. Something changed their mind, and the change is developer-shaped: the README and deployment Q&A read like documentation for people who will actually run this, not a press release.

Why developers should care

This is a rare look at how a frontier lab actually serves models at scale. The deployment Q&A that came with the code treats cost-per-answer as the core problem an inference framework has to solve, and DeepSeek claims its revenue works out to about $38 per GPU-hour at off-peak pricing. The repo crossed several thousand stars within hours of launch. Inference code for future DeepSeek models is promised to land in the same repository, which means this is the start of a public serving lineage, not a one-off dump.

If you run models in production, read the README before you read anyone's summary. And keep it in context with the broader open-source infrastructure trend: NVIDIA recently open-sourced its Run:ai GPU scheduler, and open-source tooling keeps outflanking commercial alternatives. The serving layer was the last thing the big labs held close. Now it is public too.