Skip to main content
← Back to Blog

DeepSeek-V4.1-Flash-1017 and DeepInfer: why the model has its own engine

• Dataxad Team

DeepSeek open-sourced DeepInfer, the engine behind DeepSeek-V4.1-Flash-1017 in production. The benchmarks, the per-GPU numbers, and why a model-specific engine wins.

DeepSeek just open-sourced DeepInfer, the inference engine it runs in production for DeepSeek-V4.1-Flash. It shipped on October 20, 2026, under Apache 2.0, and it supports exactly one model: the new DeepSeek-V4.1-Flash-1017 checkpoint. That narrow focus is the whole point. V4.1 Flash is an unusual architecture, and a general-purpose engine leaves a lot of its speed on the table.

Below: what changed in the 1017 checkpoint, what DeepInfer actually delivers per GPU, and why a model-specific engine beats a general one for this model. One note up front — Sam Jacobson and the Dataxad team read the sources on release day, but we have not run DeepInfer ourselves. Every performance number here is DeepSeek’s.

What’s new in DeepSeek-V4.1-Flash-1017

1017 is a point release of V4.1 Flash, published on Hugging Face on October 17 under the MIT license — and it’s a meaningful one. DeepSeek reports gains across reasoning, coding, agentic work and vision, plus something that matters just as much for your bill: it finishes tasks faster and with shorter output. Fewer output tokens per task means a lower cost per task, even at the same price per token.

The basics haven’t changed: text and image input, a 1M-token context window, routed experts stored in FP4 and the rest of the weights in FP8. There’s still no Jinja chat template. Instead, DeepSeek ships a Python reference encoder and deepseek-recipe, a set of Rust libraries with Python bindings that translate Chat Completions, Messages and Responses API requests into DeepSeek’s prompt format, and parse the output back into responses, tool calls and thinking blocks. If you self-host, use it — it’s the easiest way to make your deployment behave like the official API.

BenchmarkV4.1-Flash-1017V4.1-FlashGPT-5.6 SolOpus 5
HLE (no tools)42.636.844.556.3
Terminal-Bench 4.042.331.239.951.8
DeepSWE v1.177.074.273.074.0
ProgramBench29.120.323.037.0
Automation-Bench v1.0.660.554.845.850.3
Apex Bench v1.172.164.654.365.8

Source: DeepSeek’s model card. Terminal-Bench 4.0 was run in the Claude Code harness and DeepSWE in Mini-SWE-V2. The headline is Terminal-Bench, up 11.1 points over the previous Flash checkpoint. Opus 5 still leads on HLE, Terminal-Bench and ProgramBench, but 1017 tops this table on DeepSWE and Automation-Bench — remarkable for a “Flash” model.

What DeepInfer is (and isn’t)

DeepInfer is an inference engine, not an agent harness. A harness like Claude Code decides which tools to call and when. An engine sits underneath: it schedules requests onto the GPUs, manages the KV cache (the attention state each request carries), and launches the kernels. DeepSeek announced it on X as its in-house framework and said inference code for future DeepSeek models will land in the same repo.

"Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency." (@deepseek_ai, October 20, 2026)

DeepInfer began life as a vLLM fork, with ideas borrowed from SGLang and TensorRT-LLM. The repository lists:

  • DeepSeek’s own kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, plus custom TileLang kernels.
  • Attention data parallelism and balanced context parallelism; MoE expert parallelism with EPLB (expert-parallel load balancing).
  • DSpark draft tokens with confidence-scheduled verification.
  • SWA bounded replay.
  • An asynchronous scheduler that builds the next batch on the CPU while the GPU runs the current one.
  • Batch-invariant inference and deterministic sampling (NVIDIA only).
  • Structured output from a regex, JSON schema or EBNF grammar.

The numbers

On NVIDIA’s SM100 GPUs (the Blackwell B200/GB200 class), DeepInfer decodes roughly 26,000 output tokens per second per GPU while still giving each user about 100 tokens per second. That’s measured on 64 GPUs in a single NVLink domain, with 64-way expert parallelism and a 131,072-token context. The 64 GPUs hold the 384 routed experts plus 64 redundant copies, which EPLB uses to spread load away from the busiest experts.

Lighter load buys more speed per user: about 180 tokens/s at 128 requests per GPU, 250 at 64, and 325 at 8. At the heavy end — 256 requests per GPU — the 64-GPU cluster is pushing around 1.7 million output tokens per second. (We read these values off DeepSeek’s chart, so treat them as approximate.)

DeepInfer decode chart: output throughput per GPU against speed per user. 256 requests per GPU give about 26,000 tokens per second per GPU at about 100 tokens per second per user. 8 requests give about 325 tokens per second per user.

Decode chart from the DeepInfer repository (Apache 2.0). Point labels are requests per data-parallel rank.

Prefill — reading the prompt — tops out near 89,000 uncached input tokens per second per GPU, measured on 4 GPUs. A 4,096-token batch per GPU finishes in about 70 ms at ~60,000 tokens/s. At 16,384 tokens you get ~82,000 tokens/s in about 200 ms. Beyond that, returns flatten fast: 65,536 tokens takes ~730 ms for ~89,000 tokens/s. For an interactive service, the 16K point is the sweet spot.

DeepInfer prefill chart: non-cached input throughput per GPU against latency per prefill step. Throughput rises from about 60,000 tokens per second at 4,096 tokens to about 89,000 at 65,536 tokens.

Prefill chart from the DeepInfer repository (Apache 2.0). Point labels are prefill tokens per data-parallel rank.

Why a dedicated engine wins for this model

General engines have to support hundreds of architectures, so they take general code paths — and V4.1 Flash is full of parts those paths handle slowly or not at all. The model’s config.json tells the story:

  • 384 routed experts, 6 active per token.
  • FP4 experts with UE8M0 scale factors.
  • A single KV head, 512 dimensions wide.
  • A 128-token sliding window, plus compressed sparse attention with a two-level indexer.
  • Engram n-gram lookups.

DeepInfer has a specific answer for each.

1. Kernels built for these exact shapes

FlashMLA handles attention, DeepGEMM the FP8 and FP4 matrix math, and DeepEP the all-to-all traffic that ships tokens to experts on other GPUs. All three were written for this model family on SM100. A general engine will often dequantize the FP4 weights or fall back to a slower attention path for the unusual head layout.

2. Speculative decoding that knows when to bother

DSpark drafts several tokens ahead and the full model verifies them in one pass. In DeepSeek’s decode benchmark — 3 draft tokens, 75% per-token acceptance — that works out to τ = 3.25, meaning each full forward pass yields 3.25 tokens on average. DeepInfer adds a confidence estimate that decides which drafts are worth verifying, so it wastes less work on drafts that would be rejected anyway. This is the single biggest reason per-user speed is so high.

3. SWA bounded replay

The decoder layers use sliding-window attention that only looks back 128 tokens. So when a request moves from prefill to decode, the decoder doesn’t need to replay the whole prompt — DeepInfer replays just the last 140 tokens (the window plus a margin for backtracking). On a 131K-token prompt, that eliminates almost all of the decoder’s prefill work. A general engine has no way of knowing this shortcut is safe.

4. Expert load balancing at cluster scale

Spread 384 experts over 64 GPUs and some experts get hammered while others idle. EPLB adds redundant copies of the hot experts — 64 in decode, 32 in prefill — and splits traffic between them. Without it, the busiest GPU sets the pace for the whole cluster.

5. Reproducible runs

In batch-invariant mode, a request’s output doesn’t change depending on what else shares its batch, and a fixed seq_seed makes sampling deterministic. That’s gold for evaluations and for debugging agents: replay a failed run and you get the same tokens back.

The other half of the stack: DeepSeek Harness

DeepSeek now ships a dedicated stack at both levels: DeepInfer runs the model, and DeepSeek Harness (dsh) runs the agent. dsh launched in August 2026 under MIT, built around the idea that everything — the model adapter, the tool registry, even the agent loop — is a plugin.

The same day 1017 landed, DeepSeek released dsh v0.2.1-rc.2 with 1017 support and the new DeepSeek Search and Fetch APIs (switch them on under Web Search settings). The rc.1 build a day earlier added a plugin index, Claude Code and Codex plugin packs, and an experimental Claude Code Mods compatibility layer.

It’s still a release candidate, so kick the tires before you rely on it. And note that DeepSeek’s own agent scores come from other harnesses — the harness changes the score, so benchmark the options on your own tasks.

The local dedicated engine: DwarfStar (ds4)

If DeepInfer is the datacenter answer, DwarfStar (antirez/ds4) is the answer for hardware you can actually own. Salvatore Sanfilippo (antirez) writes it in C with Metal, CUDA and ROCm backends. It’s deliberately not a general GGUF runner — it only loads the GGUF files the project itself produces.

It’s keeping pace with DeepSeek, too. On October 9 and 10 the project added V4.1 DSpark drafting on Metal, CUDA and ROCm, and on October 19 its download script switched the default V4.1 Flash Q2 file to the 1017 checkpoint (DeepSeek-V4.1-Flash-1017-IQ2_XXS-Q2_K-imatrix.gguf).

What you need, per the ds4 model guide:

  • Q2: a 341 GiB file with 152 GiB of main weights. Q4: 483 GiB, with 294 GiB of main weights.
  • One 128 GB Mac or one DGX Spark: run with --ssd-streaming and the engine pages experts in from SSD as needed.
  • Two 128 GB Macs or two Sparks: Q2 fits fully in memory with tensor parallelism over RDMA, about 81 GiB of main weights per machine.
  • Either way, a fast SSD: both files carry 189 GiB of Engram tables that are always read straight from disk.

DwarfStar and DeepInfer are the same idea at two scales: an engine built for one model family that exploits what makes that model different — DSpark drafts, compressed attention, Engram tables. ds4 hasn’t published V4.1 speed numbers yet, so measure on your own workload before you buy hardware around it.

Who can run DeepInfer today

Right now DeepInfer runs only on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs, and this release is offline (batch) inference only. You’ll need Linux, GCC 13+, Python 3.12, and CUDA 13 or CANN 9.2. Three pieces of a full serving stack are missing: a KV cache system, prefill/decode disaggregation, and load balancing across data-parallel replicas — though DeepSeek says most of the disaggregation plumbing is already in the engine.

So today it’s a tool for batch jobs, evaluations and research teams with Blackwell clusters, not a workstation engine. At roughly 763 billion parameters (per Hugging Face), the full-precision checkpoint won’t fit in a single GPU or Mac — that’s DwarfStar’s territory.

Even if you never run it, the repo is worth reading. It shows how DeepSeek keeps its API so cheap, and it hands the vLLM and SGLang teams a reference implementation for every one of these optimizations.

What we recommend

  • Just want to use the model? Call the DeepSeek API, or your existing engine if it supports V4.1 Flash. Use deepseek-recipe for prompt encoding so your parsing matches the official API.
  • Building an agent on it? Try dsh v0.2.1-rc.2 — or keep your current harness and compare the two on your own tasks.
  • Batch evals on Blackwell? Use DeepInfer. Its eval/ folder already covers AIME 2026, HLE and GPQA Diamond with Pass@1 scoring.
  • One 128 GB Mac or Spark? Try DwarfStar with the Q2 file and SSD streaming, and measure.
  • Smaller local machine? Stick with smaller models — see our 8 GB VRAM model guide and our Strata and oMLX memory guide.

DeepInfer confirms a clear 2026 trend: the fastest way to serve a frontier open-weight model is an engine written for that model. If you’re deciding which model and engine fit your workload, talk to Dataxad.

Need help implementing this?

Book a Consultation