SC '24 — International Conference for High Performance Computing, Networking, Storage and Analysis (2024)arXiv:2408.14158 ↗
Fire-Flyer
AI-HPC & 3FS
A Cost-Effective Software-Hardware Co-Design for Deep Learning
DeepSeek-AI (Wei An, Xiao Bi, Guanting Chen, Shanhuang Chen, and 48 further authors)
This is the infrastructure paper behind DeepSeek's improbably cheap frontier models. Instead of buying NVIDIA's turnkey DGX-A100 SuperPods, DeepSeek built Fire-Flyer 2 — 10,000 PCIe A100 GPUs on a cost-optimized two-layer fat-tree network — and wrote the software to make commodity hardware perform: HFReduce (CPU-assisted allreduce that beats NCCL on this topology), HaiScale (parallelism framework), the 3FS distributed file system (later open-sourced, hitting 6.6 TiB/s aggregate reads), and the HAI scheduling platform. The result: roughly 80% of DGX-A100 performance at half the total cost and 40% less energy — the 'how they could afford it' companion to the DeepSeek model papers.
10,000
A100 GPUs
PCIe, across ~1,250 nodes
≈50%
of DGX system cost
at ~83% per-GPU performance
−40%
energy consumption
2,500 W vs 4,200 W per node
6.6 TiB/s
3FS read throughput
180 nodes · open source
Introduction
The exponential compute demands of LLMs make cluster cost the binding constraint on research. Fire-Flyer 2 is a 10,000× A100 cluster with a deliberately 'under-provisioned' network — one 200 Gbps NIC per node instead of the DGX's nine, and a two-layer instead of three — recovering the lost efficiency in software.
Four systems carry the design: HFReduce (CPU-mediated tuned to the PCIe topology), HaiScale (DP/PP/TP/EP/ parallelism engineered around the weaker interconnect), 3FS (an distributed file system), and the HAI platform (time-sharing scheduler with checkpoint-based preemption). Net result: ~83% of DGX-A100 per-GPU performance at ~60% node cost and 60% node power — about half the total system cost.
Background
The paper traces the evolution from small dense models to LLMs, where training-cluster scale — not algorithms — became the gating factor. The canonical reference design is the DGX-A100 SuperPod: 8 A100s per node fully meshed with , 9× 200 Gbps NICs, and a rail-optimized three-layer — engineered so collectives run at line rate.
Key challenges at 10k-GPU scale motivate the design: acquisition and energy cost, network congestion when storage and compute traffic share the fabric, and hardware reliability — GPU and interconnect failures become routine at fleet scale, demanding checkpointing and health-validation machinery.
Fire-Flyer 2: The Cluster
Node: 8× A100-PCIe-40GB, 2× 32-core EPYC Rome/Milan, 512 GB DDR4-3200 (16 channels), 1× InfiniBand CX6 200 Gbps NIC. Measured per GPU: TF32 107 vs 131 TFLOPS and FP16 220 vs 263 TFLOPS against DGX — 83% relative performance at 60% node price and 60% power (2,500 W vs 4,200 W).
Network: a two-zone, two-layer ; each zone is an 800-port fabric serving ≈600 GPU nodes, with limited inter-zone links and storage nodes attached to both zones. Total: 122 switches, vs ≈200 for a three-layer PCIe design and 1,320 for a DGX SuperPod at this scale — network cost ≈350 vs 4,000 (reported price units), pulling total system cost to ≈11,600 vs 23,000: half.
3.1 · The node: same silicon, different packaging
Before / after — drag to compare the two node designs
Same GPU silicon, radically different packaging: the entire cost story of the paper in one picture.
| GPUs | 8× A100 PCIe 40GB | 8× A100 SXM 40GB |
| GPU interconnect | None → NVLink Bridge pairs (600 GB/s), retrofitted | NVLink + NVSwitch, all 8 GPUs (600 GB/s) |
| CPUs | 2× 32-core AMD EPYC Rome/Milan | 2× 64-core AMD EPYC 7742 |
| Memory | 512 GB DDR4-3200 (16-channel) | 2,048 GB DDR4-3200 |
| NICs | 1× InfiniBand CX6 200 Gbps | 9× InfiniBand CX6 200 Gbps |
| TF32 GEMM (TFLOPS/GPU) | 107 | 131 | 83% |
| FP16 GEMM (TFLOPS/GPU) | 220 | 263 | 83% |
| Node price (relative) | 60% | 100% | 60% |
| Node power (W) | 2,500 | 4,200 | 60% |
3.2 · The network: two zones, two layers
The Fire-Flyer 2 cluster — click any component
≈600 GPU nodes (8× A100 PCIe · 1 NIC each)
About 600 compute nodes per zone, ≈4,800 GPUs. Each node has a single 200 Gbps NIC — the design's central bet, made viable by HFReduce's 8:1 on-node gradient reduction before anything touches the wire.
Fire-Flyer 2 as a percentage of DGX-A100
100% = the DGX-A100 reference (dashed line). Performance stays high while every cost dimension collapses.
what you keep (performance) what you pay (cost, power)
Total system cost at ~10,000-GPU scale
In the paper's reported price units (Table III), split into servers and network fabric
| Fire-Flyer 2 (2-layer fat-tree) | 122 | 350 | 11250 | 11600 |
| PCIe, 3-layer fat-tree | 200 | 600 | 11250 | 11850 |
| DGX-A100 SuperPod | 1320 | 4000 | 19000 | 23000 |
3.3 · The cluster at a glance
Accelerators
10,000× A100 PCIe
≈1,250 compute nodes × 8 GPUs; NVLink Bridges (600 GB/s per pair) retrofitted for the LLM era.
Network per node
1× 200 Gbps IB
Versus 9 NICs on a DGX-A100 — the central cost-performance bet of the design.
Network topology
2 × 800-port fat-tree
Two-layer fat-tree per zone, ≈600 GPU nodes each; limited inter-zone links; ≤1 cross-zone job.
Fabric size
122
vs 1,320 switches in a DGX-A100 SuperPod at comparable scale — 10.8× fewer.
3FS deployment
180 nodes · 2,880 SSDs
16× 15.36 TB NVMe per node, 2× 200 Gbps NICs, >20 PiB usable with mirroring.
Storage throughput
8 TB/s
Production read throughput (9 TB/s theoretical); open-source benchmark: 6.6 TiB/s.
Node power
2,500 W
vs 4,200 W per DGX-A100 node; whole cluster under ~4 MW.
Checkpoint cadence
every 5 min
Async GPU→CPU staging + 3FS batch writes (>10 GiB/s/node); max 5 minutes of work lost.
Cluster utilization
≈99%
HAI platform time-sharing with checkpoint-based preemption and resume.
Allreduce bandwidth
6.3–8.1 GB/s
HFReduce inter-node, 186 MiB messages — 2–5× NCCL on the same fabric; >10 GB/s with NVLink.
Optimizer states
CPU-staged
Optimizer/gradient state streamed to host memory asynchronously for checkpoints and reduction.
System cost
≈50% of DGX
11,600 vs 23,000 reported price units (Table III); energy −40%.
HFReduce: CPU-Assisted Allreduce
HFReduce is a topology-aware in four stages: (1) asynchronous GPU→CPU transfers of gradient chunks using the GPUs' copy engines (no kernels launched); (2) intra-node reduction on the CPUs using ; (3) inter-node allreduce of the CPU-resident sums over with a ; (4) broadcast of the result back to all 8 GPUs over PCIe.
Measured on 186 MiB messages: 6.3–8.1 GB/s inter-node bandwidth vs NCCL's 1.6–4.8 GB/s on the same fabric — while consuming ~1× the data size in PCIe traffic (NCCL's ring costs 2(n−1)/n×) and zero GPU kernel time. With the retrofitted NVLink bridges, a variant exceeds 10 GB/s. The ceiling is the EPYC PCIe root port (~37.5 GB/s) and host memory contention, bounding practice at ~12 GB/s.
HFReduce — one allreduce, step by step
1 · GPU → CPU, asynchronously
Each of the 8 GPUs streams its gradient buffer into pinned host memory using its copy engines — no CUDA kernel launches, so the streaming multiprocessors keep running backpropagation undisturbed.
Inter-node allreduce bandwidth (186 MiB messages)
Same nodes, same fabric — only the algorithm differs. Bars show the top of each reported range.
HaiScale: Training Framework
HaiScale implements (backed by HFReduce instead of NCCL), , (mapped onto the retrofitted NVLink pairs), and — each adapted to the 1-NIC PCIe topology, chiefly by aggressively overlapping communication with backward computation.
Benchmarks: VGG16 DDP — half of PyTorch-DDP's step time, ~88% scalability from 32→512 GPUs. LLaMA-13B — step time 64.1 s → 9.7 s scaling 64→512 GPUs (91% parallel efficiency as reported). DeepSeekMoE-16B — 79.6 s → 6.5 s from 40→640 GPUs (76.1% at 640; 92.9% at 320). FSDP GPT2-Medium — 95% scalability 16→128 GPUs, nearly half the step time of PyTorch FSDP.
HaiScale parallel efficiency across workloads
Percentage of ideal linear scaling retained when growing the GPU count
3FS, Congestion Control & the HAI Platform
3FS: 180 storage nodes × 16× 15.36 TB SSDs (2,880 SSDs, >20 PiB usable with mirroring), each node with 2× 200 Gbps NICs — 9 TB/s theoretical, 8 TB/s achieved read throughput. Metadata lives in behind stateless metadata services; data is replicated with for strong consistency with read scaling. The open-source release benchmarks: 6.6 TiB/s aggregate reads (180 nodes), GraySort 110.5 TiB in 30 min, reads at 40 GiB/s peak.
Congestion control: storage, HFReduce, NCCL and other traffic are isolated onto InfiniBand /virtual lanes; routing is static (adaptive routing caused head-of-line blocking); is disabled; 3FS uses request-to-send flow control against . HAI platform: time-sharing scheduler — jobs are checkpointed, preempted and resumed on demand, keeping cluster utilization around 99%.
6.1 · Inside 3FS
How 3FS replicates data — CRAQ
| CPU | 1× 64-core AMD EPYC 7742 |
| Memory | 512 GB DDR4 |
| NICs | 2× InfiniBand CX6 200 Gbps (one per compute zone) |
| SSDs | 16× 15.36 TB PCIe 4.0 NVMe |
| Node count | 180 storage nodes (2,880 SSDs total) |
| Usable capacity | > 20 PiB (mirror redundancy) |
6.2 · Measured: the benchmarks
Peak aggregate read
3FS open-source release6.6 TiB/s
Sustained aggregate read throughput on the open-source 3FS deployment: 180 storage nodes, 500+ concurrent client nodes, with background training traffic.
GraySort
3FS open-source release110.5 TiB in ~30 min
Sorted 110.5 TiB across 8,192 partitions in 30 minutes 14 seconds (3.66 TiB/min) on 25 storage + 50 compute nodes — exercising the full read-shuffle-write path.
KVCache reads
3FS open-source release40 GiB/s peak
Serving transformer KV-cache from 3FS instead of DRAM for inference: peak per-client-cluster read throughput of 40 GiB/s — DRAM-class reuse at SSD cost.
Checkpoint writes
Paper §VI–VII>10 GiB/s / node
Batch write API throughput per compute node — the enabler of 5-minute full-cluster checkpoint cadence with negligible step-time overhead.
6.3 · A job's life on the HAI platform
Submit
Job enters the HAI queue
Users submit training tasks to the platform; the whole 10,000-GPU cluster is time-shared rather than statically partitioned.
Validate
Nodes pass the health gauntlet
The validator checks link speeds, memory bandwidth, byte-level GPU memory, full-GPU GEMM occupancy and storage before the job lands — flaky nodes are drained, not scheduled.
Every 5 min
Asynchronous checkpoint
Model and optimizer state streams GPU→CPU during training, then persists via 3FS batch writes (>10 GiB/s per node). A snapshot completes in seconds.
On demand
Preempt, reschedule, resume
Higher-priority work can interrupt a job; it resumes later from its last snapshot. The scheduler allows at most one cross-zone job and keeps utilization near 99%.
On failure
Automatic recovery
NVLink error? Link flash cut? The platform restarts the task from the latest checkpoint on healthy nodes — worst case loses 5 minutes of work.
Stability & Robustness
Checkpoint manager: gradient/optimizer state is streamed GPU→CPU asynchronously and flushed via 3FS's batch write API (>10 GiB/s per node), making a full snapshot take seconds; snapshots run every 5 minutes, bounding lost work to 5 minutes. Validator: pre-flight and periodic checks — link speeds, memory bandwidth, byte-level GPU memory tests, full-GPU occupancy tests, intra-node allreduce/NVLink bandwidth, storage stress.
A year of failures: 74 (NVLink errors) is 42.57% of GPU faults — the price of the retrofitted bridges, though still below the 52.42% reported for a reference DGX fleet. Software-class faults (Xid13/31/43/45, e.g. illegal memory access) are the next block (Xid43 alone 33.48%); memory errors are only ~2% and are mostly absorbed by row-remapping plus a GPU reset.
One year of GPU faults, by class
Share of observed GPU errors (Xid codes), Fire-Flyer 2 fleet
| NVLink errors | Xid74 | 42.57% | Faults on the retrofitted NVLink Bridges — the dominant class by several times. | Reset/replace bridge; drain node |
| Software-class faults | Xid13 / 31 / 43 / 45 | 33.48% | Illegal memory access and similar; Xid43 alone is 33.48%. Possible app bugs or silent data corruption. | Job restart from checkpoint |
| Correctable memory ECC | Xid63 / 64 / 94 / 95 | 2% | GPU memory ECC events — rare; row remapping absorbs most. | Row remap + GPU reset |
| Uncorrectable GPU faults | Xid44 / 48 / 61 / 62 / 69 / 79 | — | Uncorrectable errors requiring intervention. | GPU reset or node reboot |
| GSP failure | Xid119 | — | GPU System Processor failure. | RMA (hardware return) |
| IB link flash cuts | — | 30% | InfiniBand link flaps; ~30% of task-impacting incidents excluding Xid74. | Validator drains flaky links |
Discussion
Key positions defended: (1) congestion control — with SL/VL isolation and static routing, disabling outperformed every parameter set they tried on mixed traffic; (2) NVLink as opt-in — the bridges serve TP for LLMs but dominate the failure budget (Xid74, 42.57%), so interconnect should follow workload, not fashion; (3) costs — the 83%-performance/50%-cost point is argued to be Pareto-superior for DP-heavy research fleets; (4) stability — their NVLink failure share (42.57%) compares favorably to a referenced DGX fleet (52.42%).
Future Work & Conclusion
Summarized contributions: a 10,000-GPU PCIe A100 cluster at ~half DGX cost and 40% less energy, held near reference performance by HFReduce, HaiScale, 3FS and the HAI platform, with a year of production reliability data. Future work: a next-generation architecture with a 1:1 GPU-to-NIC ratio and multi-plane network topology, targeting all-to-all traffic — plus RoCE/next-gen interconnect evaluation.
Legacy: 3FS was open-sourced in February 2025 (6.6 TiB/s demonstrated); the multi-plane, NIC-rich design maps onto the cluster documented in the DeepSeek-V3 technical report. The paper stands as the public blueprint for cost-engineered frontier-model infrastructure.
Code & Resources
deepseek-ai/3FS→
The Fire-Flyer File System, open-sourced February 2025 — C++ core, FoundationDB metadata, CRAQ replication, RDMA data path. Includes the GraySort and KVCache benchmark setups.
HFAiLab/hai-platform→
The HAI platform — the time-sharing, checkpoint-based task scheduler described in §VI — was also open-sourced by the team (as was hfreduce/haiscale tooling in the HFAiLab organization).
HFReduce, as annotated pseudocode
The paper describes the algorithm rather than listing code; this pseudocode preserves each stage and the measured numbers from §IV.
# HFReduce, as pseudocode (paper §IV) — allreduce for 1-NIC PCIe nodes.
def hfreduce_allreduce(grad_buffers, node_group):
"""Allreduce across N nodes x 8 GPUs without GPU kernels or GPUDirect."""
# (1) GPU -> CPU: async device-to-host copies via the GPU copy engines.
# No CUDA kernel is launched, so SMs keep running backprop.
host_chunks = [gpu.copy_to_host_async(buf, pinned=True)
for gpu, buf in grad_buffers]
wait_all(host_chunks)
# (2) Intra-node reduce on the CPUs with SIMD (AVX) instructions:
# 8 GPU buffers -> 1 node-local sum. 8:1 traffic cut before the NIC.
node_sum = simd_vector_sum(host_chunks)
# (3) Inter-node allreduce over InfiniBand RDMA using the
# double binary tree -- bandwidth-optimal, congestion-friendly.
# Measured: 6.3-8.1 GB/s vs NCCL's 1.6-4.8 GB/s on this fabric.
global_sum = rdma_double_binary_tree_allreduce(node_sum, node_group)
# (4) CPU -> GPU: broadcast the result back over PCIe (~1x data size,
# vs NCCL ring's 2(n-1)/n x). Again copy engines only.
for gpu, _ in grad_buffers:
gpu.copy_from_host_async(global_sum)
# Ceiling: EPYC PCIe root port ~37.5 GB/s -> ~13.3 GB/s theoretical,
# ~12 GB/s practical; DDR4 contention lands production at ~8 GB/s.Glossary
Every dotted-underlined term on this page links back here. Hover any of them in context, or browse the full definitions below.
RDMA
Remote Direct Memory Access lets a network card move data directly between machines' memory, bypassing the operating system and CPU on the data path. It is the foundation of both HFReduce's inter-node transfers and 3FS's storage path.
InfiniBand
A switched network fabric designed for HPC: microsecond latencies, native RDMA, and features like Service Levels for traffic isolation. Fire-Flyer 2 uses Mellanox CX6 200 Gbps NICs — one per compute node, two per storage node.
Fat-tree
The standard HPC network topology: servers connect to leaf switches, leaves to spines, with enough uplinks that all servers can (ideally) communicate at full rate. Layers cost money — Fire-Flyer uses a two-layer fat-tree (122 switches) instead of the DGX SuperPod's three-layer, 1,320-switch design.
Allreduce
In data-parallel training, every GPU computes gradients on its own batch shard; allreduce sums these across all GPUs and distributes the sum back. It is the dominant communication pattern in DP training, and the operation HFReduce reimplements.
Double binary tree
An allreduce/broadcast algorithm (Sanders et al.) that builds two binary trees where each node is internal in one and a leaf in the other, keeping every link busy in both directions — achieving full bandwidth with logarithmic latency. HFReduce uses it for the inter-node stage.
NCCL
The NVIDIA Collective Communications Library implements allreduce, all-gather etc. with GPU kernels, optimized for NVLink/NVSwitch-rich nodes. On Fire-Flyer's 1-NIC PCIe nodes it manages only 1.6–4.8 GB/s inter-node — the gap HFReduce closes.
NVLink
A point-to-point GPU interconnect vastly faster than PCIe. DGX nodes mesh all 8 GPUs via NVSwitch; Fire-Flyer retrofitted NVLink Bridges between GPU pairs when LLM tensor parallelism demanded it — gaining bandwidth and, as Table V shows, its largest hardware failure class.
PCIe
PCI Express connects GPUs, NICs and SSDs to the CPU. PCIe A100s cost far less than SXM modules but communicate through the CPU's PCIe root complex (~37.5 GB/s per EPYC port) — the bandwidth ceiling that shapes HFReduce's design.
SXM
SXM modules mount GPUs directly on a baseboard with full-mesh NVLink and higher power budgets (hence 263 vs 220 FP16 TFLOPS). DGX systems use SXM; Fire-Flyer's cost saving starts with choosing PCIe instead.
DGX-A100 / SuperPod
The DGX-A100 is NVIDIA's flagship A100 node: 8 SXM GPUs fully NVLinked, 2 TB RAM, 9× 200 Gbps NICs. A SuperPod networks many DGXs in a rail-optimized three-layer fat-tree. It is the cost/performance baseline against which Fire-Flyer 2 is measured.
AMD EPYC
Fire-Flyer compute nodes use two 32-core EPYC Rome/Milan CPUs; their 16 DDR4-3200 channels and PCIe 4.0 root complexes are exactly the resources HFReduce turns into a reduction engine. Storage nodes use a single 64-core EPYC 7742.
SIMD
Modern CPUs process many values per instruction (AVX2/AVX-512). HFReduce uses SIMD to sum eight GPUs' gradient buffers in host memory fast enough that the CPU stage never becomes the allreduce bottleneck.
GEMM
General Matrix-Matrix multiplication dominates deep-learning compute, so sustained GEMM TFLOPS is the standard per-GPU performance measure: 107/220 (TF32/FP16) on Fire-Flyer's PCIe A100s vs 131/263 on DGX SXM — the 83% figure.
CRAQ
3FS's replication protocol (Terrace & Freedman, 2009): writes flow down a chain of replicas head→tail and commit at the tail; reads can be served by any replica holding a clean version, so read throughput scales with chain length while writes stay linearizable. A perfect fit for write-once, read-many training data.
FoundationDB
An ordered, strictly-serializable key-value database (open-sourced by Apple). 3FS keeps file-system metadata in FoundationDB behind stateless metadata services — metadata operations become ACID transactions, and metadata servers can scale out or fail without state loss.
NVMe SSD
Non-Volatile Memory Express SSDs speak PCIe directly, delivering GB/s-class throughput each. 3FS's design premise: thousands of NVMe drives + RDMA equals a storage system fast enough that GPUs never wait on data.
Incast
When hundreds of storage nodes answer one client at once, the client's link floods and switch buffers overflow. 3FS tames incast with request-to-send flow control: the client admits transfers at its own line rate.
KVCache
Transformer decoding caches every layer's keys and values for all previous tokens. Serving this from 3FS (40 GiB/s peak reads) instead of GPU/CPU DRAM lets inference clusters reuse long-context caches cheaply — a building block of disaggregated prefill/decode serving.
Checkpointing
Fire-Flyer checkpoints every training job every 5 minutes: states stream GPU→CPU asynchronously during training, then persist via 3FS batch writes at >10 GiB/s per node. The HAI platform restarts failed jobs from the latest snapshot automatically.
Data parallelism (DP)
The dominant parallelism mode: replicate the model, shard the batch, synchronize gradients each step via allreduce. DP's communication pattern is exactly what HFReduce accelerates.
Pipeline parallelism (PP)
Layers are partitioned into stages on different GPUs, and micro-batches stream through so all stages work concurrently. HaiScale tunes the schedule so the small inter-node bandwidth of the 1-NIC nodes stays hidden.
Tensor parallelism (TP)
Each layer's matrices are sharded so every forward/backward pass requires tight synchronization. TP is why NVLink Bridges were retrofitted: on Fire-Flyer, TP runs across the 600 GB/s GPU pairs.
Expert parallelism (EP)
In MoE models each token visits only a few 'expert' networks, which live on different GPUs — creating all-to-all traffic. EP scaling (92.9% at 320 GPUs, 76.1% at 640) is where Fire-Flyer's slim fabric shows, motivating the multi-plane next-gen design.
FSDP
Instead of replicating the model (DP), FSDP shards everything and gathers parameters just-in-time per layer — trading communication for memory. HaiScale's FSDP halves PyTorch-FSDP's step time with 95% scalability on GPT2-Medium.
Mixture-of-Experts (MoE)
MoE layers route tokens to a subset of experts, decoupling parameter count from per-token FLOPs. DeepSeek's model line is MoE-heavy, and MoE's all-to-all traffic drives the paper's future-work network design.
Xid error
Xid codes classify GPU faults: Xid74 = NVLink error, Xid63/64/94/95 = memory ECC events, Xid43 = illegal memory access, Xid119 = GSP failure, etc. Table V uses a year of Xid statistics to characterize fleet reliability.
ECC / row remapping
GPU HBM includes error-correcting codes and spare rows. Correctable ECC events (≈2% of faults here) trigger row remapping; after a reset the GPU returns to full performance — one reason memory errors barely dent fleet availability.
DCQCN
Data Center Quantized Congestion Notification throttles RDMA senders based on ECN marks. With four very different traffic classes sharing the fabric, no parameter setting worked; Fire-Flyer instead isolates classes on separate virtual lanes and uses static routing plus endpoint flow control.
Service Level / Virtual Lane
Packets tagged with a Service Level map to distinct Virtual Lanes (queues) in every switch. Fire-Flyer isolates HFReduce, NCCL, 3FS and other traffic on separate lanes — congestion control by architecture rather than by algorithm.
FAQ
Cost engineering. A PCIe A100 node delivers 83% of the DGX's per-GPU GEMM throughput (220 vs 263 FP16 TFLOPS) at 60% of the price and 60% of the power. The catch — no NVLink mesh and one NIC instead of nine — is a communication problem, and DeepSeek bet they could solve communication in software (HFReduce, HaiScale) rather than pay for it in hardware. The bet paid: ~half total system cost at approximately DGX-class training throughput.
Because the topology changed the rules. NCCL's GPU-kernel rings assume GPUs can reach the network directly and have NVLink between them. On a 1-NIC PCIe node, everything funnels through the CPU's PCIe root complex anyway — so staging in host RAM costs nothing extra, and it buys an 8:1 on-node reduction before the wire, copy-engine transfers that don't occupy GPU compute units, and a bandwidth-optimal double binary tree across nodes. Result: 6.3–8.1 GB/s vs NCCL's 1.6–4.8 GB/s on identical hardware.
Three decisions: one NIC per node instead of nine (fewer ports to switch), a two-layer instead of three-layer fat-tree (122 vs 1,320 switches, cheaper optics), and splitting the cluster into two 600-node zones with only limited links between them — with the scheduler enforcing that at most one job spans zones. Network cost falls from ~4,000 to ~350 reported price units: about 9% of the SuperPod's fabric cost.
It's specialized for AI I/O: read-dominated, random access over huge files, from hundreds of clients at once. So 3FS skips client caching entirely, drives NVMe SSDs over RDMA directly, keeps metadata in FoundationDB behind stateless services, and replicates with CRAQ — strong consistency whose read throughput scales with replicas. The proof points: 8 TB/s production reads (6.6 TiB/s in the open benchmark), a 110.5 TiB GraySort in 30 minutes, and >10 GiB/s per-node checkpoint writes.
LLM inference keeps attention keys/values for every previous token; long contexts make that cache enormous and DRAM is the most expensive place to keep it. 3FS serves KVCache at up to 40 GiB/s from SSDs — fast enough to reuse caches across requests and to split prefill from decode onto different machines. It's one of the quiet reasons DeepSeek could serve long-context models cheaply.
Their fabric carries four very different traffic classes — latency-sensitive collectives (HFReduce, NCCL), throughput-hungry storage (3FS), and management traffic. No DCQCN parameter set handled all of them. Instead they isolated each class on its own InfiniBand virtual lane, used static routing (adaptive routing caused head-of-line blocking), and let 3FS do endpoint request-to-send flow control. Structure replaced algorithm — and it worked in production.
The NVLink bridges — Xid74 errors are 42.57% of GPU faults, several times any other class (still better than the 52.42% NVLink share of a referenced DGX fleet). Software-class faults like illegal memory access are next (~33%). True memory failures are rare: ECC events are ~2% and row remapping absorbs most. The mitigation stack — 5-minute checkpoints, automated restart, and a validator that drains flaky nodes — turns all of this into minutes of lost work.
Directly. This paper's future-work section — 1:1 GPU-to-NIC ratio, multi-plane network topology, MoE-oriented design — describes the architecture generation that trained DeepSeek-V3 (and thus R1). The doctrine is identical: measure the real workload, buy exactly the hardware it needs, write software to close the gaps. Fire-Flyer 2 is where that playbook was developed and de-risked.
Yes — it was open-sourced in February 2025 at github.com/deepseek-ai/3FS (C++ core, FoundationDB dependency, RDMA network required for full performance). The repo includes the GraySort and KVCache benchmark setups. You won't hit 6.6 TiB/s without 180 storage nodes, but the architecture — chain-replicated NVMe over RDMA with transactional metadata — runs at small scale too.
References
The works the paper builds on — every entry links to the cited paper.
- [1]
Terrace, J., & Freedman, M. J. (2009). Object storage on CRAQ: High-throughput chain replication for read-mostly workloads. USENIX ATC 2009
- [2]
van Renesse, R., & Schneider, F. B. (2004). Chain replication for supporting high throughput and availability. OSDI 2004
- [3]
Zhou, J., Xu, M., et al. (Apple) (2021). FoundationDB: A distributed unbundled transactional key value store. SIGMOD 2021
- [4]
Sanders, P., Speck, J., & Träff, J. L. (2009). Two-tree algorithms for full bandwidth broadcast, reduction and scan. Parallel Computing 35(12)
- [5]
NVIDIA (2020). NVIDIA DGX A100 system architecture. NVIDIA technical whitepaper
- [6]
NVIDIA (2020). NCCL: NVIDIA Collective Communications Library. developer.nvidia.com
- [7]
Zhu, Y., Eran, H., Firestone, D., et al. (2015). Congestion control for large-scale RDMA deployments (DCQCN). SIGCOMM 2015
- [8]
Li, S., Zhao, Y., Varma, R., et al. (2020). PyTorch distributed: Experiences on accelerating data parallel training. VLDB 2020 · arXiv:2006.15704
- [9]
Zhao, Y., Gu, A., Varma, R., et al. (2023). PyTorch FSDP: Experiences on scaling fully sharded data parallel. VLDB 2023 · arXiv:2304.11277
- [10]
Shoeybi, M., Patwary, M., Puri, R., et al. (2019). Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv:1909.08053
- [11]
Huang, Y., Cheng, Y., Bapna, A., et al. (2019). GPipe: Efficient training of giant neural networks using pipeline parallelism. NeurIPS 2019 · arXiv:1811.06965
- [12]
Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2020). ZeRO: Memory optimizations toward training trillion parameter models. SC 2020 · arXiv:1910.02054
- [13]
Shazeer, N., Mirhoseini, A., Maziarz, K., et al. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR 2017 · arXiv:1701.06538
- [14]
Dai, D., Deng, C., Zhao, C., et al. (2024). DeepSeekMoE: Towards ultimate expert specialization in mixture-of-experts language models. arXiv:2401.06066
- [15]
Touvron, H., Lavril, T., Izacard, G., et al. (2023). LLaMA: Open and efficient foundation language models. arXiv:2302.13971
- [16]
DeepSeek-AI (2025). 3FS: Fire-Flyer File System (open-source release). GitHub
- [17]
DeepSeek-AI (2024). DeepSeek-V3 technical report. arXiv:2412.19437
Cite This Paper
@inproceedings{an2024fireflyer,
title = {Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware
Co-Design for Deep Learning},
author = {An, Wei and Bi, Xiao and Chen, Guanting and Chen, Shanhuang
and others ({DeepSeek-AI})},
booktitle = {SC24: International Conference for High Performance
Computing, Networking, Storage and Analysis},
year = {2024},
publisher = {IEEE},
url = {https://arxiv.org/abs/2408.14158}
}