Skip to content

A Scalable NorthPole System with End-to-End Vertical Integration for Low-Latency and Energy-Efficient LLM Inference

🕒 Published (v1): 2025-11-20 00:53 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IBM presents a vertically integrated research-prototype inference system pairing 288 NorthPole neural accelerator cards with a containerized software pipeline and quantization-aware training to serve LLMs at 2.8 ms inter-token latency under 30 kW of air-cooled rack power. All model weights and KV caches reside entirely in on-chip SRAM, eliminating off-chip memory traffic during inference. The system exposes an OpenAI-compatible API and integrates directly with IBM watsonx Orchestrate for agentic application deployment.

Problem

GPU-based LLM inference incurs prohibitive memory-bandwidth overhead because weights and KV caches must be transferred between HBM and compute on every forward pass, driving up latency and power. Existing systems also require specialized fabrics, liquid cooling, and facility upgrades, limiting deployability in ordinary data centers. The paper addresses how to deliver enterprise-grade, low-latency LLM inference with commodity rack infrastructure while matching the accuracy of bfloat16 models.

Method

The system is structured across four co-designed layers:

  1. Hardware: The NorthPole chip (22 B transistors, 12 nm, 224 MB on-chip SRAM, 13 TB/s on-chip bandwidth) eliminates off-chip weight reads by keeping all weights and KV cache in on-chip memory. Cards communicate over PCIe via direct card-to-card (C2C) DMA using FPGA-managed credit-tracking and pre-stored DMA descriptor chains, avoiding host CPU mediation.

  2. Model mapping: Transformer layers are partitioned using pipeline parallelism per layer (attention + MLP each on separate cards) and tensor parallelism for the output layer. For Granite-3.3-8b (A8-C8-W4), 40 layers × 2 cards + 4 output cards = 84 cards across 6 servers.

  3. Inference pipeline: Three container types orchestrate inference—a sequence head container (tokenization, dynamic batching via a worker-thread pool, AMQP queue consumption), a pipeline management container (ring-based consensus for startup, passthrough I/O), and per-node NorthPole application containers (model loading, C2C relay). An AMQP broker (RabbitMQ on IBM Cloud) decouples API endpoints from inference instances and provides priority-based load balancing.

  4. Quantization recovery: The SiLQ quantization-aware training algorithm fine-tunes the bfloat16 model to A8-C8-W4 precision on 8×H100 GPUs with batch size 128 and sequence length 1024, recovering accuracy to within noise of the original.

Key Contributions

  • Scale-out from single-node (16 cards) to full-rack (288 cards, 18 servers) NorthPole LLM inference system.
  • Pipelined-parallelism mapping strategy keeping all weights and KV caches fully on-chip across the card chain.
  • Containerized, cloud-ready inference harness with dynamic batching, AMQP queuing, and OpenAI-compatible API.
  • Demonstration that SiLQ fine-tuning recovers bf16-equivalent accuracy at A8-C8-W4 quantization.

Results

  • System peak: 115 peta-ops/s (4-bit INT), 3.7 PB/s aggregate memory bandwidth, 30 kW power, 0.67 m² footprint.
  • Granite-3.3-8b at 2k context, 28 users: inter-token latency (ITL) = 2.8 ms, time-to-first-token (TTFT) = 64.8 ms.
  • Granite-3.3-8b at 4k context, 14 users: ITL = 2.8 ms, TTFT = 96.2 ms.
  • 3B Granite model, 1 server node: 28,356 token/s system throughput, <1 ms/token ITL, 672 W card power.
  • Quantization accuracy (A8-C8-W4 vs. bf16 on 19 benchmarks): within ≤2 pp on most tasks (e.g., BoolQ 86.9 vs. 86.9, HellaSwag 74.7 vs. 74.5, MMLU 65.0 vs. 64.1); slight degradation on BBH (8.8 vs. 8.2) and GPQA (34.0 vs. 34.7).
  • No liquid cooling, no specialized fabric, and no facility power upgrades required.

Limitations

  • Context length and simultaneous-user count are jointly constrained by on-chip KV-cache capacity; scaling one reduces the other.
  • SiLQ fine-tuning still requires 8×H100 GPUs for quantization recovery—the training side is not on-NorthPole.
  • Full accuracy evaluation covers only Granite-3.3-8b; larger models (gpt-oss-20b, 120b) are described architecturally but not benchmarked for accuracy or latency in this paper.
  • The paper does not report prefill throughput or measure performance under variable request arrival rates (e.g., Poisson load).
  • On-chip memory ceiling limits maximum context length; 4k context at 8B scale requires halving the user batch to 14.

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete, production-oriented example of an end-to-end inference harness: the three-container architecture (sequence head → pipeline management → application containers) is structurally analogous to a meta-harness that coordinates specialized workers (NorthPole cards) behind a unified API surface. The AMQP-based task queue with priority lanes mirrors the job-dispatch layer seen in LLM serving harnesses like vLLM's scheduler or Ray Serve's actor pool. For researchers tracking harness design, the key lesson is how hardware-level pipeline parallelism is abstracted behind a cloud-native container orchestration layer, enabling the same harness to scale from a single server node to a 288-card rack without application-layer changes. The direct integration with watsonx Orchestrate further positions this as infrastructure for agentic workflow harnesses, where the NorthPole system acts as a low-latency LLM executor called by a higher-level orchestration layer.