Hogwild! Inference: Parallel LLM Generation via Concurrent Attention¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Hogwild! Inference runs multiple LLM instances in parallel over a shared, concurrently-updated KV cache, letting the workers see each other's tokens as they are generated and self-organize a collaboration strategy without any pre-defined framework or fine-tuning. By using RoPE query rotation instead of re-encoding cached tokens, the approach achieves ~2× throughput with 2 workers and ~3.4× with 4 workers on a 32B model. Experiments show accuracy-per-forward-pass improvements over sequential inference, Self-Consistency, and Skeleton-of-Thought across math and coding benchmarks.

Problem¶

Existing parallel LLM inference methods (Self-Consistency, Skeleton-of-Thought, PASTA) impose a fixed collaboration structure (vote, plan-then-execute, spawn-subtasks) that fails when the initial decomposition is wrong, when one thread becomes a straggler, or when the problem cannot be cleanly split ahead of time. No single framework is universally effective, and all prior methods incur re-encoding overhead when sharing context between workers.

Method¶

Multiple instances of the same LLM run as a batch sharing a single KV cache partitioned into three regions per worker: (1) a common cache holding the system prompt and completed reasoning steps from all workers; (2) other-worker blocks containing the latest in-progress paragraph of each peer; (3) the current-worker block for the active worker's unfinished step.

Because the same KV entries appear at different sequence positions for each worker, the paper avoids re-encoding by rotating only the current-token query rather than all cached keys. For a query at position \(i_q\) attending to a cache block starting at \(i_k\), the query is rotated by \(i_q - i_k\); this is done independently per cache block using a Flash-Decoding–style tiled attention kernel that holds one copy of each block and applies the appropriate query offsets per worker.

When a worker finishes a paragraph (\n\n), its KV block is moved—with proper RoPE rotation—into the shared common history, so workers always see each other's latest output as recent tokens. Workers are zero-shot prompted with the collaboration rules; every ~1000 generated tokens a random worker is injected with a redundancy-check prompt ("Wait, am I doing redundant work?") to prevent tunnel-vision.

Key Contributions¶

Concurrent Attention: a shared KV cache mechanism that gives all workers instant (token-level) access to each other's partial generations without re-encoding.
RoPE query rotation trick: rotate current-token queries instead of all past keys, making multi-block attention O(1) in cache writes and avoiding cubic re-encoding overhead.
Chat-like cache layout: finished paragraphs are appended to a common history so recency of peer information is preserved over long generations.
Zero-shot collaboration: shows that reasoning-capable LLMs (QwQ-32B, DeepSeek-R1, Qwen3, Phi-4-Reasoning+) can self-organize collaboration without fine-tuning.
Efficient GPU kernel: custom Flash-Decoding variant; 2-worker Hogwild! generates nearly 2× tokens/s, 4-worker ~3.4× vs. single-worker FlashAttention v2 on a 32B-AWQ model on L40S.

Results¶

Throughput (QwQ-32B-AWQ, L40S): 1 worker → 19–20 tok/s; 2 workers → ~36 tok/s (~1.9×); 4 workers → ~68 tok/s (~3.4×). Latency per forward pass increases modestly (49.7 ms → 55–58 ms for 2–4 workers at short contexts).
LIMO (817 hard math problems, QwQ-32B): Hogwild! 2w reaches the accuracy of baseline single-worker at roughly half the forward passes; 4w provides further gains. Self-Consistency and SoT both underperform Hogwild! at equivalent forward-pass budgets.
OlympiadBench Math (QwQ-32B, 2w): consistent accuracy improvement over the sequential baseline across all forward-pass budgets up to 8k.
LiveCodeBench v5 (QwQ-32B, 2w): Pass@1 improvements over baseline; generalizes to Qwen3-8B and Phi-4-Reasoning+.
AIME'25 (Qwen3-235B-A22B, DeepSeek-R1, 2w): visible accuracy gains versus single-agent baselines.
Collaborativeness (GPT-4o judge, 1–6 scale): token-wise sync scores ~3.5 vs. step-wise ~2.5 vs. no-sync ~1.5 for QwQ-32B, confirming that instant cache sharing enables qualitatively richer coordination than step-level exchange.
Failure mode: Qwen3 models at 1.7B and 4B fail to adapt; Qwen3-14B on OlympiadBench Physics overthinkes and regresses after ~4k tokens.

Limitations¶

Reduced robustness for smaller models (Qwen3 ≤4B cannot reliably self-coordinate).
Degraded behavior at very long contexts, indicating scalability challenges with sequence length.
Collaboration quality metric relies on GPT-4o as judge, which is proprietary and limits reproducibility.
s1-style redundancy prompts help but do not fully prevent workers from ignoring each other's progress.
Current implementation targets single-GPU or small multi-GPU; distributed scaling is left to future work.
No fine-tuning means collaboration is opportunistic; emergent strategies can be suboptimal.

Relevance to Agentic AI / LLM Agents¶

Hogwild! Inference directly addresses multi-agent coordination at the inference-engine level rather than at the orchestration or prompting level, which is a meaningfully different point in the stack from most LLM-agent frameworks. By showing that reasoning-capable LLMs can self-organize parallelism through a shared memory substrate—without hand-coded role assignments or fixed task graphs—it suggests that emergent, dynamic collaboration is an achievable property of current models given the right architectural primitive. The shared KV cache acts as a form of working memory visible to all agents simultaneously, which is conceptually analogous to blackboard architectures in classical multi-agent systems and could inform how future agentic systems manage shared state. The RoPE rotation trick is a concrete, reusable mechanism for efficiently fusing the context streams of concurrently reasoning agents, relevant to any system that needs low-latency shared memory across LLM workers.