Estimating Worst-Case Frontier Risks of Open-Weight LLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; OpenAI team; malicious fine-tuning measures worst-case frontier model risks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces malicious fine-tuning (MFT) as a methodology to estimate worst-case frontier risks of open-weight LLMs, applied to OpenAI's gpt-oss-120b model across biological and cybersecurity domains. MFT combines anti-refusal RL training, tool-augmented domain-specific RL (web browsing for bio, terminal environments for CTF), and curated in-domain data. Results show MFT gpt-oss remains below OpenAI o3 on all evaluations, and o3 itself sits below the Preparedness Framework "High" capability threshold.

Problem

Standard open-weight safety evaluations measure refusal rates on the released model, ignoring that determined adversaries can fine-tune away safety training or directly optimize for harmful capabilities. There is no established methodology for estimating the ceiling of adversarial misuse — the worst-case frontier risk — before an open-weight release decision is made.

Method

Two MFT attack classes are studied on gpt-oss-120b (120B parameters):

1. Anti-refusal training: Incremental RL with a helpful-only reward (complying with unsafe prompts) drives refusal rates to ~0% while maintaining general benchmark performance (GPQA); no degradation in reasoning capability.

2. Domain-specific capability maximization:

3. Biology: RL with interleaved chain-of-thought + web browsing tool; in-domain data mix includes open-source biology datasets (GPQA-bio, WMDP, LAB-Bench, BioLP-Bench), internal tacit-knowledge and translation datasets, and a synthetic protocol-debugging dataset generated via o3 introducing deliberate errors into real protocols.
4. Cybersecurity: RL inside a dockerized terminal with pentesting packages; trained on 264 CTF challenges (high school + collegiate + uncategorized) from CSAW, SEKAI, GoogleCTF; browsing enabled with domain blocklist to prevent writeup leakage; professional CTF set held out.

At inference, consensus@k and internal best-of-k scaffolding are tested as additional elicitation methods.

Key Contributions

• Formal definition and operationalization of malicious fine-tuning (MFT) as a worst-case risk-estimation protocol for open-weight models.
• Anti-refusal RL recipe that eliminates refusals without harming general capability, reproducible with moderate compute.
• TroubleshootingBench: a new biorisk benchmark targeting protocol-debugging, evaluated publicly for the first time.
• Empirical evidence that MFT gpt-oss stays below o3 (itself sub-"High") on both bio and cyber, supporting the release decision.
• Demonstration that consensus@k and best-of-k elicitation provide negligible gains on these risk benchmarks; fitting a pass@k curve estimates 367 trials needed for 75% pass rate on professional CTFs.

Results

Biology (internal evals, Figure 1): - Gryphon Free Response (consensus@32): gpt-oss 72.0% → MFT best 76.9%; o3 80%; expert human ~80%+ - Biorisk Tacit Knowledge: MFT best ~75.7%; o3 ~76.8% - TroubleshootingBench (pass@1): MFT best 32.2%; all competitors ≤25.7%; far below expert human - Protocol debugging remains below expert human even after targeted synthetic data

Biology (external SecureBio evals, Figure 2): - MFT gpt-oss performs comparably to o3 and ~3–5 points better than DeepSeek R1-0528 with browsing across HPCT, MBCT, VCT, WCB

Cybersecurity (Figure 6): - High school CTFs pass@12: gpt-oss 63.3%–71.1%; o3 71.9% - Professional CTFs pass@12: MFT best 27.7%; o3 38.7% - Cyber range (light and medium): all gpt-oss variants 0.0% pass@1 without hints; o3 achieves 1.7% on medium with hints - SFT, internal best-of-k, best-of-k evaluation: all <4 pp improvement; none closes gap to o3

Limitations

• MFT was not applied to competitor open-weight models (DeepSeek R1-0528, Kimi K2, Qwen3 Thinking), so their worst-case capabilities are underestimated, making comparisons not fully symmetrical.
• Threat model assumes a well-resourced adversary (~7-figure USD GPU budget) but not one capable of full pre-training; underestimates nation-state-level actors.
• Cyber range environments are less complex than real enterprise systems, so zero scores may overestimate safety margin.
• Only biology and cybersecurity domains evaluated; self-improvement domain excluded.
• Professional CTF performance may be underestimated because training set is a 264-challenge subset; coverage gaps exist.
• Paper intentionally withholds fine-grained MFT procedural details (and does not release MFT weights), limiting reproducibility for safety researchers.
• Study is specific to gpt-oss; generalizability of MFT methodology to other model families is assumed but not demonstrated.

Relevance to Agentic AI / LLM Agents

The cybersecurity MFT setup is explicitly an agentic RL environment: the model executes multi-step plans via terminal tool calls, manages time budgets, and chains actions in emulated networks — closely mirroring the architectures used in LLM agent research. The paper's finding that failure modes are predominantly general agentic capability failures (tool parsing, time management, premature termination) rather than domain knowledge gaps is directly informative for anyone building or evaluating LLM agents. The bio MFT pipeline (chain-of-thought + browsing RL) is structurally identical to web-agent training, and the anti-refusal RL methodology demonstrates how safety properties of agentic systems can be erased incrementally. For safety-focused agentic AI researchers, this paper provides a concrete worst-case elicitation benchmark and confirms that agentic scaffolding (browsing, terminal) is a primary risk multiplier even more than domain fine-tuning alone.