Skip to content

GLM-5V-Turbo: Toward a Native Foundation Model for Multimodal Agents

🕒 Published (v1): 2026-04-29 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

GLM-5V-Turbo: native foundation model explicitly designed for multimodal agent deployment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GLM-5V-Turbo is a multimodal foundation model designed natively for agentic tasks, integrating vision perception into the full reasoning–planning–execution loop rather than treating it as an auxiliary interface. It introduces CogViT (a new vision encoder), Multimodal Multi-Token Prediction (MMTP), joint RL over 30+ task categories, and explicit integration with external agent frameworks (Claude Code, AutoClaw/OpenClaw). The result is strong performance on GUI agent, multimodal search, and coding benchmarks while retaining text-only coding capability.

Problem

Existing foundation models treat multimodal perception as a bolt-on to language reasoning, creating a mismatch for agentic settings where heterogeneous inputs (images, GUIs, documents, video, webpages) must be natively unified with planning and tool use. Separately, large-scale multimodal RL training lacks infrastructure support for variable-length visual sequences, mixed verifier types, and long-tail rollout latency—limiting practical optimization of agentic capability.

Method

CogViT vision encoder: Two-stage pretraining—(1) distillation-based masked image modeling with dual teachers (SigLIP2 for semantics, DINOv3 for texture), 35% masking, Muon optimizer with QK-Norm; (2) contrastive image-text pretraining with NaFlex variable-resolution, SigLIP sigmoid loss at 64K batch size, on an 8B bilingual corpus.

Multimodal Multi-Token Prediction (MMTP): Extends text MTP to multimodal by replacing all visual tokens at the MTP head input with a single shared learnable <|image|> special token, avoiding propagation of visual embeddings across pipeline-parallel stages while remaining compatible with sequence/context parallelism.

Joint RL over 30+ task categories: Covers perception (grounding, OCR, video, 3D), reasoning (STEM, math, logic), and agentic tasks (GUI, coding, tool use). Includes relative visual policy optimization for UI-to-code. A unified VLM RL Gym provides a consistent environment interface for single- and multi-step tasks with an independent reward system that combines synchronous rule-based and asynchronous model-based verifiers.

RL infrastructure: Full-pipeline decoupling (rollout inference, reward evaluation, batch construction, weight transfer overlap); per-completion reward callbacks to reduce long-tail idle time; separate ViT/projector memory management with CPU offloading; topology-aware CP/TP partitioning moved upstream to data-loading stage; joint bin-packing over sequence length and ViT token count.

Toolchain and framework integration: Proprietary zai_* tool suite (recognition, multimodal search, browser, image processing, web/slide creation, deep research) plus official skill packages wrapping GLM-5V-Turbo as a MaaS API for OpenClaw, AutoClaw, and Claude Code.

ImageMining benchmark: 217 curated test cases across 7 domains and 5 reasoning categories; requires multi-step agentic tool use with mandatory "Visual Jump" hops (WEB_VISUAL constraint) ensuring visual rather than textual shortcuts during knowledge discovery.

Key Contributions

  • CogViT: a parameter-efficient vision encoder with two-stage pretraining (distillation MIM → contrastive alignment) and NaFlex variable-resolution support
  • MMTP: multimodal multi-token prediction using a learnable <|image|> placeholder, improving training stability and system scalability versus direct visual embedding propagation
  • Broad multi-task RL framework covering 30+ heterogeneous task categories with a unified VLM RL Gym and independent reward orchestration layer
  • Scalable multimodal RL infrastructure: async pipeline overlap, ViT-specific memory offloading, topology-aware partitioning, bin-packing for variable-length visual sequences
  • Multimodal toolchain (zai_* tools) enabling a perception–planning–execution loop grounded in visual environments
  • Official skill packages exposing GLM-5V-Turbo as MaaS API within Claude Code, OpenClaw, and AutoClaw frameworks
  • ImageMining: new vision-centric deep search benchmark with mandatory visual reasoning hops

Results

  • ImageMining (multimodal deep search): 30.7 (GLM-5V-Turbo); ~8Ă— improvement over GLM-4.6V on MMSearchPlus (30.0 vs prior generation)
  • BrowseComp-VL: 51.9
  • MMSearch: 72.9
  • SimpleVQA: 78.2
  • AndroidWorld (GUI agent): 75.7
  • OSWorld (GUI agent): 62.3; RL stage +4.9% over SFT baseline
  • PinchBench (Claw-based): 87.0 / 80.7
  • ClawEval: 57.7 / 75.0
  • ZClawBench: 57.6
  • Design2Code (multimodal coding): 94.8, outperforming Claude Opus 4.6
  • Text-only coding (CC-Backend / CC-Frontend / CC-RepoExploration): 22.8 / 68.4 / 72.2, surpassing base model GLM-5-Turbo
  • RL gains over SFT baseline (selected): RefCOCO-avg +4.8%, PointBench +3.2%, MVBench +5.6%, SUNRGBD 3D grounding +7.7%, OCRBench +4.2%, CharXiv +7.7%, MMSearch +3.5%

Limitations

  • Multi-task RL coverage boundary: capabilities absent from the RL task distribution can decline post-training, as model capacity concentrates around sampled tasks
  • ImageMining is self-collected (217 cases); scale and domain diversity are modest relative to established benchmarks
  • Proprietary zai_* tools are not open-sourced, limiting reproducibility of the full toolchain evaluation
  • MMTP ablation is conducted only on a 0.5B model; generalization of the design choice to full scale is assumed but not ablated directly
  • Integration benchmarks (PinchBench, ClawEval, ZClawBench) depend on proprietary Claw infrastructure, making independent verification difficult

Relevance to Harnesses / Meta-Harnesses

GLM-5V-Turbo is directly relevant as an exemplar of a model built to be an agent harness component: it exposes official skill packages that wrap the model as a MaaS API callable by external orchestration frameworks (Claude Code, OpenClaw, AutoClaw), and its VLM RL Gym constitutes a training-time meta-harness that homogenizes heterogeneous single- and multi-step tasks under a unified environment interface with pluggable reward verifiers. The independent reward orchestration layer—combining synchronous rule-based and async model-based verifiers with configurable aggregation—is a concrete harness design pattern for multi-task RL pipelines. The report's lessons on hierarchical optimization (perception → reasoning → agentic tasks trained in stages) and end-to-end verification (clear specification + reliable verifier required for stable RL) are directly actionable for anyone designing meta-harnesses that orchestrate LLM agents over long-horizon tasks.