Code Aesthetics with Agentic Reward Feedback¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Agentic reward feedback teaches LLMs visually-oriented coding tasks beyond correctness

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LLMs struggle with visually-oriented coding tasks (chart generation, webpage design) because existing reward signals are text-only and blind to visual rendering. This paper introduces a multi-agent reward framework (agentic reward feedback) that evaluates executability, static visual quality, and interactive usability, then trains models via GRPO to jointly optimize functionality and aesthetics. The resulting AesCoder-4B outperforms GPT-4o and GPT-4.1 on both plot generation and webpage design benchmarks.

Problem¶

Standard coding LLM reward signals (execution success, unit tests, text-based reward models) operate on a single textual modality and cannot assess the visual rendering of code outputs. This leaves LLMs unaware of aesthetic failures—overlapping elements, inconsistent color schemes, non-functional interactive components—in visually-oriented tasks such as HTML/CSS webpage design and Python plot generation.

Method¶

The pipeline has four components:

AesCode-358K dataset: 158K Python plot samples (regenerated from VisCode-200K using Qwen3-Coder-480B for executability) + 200K webpage design samples (GPT-4o/GPT-5 generated HTML, rendered via Playwright/Selenium, quality-filtered by GPT-5 scoring). Covers five webpage categories and underwent t-SNE-guided deduplication.
Agentic Reward Framework: Three specialized agents run sequentially/in parallel:
Execution Agent: Extracts HTML, runs HTMLHint rule-based syntax validation; returns \(r_\text{exec} \in \{-1, +1\}\).
Static Aesthetics Agent: Screenshots the rendered page via Playwright headless mode, then uses GPT-5 as a VLM judge scoring instructional alignment (40 pts), visual elements (30 pts), and structural cohesion (30 pts); returns \(r_\text{static}\).
Interactive Aesthetics Agent: Built on WebVoyager + GPT-4o; plans interaction candidates from the user instruction, executes them step-by-step in headless Chrome, and aggregates binary success/failure into \(r_\text{interact} = \sum_{i=1}^{N} s_i\).
Combined reward: \(r = w_\text{exec} \cdot r_\text{exec} + w_\text{static} \cdot r_\text{static} + w_\text{interact} \cdot r_\text{interact}\)
GRPO-AR training: Stage I is SFT on AesCode-358K; Stage II applies GRPO using the agentic reward. For each prompt \(p\), a group of \(G\) outputs \(\{o_1,\ldots,o_G\}\) is sampled, advantages computed as \(\hat{A}_{i,t} = \frac{r_i - \text{mean}(r)}{\text{std}(r)}\), and the standard clipped GRPO objective with KL penalty against the SFT reference policy is maximized.
OpenDesign benchmark: 840 real-world webpage cases evaluated by the same static + interactive agents; validated against Design Arena via Spearman \(\rho = 0.98\) and Kendall \(\tau = 0.91\).

Key Contributions¶

Introduces the code aesthetics concept for visually-oriented coding tasks and investigates LLM aesthetic awareness.
AesCode-358K: first large-scale instruction-tuning dataset for code aesthetics (Python plots + webpage design).
Agentic reward feedback: three-agent system providing multi-modal reward (text executability, static visual, interactive usability) for RL training.
GRPO-AR: integration of the agentic reward into the GRPO algorithm enabling joint optimization of functionality and aesthetics.
OpenDesign: first benchmark (840 cases) for automated webpage aesthetic evaluation, validated against human preferences (GPT-human agreement 80.9%).

Results¶

AesCoder-4B on PandasPlotBench: error rate 0.09, avg score 70, good rate 0.63 — matches 7B baselines and outperforms the Qwen3-4B-Instruct-2507 baseline (err 0.13, avg 65, good 0.55).
AesCoder-4B on OpenDesign: Total static score 81.92, InterAes 1.04 — surpasses GPT-4o (48.08 / 0.44) and GPT-4.1 (65.79 / 0.74) on both static and interactive dimensions; competitive with GPT-5 (81.03 / 1.37) and Claude Sonnet 4 (81.05 / 0.92).
AesCoder-4B matches or exceeds open-source models with 30B–685B parameters (e.g., Qwen3-Coder-480B: 79.90 / 0.70; DeepSeek-R1-0528: 81.23 / 0.94) on OpenDesign.
OpenDesign vs. Design Arena: Spearman = 0.98 (\(p < 1.5 \times 10^{-6}\)), Kendall = 0.91 (\(p < 3.0 \times 10^{-5}\)); 80.0% top-5 rank overlap.

Limitations¶

Interactive aesthetics agent (WebVoyager-based) may fail on corner cases (confusing UI elements, irrelevant text), producing scores lower than ground truth — agent failures are conflated with bad design.
Static evaluation via screenshots cannot capture dynamic or animated content beyond a single-frame snapshot.
RL data (20K from WebSight v0.2) is limited to webpage tasks; generalization of GRPO-AR to other visually-oriented coding domains (e.g., game graphics, SVG) is not demonstrated.
GPT-5 as the static judge introduces cost and latency; reproducibility depends on proprietary model access.
Design Arena rankings used for validation were frozen at a single point in time (September 22, 2025) and may drift.

Relevance to Agentic AI / LLM Agents¶

This paper is a direct application of multi-agent systems as reward providers — replacing a monolithic scalar reward with a pipeline of specialized agents (execution, vision-based judge, GUI interaction agent) that each handle a distinct evaluation modality. For researchers tracking LLM agents, the interactive aesthetics agent is particularly notable: it demonstrates that a web-navigation agent (WebVoyager) can be repurposed as an automated evaluator within an RL training loop, tightening the feedback cycle for tasks where correctness is inherently interactive. The GRPO-AR framework also shows how verifiable, environment-grounded signals (does the webpage button actually work?) can substitute for human preference labels, which is a key challenge in scaling agentic training. This work pushes the boundary of agent-as-judge beyond text and images into full browser interaction.