Skip to content

Object Aligner: A Configurable JSON Schema Similarity Score for Graphs, Applied to LLM Prompt Optimization

🕒 Published (v1): 2026-07-02 10:07 UTC · Source: Arxiv · link

Why this paper was selected

Configurable JSON schema similarity metric directly applicable to tool-calling and agentic planning evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Object Aligner (OA) is a deterministic, schema-driven similarity scorer for JSON-structured LLM outputs that recursively aligns predicted and gold trees using optimal assignment (Hungarian algorithm / sequence-alignment DP) and awards partial credit at schema-declared granularity. Its central contribution is referential alignment, which scores graph-structured JSON invariantly under identifier renaming by approximating a gold↔candidate identifier bijection via Weisfeiler–Leman color refinement. Applied as a reward inside the GEPA prompt optimizer, OA helps or stays neutral across all evaluated datasets.

Problem

LLMs increasingly output JSON conforming to fixed schemas (tool calls, agentic plans, knowledge graphs), yet no existing scorer simultaneously offers: (1) determinism and reproducibility without LLM-judge calls; (2) schema-aware partial credit over arbitrarily nested structures; (3) invariance to identifier renumbering in graph-structured outputs; and (4) structured feedback suitable for prompt-optimization inner loops. Exact match is brittle, generic text similarity discards structure, and LLM-as-a-judge is expensive, noisy, and non-deterministic.

Method

OA takes a gold object \(g\), candidate \(p\), and schema \(S\), and returns \(s(g, p; S) \in [0,1]\) via recursive tree alignment:

  • Primitive nodes: configurable comparators per field — exact, Jaro, Levenshtein, invdiff (\(1/(1+|g-p|)\)), or user-supplied (e.g., embedding similarity).
  • Order-agnostic sequences: per-pair cost matrix \(M_{ij} = s(g_i, p_j)\) solved with the Hungarian algorithm; score is \(\frac{1}{D}\sum_{(i,j)\in\mathcal{M}} M_{ij}\) where \(D\) counts all gold and excess candidate slots.
  • Order-sensitive sequences: monotone insertion/deletion-aware DP (for ranking and planning outputs).
  • Positional tuples: fixed-arity sequences with per-slot weights and comparators.
  • Referential alignment (main contribution): JSON records keyed by arbitrary identifiers (analogous to primary/foreign keys) are matched by inferring a bijection \(\pi: \text{gold IDs} \to \text{candidate IDs}\). Because exact recovery is graph isomorphism (NP-hard), OA approximates it with Weisfeiler–Leman (WL) color refinement — nodes are iteratively colored by their neighborhood; gold/candidate nodes with matching colors are matched via bipartite assignment. All ref fields are then scored through \(\pi\), making the score invariant to relabeling.
  • Repair feedback: the same alignment tree that produces the score also emits ranked repair operations — edits ordered by the exact score delta each recovers — at no extra cost and with no LLM call.

Schema configuration is expressed entirely through JSON Schema extensions (idScope, ref, order, valueWeight, score, etc.), requiring no code changes per task.

Key Contributions

  • Open-source Python library configured via JSON Schema annotations; drop-in reward for DSPy, GEPA, TextGrad.
  • Referential alignment: identifier-renaming-invariant scoring for (hyper)graph-structured JSON via WL color refinement.
  • Per-list order semantics: schema-declared choice among order-agnostic (Hungarian), order-sensitive (DP), and positional-tuple regimes.
  • Deterministic ranked repair feedback: mismatches emitted as score-delta-ranked edit operations from the match tree — no LLM call required.
  • First prompt-optimization pipeline driven by schema-aware partial credit over nested structures (via GEPA integration).
  • Empirical study on both synthetic and real-world datasets including AMR benchmarks.

Results

  • OA used as reward in GEPA helps or stays neutral across all datasets (no dataset where it hurts).
  • On the running org-chart example (Fig. 1), the identifier bijection \(\pi = \{1\mapsto91, 2\mapsto94, 3\mapsto93, 4\mapsto92, 5\mapsto90, 6\mapsto\bot\}\) recovers all mentorship matches under renumbering, yielding \(s = 0.77\) despite Eve being omitted, agenda items transposed, and a year-off error.
  • Evaluated on AMR datasets where referential alignment reduces to the same variable-alignment problem as Smatch; OA is compared against Smatch++ as the closest non-JSON relative.
  • Synthetic experiments isolate specific score properties (partial credit, order sensitivity, identifier invariance) under controlled conditions.

(Headline quantitative ablation numbers are not provided in the supplied text beyond the above.)

Limitations

  • Referential alignment approximates graph isomorphism via WL refinement; WL can fail to distinguish non-isomorphic graphs (e.g., certain regular graphs), so the bijection may be suboptimal on adversarial or highly symmetric structures.
  • User-supplied embedding-based primitive comparators break determinism.
  • Schema must be manually annotated with OA-specific extensions; initial setup cost exists for new tasks.
  • Empirical evaluation is limited to GEPA as the optimizer; generalization to DSPy/TextGrad is demonstrated architecturally but not empirically in the available text.
  • The paper is a preprint under review, not yet peer-reviewed.

Relevance to Agentic AI / LLM Agents

Structured JSON output — tool calls, function arguments, agentic plans, knowledge-graph triples — is the primary interface between LLM agents and their execution environments, and reliably scoring that output is a foundational requirement for both evaluation and automated prompt/policy optimization. OA directly addresses the scoring bottleneck for agentic planning and tool-calling pipelines by providing a deterministic, schema-aware reward that can replace LLM-judge calls in the inner loop of prompt optimizers like GEPA or DSPy. The referential alignment contribution is especially relevant to multi-hop agentic tasks where outputs encode entity graphs (e.g., extracted knowledge graphs, dependency plans) whose node identifiers are arbitrary — a failure mode that invalidates exact-match and most prior structural metrics. The ranked repair feedback mechanism also opens a path toward self-correcting agent loops driven by structural diffs rather than expensive LLM critiques.