Skip to content

What Makes an Ideal Quote? Recommending "Unexpected yet Rational" Quotations via Novelty

🕒 Published (v1): 2025-12-15 12:19 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NovelQR is a quotation recommendation system grounded in the principle that ideal quotes are "unexpected yet rational," combining a generative label agent for deep-semantic retrieval with a token-level novelty estimator that mitigates auto-regressive continuation bias in logit-based metrics.

Problem

Existing quotation recommendation systems optimize for surface-level semantic relevance, producing contextually appropriate but clichéd suggestions. They fail to capture the deep meanings behind quotations (even strong LLMs score poorly on raw quote text alone) and lack any mechanism to quantify or maximize the aesthetic novelty dimension that users demonstrably prefer.

Method

NovelQR operates in three stages:

  1. Label Enhancement (offline): A generative label agent (Qwen3-8B) enriches the quotation knowledge base through four steps — comprehensive analysis using author/source context, deep-meaning generation (<50 words), multi-round self-critique correction (up to R=3 rounds, rejecting ~4.6% of outputs), and extraction of structured semantic labels (Core Domain, Value, Insight, Applicability, Sentiment).

  2. Rationality Retrieval: Deep-meaning embeddings replace raw quote embeddings for Top-N=50 retrieval. A label-similarity hard filter (threshold T=0.7 on Core Domain/Value/Insight dimensions) prunes semantically incoherent candidates.

  3. Novelty Reranking: A token-level novelty score SN computes weighted log-probability differences between unconditional and context-conditioned distributions. To mitigate auto-regressive continuation bias, tokens are weighted by the second-order difference of their self-perplexity sequence (|δ₂(t)|), concentrating weight on "novelty tokens" — positions of sudden PPL change — and down-weighting predictable continuation segments. Final score combines SN (λ₁=0.70), web-based popularity SP (λ₂=0.20), and semantic match SM (λ₃=0.10).

Key Contributions

  • User study (N=964) formalizing "unexpected yet rational" as the target objective for quotation recommendation, grounded in Shklovsky's defamiliarization theory.
  • Generative label agent pipeline converting raw quotations into multi-dimensional deep-meaning labels, enabling label-enhanced retrieval that outperforms quote-text-based retrieval.
  • Identification of auto-regressive continuation bias in standard logit-based novelty estimators (surprisal, KL-divergence) and a second-order-difference token-weighting scheme to mitigate it.
  • NovelQR-Bench: a new bilingual benchmark test set (100 instances) spanning diverse real-world domains.
  • LLM-as-judge framework calibrated against expert human ratings (ICC=0.81/0.76, ρ>0.79 correlation).

Results

  • Main comparison (averaged across QuoteR, QUILL, NovelQR-Bench): LR+Ours achieves HR@5=0.70, nDCG@5=0.51, MRR@5=0.45, outperforming the best baseline (LR+GPT: 0.66/0.47/0.43) and QUILL (0.15/0.12/0.11).
  • Label-enhanced retrieval vs. quote-based: Match score improves from 3.99 (QR) to 4.55 (LR) on the combined test sets.
  • Novelty estimation ablation (NovelQR-Bench): Ours (Qwen3-8B) achieves HR@5=0.70, nDCG@5=0.51 vs. Surprisal (0.55/0.44), KL-Div (0.61/0.43), and adding novelty-token weighting to Surprisal improves it from 0.55→0.62/0.44→0.45.
  • Human preference: 78% of human raters select NovelQR's recommendation in a multiple-choice study.
  • Cross-model robustness: Novelty estimator performance is stable across Qwen3-0.6B through Qwen3-32B and Llama3-8B/GLM3-6B.
  • Online latency: ~772ms (±431ms/-30ms) per query.

Limitations

  • Novelty is inherently subjective; the LLM-as-judge proxy cannot fully capture individual variation in perceived unexpectedness.
  • Offline KB construction cost scales linearly with quotation pool size, making very large pools expensive to label initially.
  • The web-based popularity signal (Bing exact-phrase counts) is collected at fixed UTC snapshots and may not reflect real-time or culturally specific familiarity.
  • Evaluation datasets are limited to 100 instances per test set, which constrains statistical power for fine-grained comparisons.

Relevance to Agentic AI / LLM Agents

The generative label agent architecture — where an LLM iteratively analyzes, generates, self-critiques, and extracts structured outputs from opaque knowledge items — is a direct instance of agentic tool-use patterns (multi-step reasoning with self-correction loops) applied to knowledge base construction. The identification and mitigation of auto-regressive continuation bias in logit-based scoring is relevant to any agent system using LLM self-evaluation or reward modeling to rank candidate actions or outputs. The "retrieve-then-filter-then-rerank" pipeline with a neural agent enriching the retrieval index is a reusable design pattern for RAG-based agents that must balance relevance with diversity or unexpectedness in their retrieved context.