Skip to content

MINDS: A Cross-cultural Dialogue Corpus for Social Norm Classification and Adherence Detection

🕒 Published (v1): 2025-11-13 03:33 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Norm-RAG is a four-stage retrieval-augmented agentic pipeline for social norm classification and adherence/violation detection in multi-turn bilingual dialogues. The authors also release MINDS, a 31-session corpus of real Mandarin-English and Spanish-English conversations annotated at the utterance level for norm category and adherence status. Norm-RAG consistently outperforms zero-shot LLM baselines by structuring norms into four pragmatic attributes and grounding inference in dynamically retrieved norm documentation.

Problem

Prior norm datasets (SocialChem, NormDial, SocialDial) target isolated utterances or synthetically generated dialogues, lacking fine-grained annotations for intent, interpersonal framing, and turn-level context shifts. No real-world, bilingual, multi-turn corpus existed with per-utterance norm adherence labels, and no agentic framework modeled how normative context evolves across dialogue turns.

Method

Norm-RAG is a four-stage pipeline:

  1. Semantic Chunking for Norm Chunking — a norm document is embedded sentence-by-sentence using a sentence transformer; a greedy algorithm segments by cosine similarity threshold \(\epsilon_\text{seg}\), then merges adjacent clusters whose mean embeddings satisfy \(\cos(\mu_k, \mu_{k+1}) > \epsilon_\text{merge}\), yielding semantically cohesive norm chunks \(\{C_k\}\).

  2. Structured Norm Attribute Extraction — each chunk \(C_k\) and each dialogue window \(\bar{D}_t\) are independently mapped via GPT-4o-mini to a four-part attribute tuple: Communicative Intent (CI), Interpersonal Framing (IF), Linguistic Features (LF), and Contextual Triggers and Constraints (CTC).

  3. Attribute-Based Semantic Retrieval + Re-ranking — the aggregated similarity between dialogue and norm chunk is \(\text{sim}_k = \frac{1}{4}\sum_{i=1}^4 \varphi(a^i_{D_t}, a^i_{C_k})\); top-\(n\) candidates exceeding a global threshold \(\mu_\text{sim}\) are re-ranked by an LLM that generates a natural-language justification.

  4. Dialogue-Aware Norm Detection Agent — an agentic loop processes utterances sequentially; at each step \(t\), the LLM receives \(\bar{D}_t\), retrieved norm chunks \(\{N_1^*, \ldots, N_n^*\}\), attribute vectors, and a carry-over feedback signal \(F_{t-1}\) encoding tone, contradiction, or escalation observations from the prior turn. It outputs norm category \(y_t\), adherence status \(s_t \in \{\text{adhered}, \text{violated}\}\), an explanation, and updated \(F_t\).

Key Contributions

  • MINDS corpus: 31 real bilingual multi-turn sessions (16 Mandarin-English, 15 Spanish-English), per-utterance norm category and adherence/violation labels, multi-annotator consensus with Cohen's Kappa IRR analysis.
  • Norm-RAG: end-to-end agentic RAG pipeline with feedback-driven temporal coherence across dialogue turns.
  • Semantic Chunking: block-diagonal clustering of norm documents replacing keyword matching with embedding-space segmentation.
  • Four-dimensional norm representation (CI, IF, LF, CTC) enabling interpretable, attribute-disentangled retrieval.
  • Benchmarking of both closed-source (GPT-4o-mini) and open-source (LLaMA 3.1 8B, Qwen-3 32B, Phi-4 14B) baselines under ablated input configurations.

Results

  • Norm-RAG w/ GPT-4o: 70.4% norm accuracy, 63.6% adherence/violation accuracy vs. GPT-4o-mini zero-shot baseline of 64.6% / 57.2%.
  • Norm-RAG w/ LLaMA 3.1 8B: 64.4% / 54.4% vs. 56.4% / 44.7% baseline (+14.1% norm, +21.7% adherence).
  • Norm-RAG w/ Qwen-3 32B: 67.9% / 60.2% vs. 61.1% / 54.9% baseline.
  • Norm-RAG w/ Phi-4 14B: 69.1% / 60.2% vs. 66.4% / 58.6% baseline (modest +2.7% norm; slight adherence drop of −2%).
  • Adding dialogue history alone yields small gains across models; adding retrieved norm documentation consistently drives the larger improvements.

Limitations

  • MINDS is small (31 sessions, ~1,800 annotated turns); conclusions about cross-cultural generalization are limited by dataset scale.
  • Only two language pairs (Mandarin-English, Spanish-English); transferability to other cross-cultural dyads is untested.
  • Phi-4 shows marginal or negative gains from Norm-RAG retrieval, suggesting the pipeline's benefit is model-dependent and may break for instruction-tuned models preferring concise prompts.
  • Annotation coverage is partial: no single annotator covered all sessions, and many turns were reviewed by only one annotator despite the multi-annotator design.
  • The recency window length \(l\) in \(\bar{D}_t\) is determined by an LLM call, introducing a latent hyperparameter with no fixed ablation.
  • \(\epsilon_\text{seg}\) and \(\epsilon_\text{merge}\) are tuned via grid search on the same corpus used for evaluation; risk of overfitting chunking hyperparameters.

Relevance to Harnesses / Meta-Harnesses

Norm-RAG is a concrete example of a multi-stage LLM harness: a deterministic pipeline that chains semantic chunking, structured attribute extraction, retrieval+reranking, and a stateful agentic loop — each stage invoking an LLM with a distinct purpose and passing structured intermediate state forward. The carry-over feedback variable \(F_t\) is an explicit harness-level mechanism for maintaining coherent state across agent invocations within a session, directly analogous to context-propagation patterns in agentic meta-harnesses. The design choice to decompose norm knowledge into four independent attributes before retrieval mirrors the "decompose-then-aggregate" compositional pattern common in research harnesses that fan out subtasks to specialized agents. For researchers building harnesses, the attribute-disentangled retrieval stage (Eq. 6) is a reusable pattern for grounding any structured reasoning task in domain documentation.