Language Models as Measurement Apparatus for Culture

🕒 Published (v1): 2026-07-02 17:25 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This position paper argues that using language models to measure culture is a material-discursive practice: the apparatus (model architecture, training data, annotation scheme) actively constitutes the cultural reality it measures rather than passively recording it. Drawing on Karen Barad's concept of the agential cut, the author shows that every design choice — label taxonomy, context window, modality, norm baseline — draws a contingent boundary between phenomenon and instrument. Three case studies on film/TV dialogue operationalize structure, interaction, and deviation as distinct modes of cultural measurement.

Problem

Computational cultural analytics lacks an explicit theory of what it means to measure culture with NLP tools. Existing frameworks treat model, data, task, and metric as separable pipeline stages, but for cultural measurement this separation collapses: LLMs have already internalized the cultural material they are asked to measure (training-time entanglement), and design choices are constitutive of — not preliminary to — the measurement. This is a prior and distinct problem from the representational gap between data and cultural reality.

Method

The paper develops a methodological framework centered on two constructs:

1. Agential cut (from Barad 2007): the contingent boundary an apparatus enacts between what counts as phenomenon vs. instrument. A design choice rises to an agential cut when it changes which cultural phenomena can appear in the measurement (e.g., label taxonomy, training corpus, anonymization procedure), not mere implementation details (random seeds, batch sizes).

2. Three modes of cultural measurement instantiated through case studies on TV/film dialogue:

3. Structure — conversation disentanglement: a BERT-based model predicts reply-to links \(P(\text{parent}(u_i)=u_j) \propto \exp g(h_i, h_j)\), extracting a directed thread graph from scripted multi-party dialogue (808 movies).
4. Interaction — conversational role attribution: a multimodal model assigns Goffmanian speaker/addressee/side-participant labels to 350,842 utterances across four TV series; gender effects isolated via multinomial logistic regression \(\log \frac{P(\text{role}=j)}{P(\text{role}=\text{speaker})} = \beta_{j,0} + \beta_{j,\text{female}} \cdot \mathbb{1}(\text{female}) + \sum_s \gamma_{j,s} \cdot \mathbb{1}(\text{show}_s)\).
5. Deviation — stereotypic relation extraction: a Longformer trained on 787 pilot teleplays with an attentive-pooling context mechanism (\(\alpha = \text{softmax}(w_A^\top h_S \odot M)\)) measures subversion as the gap between model predictions and ground-truth relationship labels.

The paper also examines three apparatus-level interventions: cultural marker erasure, historical-period attunement via fine-tuning, and distribution of measurement across agentic workflows.

Key Contributions

• Imports Barad's agential cut into cultural analytics as a precise, methodologically bounded concept distinguishing substantive design choices from incidental ones.
• Frames cultural NLP as a material-discursive practice where the apparatus is always already entangled with its object (pre-training on cultural material).
• Proposes taxonomy of agential cut types: structural (formalization), categorical/modal (role label + modality selection), and normative (learned baseline vs. deviation).
• Demonstrates that standard NLP evaluation logic (accuracy = goal, error = failure) inverts for cultural measurement: prediction error can be the primary signal (subversion measurement).
• Identifies the agentic workflow as a new apparatus configuration in which autonomous agents each carry independent cultural memory, creating compound entanglement.

Results

• Structure / conversational agency (808 movies): Female characters initiate only 30.4% of threads, but when normalized by speaking time, women initiate threads at a rate exceeding their speaking-time share by +1.0 percentage points on average (\(p < 0.05\)).
• Interaction / gendered roles (350,842 utterances, 4 TV series): Men occupy the speaker role 33.0% of the time vs. 28.7% for women; women are cast as addressee (39.4% vs. 38.7%) and side-participant (31.9% vs. 28.3%) at higher rates. Odds ratios for women occupying listening roles: 1.19 (addressee) and 1.20 (side-participant), both \(p < 0.001\).
• Deviation / stereotypic relation extraction (Frasier, 11 seasons): sibling_of prediction probability drops from ~51% in Season 1 to ~27% in Season 11; spouse_of predictions peak at ~19% in Season 4 — formalizing a well-documented "crypto-gay" quality critics have attributed to the Crane brothers' dialogue.
• All results are presented as apparatus-dependent measurements, not objective ground truth.

Limitations

• Purely positional/theoretical paper; no new model is trained or benchmark established.
• Case studies are limited to scripted Western (US) television and film, constraining generalizability.
• The text is truncated; examinations of cultural marker erasure, historical attunement, and the agentic workflow apparatus are described abstractly in the abstract/intro but not detailed in the provided excerpt.
• The agential cut framework offers principled critique but does not prescribe concrete remedies for entanglement; it acknowledges every cut forecloses alternative measurements without providing selection criteria.
• Deviation measurement depends entirely on the training corpus's stereotypic baseline; no cross-corpus robustness analysis is reported.

Relevance to Agentic AI / LLM Agents

The paper's most direct hook for this topic is its treatment of the agentic workflow as a measurement apparatus: when cultural measurement is distributed across multiple autonomous agents each carrying independent "memory of the culture it measures," compound entanglement arises that single-model deployments do not exhibit. This raises an underexplored alignment concern — agent memory is not a neutral retrieval system but a culturally formed prior that shapes what the agent can perceive and report. More broadly, the paper's insistence that LLM design choices constitute (not merely influence) their outputs is a theoretical framing relevant to any agentic system where the agent's cultural formation shapes its behavior as an actor, not just a classifier. The framework contributes a vocabulary (agential cut, material-discursive entanglement) for diagnosing hidden normative commitments baked into agentic pipelines operating on social or cultural content.