Language Models as Measurement Apparatus for Culture¶

🕒 Published (v1): 2026-07-02 17:25 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Using LLMs to measure cultural phenomena is a material-discursive practice: the apparatus—model architecture, training data, annotation scheme, evaluation criteria—actively constitutes the cultural realities it purports to measure rather than neutrally recording them. Drawing on Karen Barad's agential cut, the paper develops a framework that treats every design choice in an LLM measurement pipeline as a simultaneous methodological and ethical commitment. Three empirical case studies on film and TV dialogue (structure, interaction, deviation) illustrate how different apparatus configurations produce incompatible cultural measurements.

Problem¶

NLP cultural analytics lacks an explicit theory of what it means to measure culture computationally. The standard linear research narrative (task → data → model → metric) treats these components as separable, but for cultural measurement this collapses: representation, annotation, and evaluation choices are not preliminary to measurement but constitutive of it. Compounding this, LLMs have often already encountered the cultural material they measure during pre-training, creating an entanglement between instrument and object that standard contamination/memorization framings are too narrow to capture.

Method¶

The paper adapts Barad's agential cut—the contingent boundary an apparatus enacts between phenomenon and instrument—for cultural analytics. A design choice constitutes a cut when it determines which cultural phenomena can appear at all (label taxonomy, context window, modality, anonymization, training norm), not merely incidental details (random seed, batch size). Three case studies instantiate distinct cut types:

Structure — BERT-based conversation disentanglement on 808 movie scripts: \(P(\text{parent}(u_i) = u_j) \propto \exp\, g(h_i, h_j)\), recovering directed reply-to thread graphs to measure conversational agency by gender.
Interaction — Multimodal model assigns Goffmanian roles (speaker, addressee, side-participant) to 350,842 utterances across 4 TV series; a multinomial logistic regression with show-level fixed effects isolates gender effects: \(\log \frac{P(\text{role}=j)}{P(\text{role}=\text{speaker})} = \beta_{j,0} + \beta_{j,\text{female}} \cdot \mathbf{1}[\text{female}] + \sum_s \gamma_{j,s}\).
Deviation — Longformer trained on 787 pilot teleplays predicts social relationship types from dyadic dialogue with attentive pooling over ambient scene context (\(\alpha = \text{softmax}(w_A^\top h_S \odot M)\)); prediction errors against ground-truth labels are the cultural signal (subversion), not noise.

The paper also examines the apparatus itself: cultural marker erasure (anonymization), historical attunement via fine-tuning, and distributing measurement across an agentic workflow of autonomous models.

Key Contributions¶

Reconceptualizes LLM measurement pipelines as apparatus that produces cultural reality rather than records it, importing Barad's philosophy of physics into NLP.
Operationalizes the agential cut as a practical criterion: a design choice is a cut if it changes which cultural phenomena can appear in measurement.
Three case studies grounding the framework empirically in scripted film/TV dialogue at scale.
Distinguishes entanglement (instrument has internalized its object via pre-training) from contamination/memorization—a structurally prior problem.
Argues that "error" in cultural NLP models can be the finding (deviation from learned norms as cultural signal), inverting the standard accuracy-maximization paradigm.

Results¶

In 808 movies: female characters initiate only 30.4% of conversational threads, but normalized by speaking time, they initiate at +1.0 percentage points above their share (p < 0.05).
In 350,842 TV utterances: men occupy the speaker role 33.0% vs. 28.7% for women; women are ~1.2× more likely than men to appear in a listening role (odds ratios: addressee 1.19, side-participant 1.20; both p < 0.001).
In Frasier: model trained on stereotypic sibling dialogue predicts spouse_of for the Crane brothers persistently (e.g., 9.8% spouse_of vs. 51.0% sibling_of in Season 1, with spouse_of remaining elevated through Season 11 at 11.4%), formalizing a documented queer-coded quality of their relationship.

Limitations¶

Primarily a theoretical position paper; empirical case studies are illustrative, not systematic comparisons of apparatus configurations.
No formal criterion is offered for distinguishing constitutive (cut) from incidental design choices—the line is drawn by example and argument.
All case studies draw from scripted film/TV dialogue; generalizability to other cultural domains is asserted but untested.
The "agentic workflow" examination (distributing measurement across autonomous agents) is described in the abstract but the paper text is truncated before those findings appear.
Entanglement is identified as a structural problem distinct from contamination, but no mitigation strategy is proposed.

Relevance to Harnesses / Meta-Harnesses¶

This paper directly theorizes the LLM measurement pipeline as an "apparatus"—functionally what the harness/meta-harness literature calls a scaffold—and argues that its configuration is never neutral. The agential cut framework gives vocabulary for analyzing how harness design choices (model selection, context window, taxonomy, evaluation criteria) determine which phenomena can emerge from a pipeline at all, which is precisely the question meta-harness designers face when composing multiple agents or evaluation stages. The paper's treatment of agentic workflows—distributing cultural measurement across autonomous models, each carrying its own pre-trained cultural past—raises a compositional entanglement problem: in multi-agent harnesses, accumulated apparatus-dependence across stages may silently shape what the composed system can find. This positions the work as a philosophical foundation for principled harness design in any domain where the instrument's training history overlaps its measurement target.