Skip to content

SoMe: A Realistic Benchmark for LLM-based Social Media Agents

🕒 Published (v1): 2025-12-09 08:36 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SoMe is the first comprehensive benchmark for evaluating LLM-based social media agents across 8 tasks (event detection, misinformation detection, user simulation, recommendation, etc.) using ~9.2M real posts, 6,591 user profiles, and 17,869 annotated queries from 32 platforms. Evaluation of 13 mainstream agentic LLMs reveals that no current model handles social media agent tasks satisfactorily, with most scoring below 70/100 and many below 50 on open-ended tasks.

Problem

Existing evaluations of LLMs on social media are single-task, use limited data, and often lack ground truth (relying on LLM-as-judge without references). No benchmark provides a multi-task, tool-augmented, realistic environment covering the full spectrum of social media agent capabilities—from post analysis to user behavior prediction to cross-modal reasoning.

Method

SoMe constructs a benchmark platform with 8 MCP-compatible agent tools (e.g., SearchPost, RetrievePost, PostClustering, RetrieveKnowledge) that agents invoke via step-by-step reasoning. Data is sourced from 32 platforms (X, Weibo, etc.), with images/videos converted to text via Qwen2.5-VL-72B and GPT-4o. Annotations follow a semi-automated pipeline: template-based ground truth for behavioral tasks (UBP, UCS, MCR) and human-LLM interactive annotation (Qwen3-32B + 10 professional annotators) for open-ended tasks (RED, SES, UEA, SMQ). Evaluation uses task-specific metrics—LLM-based semantic scoring for open-ended tasks and accuracy for classification tasks—plus a Task Completion Rate (TCR) measuring successful multi-step tool-chain execution.

Key Contributions

  • First versatile, realistic benchmark for LLM-based social media agents spanning 8 tasks across 3 categories (post-centered, user-centered, comprehensive).
  • Dataset of 9,164,284 posts, 6,591 user profiles, 25,686 fact-check reports, and 17,869 annotated queries from 32 platforms.
  • An MCP-compatible agent platform with 8 tools enabling realistic multi-round data acquisition and reasoning.
  • Systematic evaluation of 13 agentic LLMs (closed- and open-source), including challenge analysis of tool-chain planning errors, tool-use hallucination, and parameter mistakes.

Results

  • Best API-based model: Gemini-2.5-Flash (avg. 54.33/100); best open-source: Qwen3-32B (avg. 53.98/100).
  • Most models score below 50 on RED and SES (e.g., Llama-3.1-8B: RED=3.37, SES=20.78); only SMQ and MCR see scores above 60 for top models.
  • Llama-3.1-8B-Instruct achieves only 65.76% average TCR, indicating frequent tool-chain failure.
  • DeepSeek-R1-0528-Qwen3-8B underperforms Qwen3-8B across all 8 tasks by 21–56%, showing enhanced reasoning does not translate to agentic social media capability.
  • Tool-use hallucination rates: DeepSeek-R1-0528-Qwen3-8B (29% wrong format), Devstral-Small-2507 (28%); even Kimi-K2-Instruct (1T params) hallucinates tool responses at 7%.
  • Larger models generally outperform smaller ones (Llama-3.3-70B outperforms Llama-3.1-8B by 52.7% on average score).

Limitations

  • Benchmark data skews toward English and Chinese content from X and Weibo; cross-lingual and platform diversity is limited.
  • Visual/audio modalities are pre-converted to text captions/ASR, removing native multimodal reasoning requirements from agents.
  • LLM-based evaluation metrics for open-ended tasks (RED, SES, SMQ) introduce scorer bias and may not perfectly capture factual correctness.
  • No ground truth for some tasks was independently verified beyond LLM-generated annotations filtered by human annotators, which may propagate model-specific biases.
  • The platform is a simulated environment, not a live deployment; real-time latency and API rate-limit challenges are not modeled.

Relevance to Agentic AI / LLM Agents

SoMe directly benchmarks the core agentic loop—tool selection, multi-round execution, long-context reasoning—in a uniquely noisy, open-world domain that existing benchmarks (OSWorld, WebArena) do not cover. The finding that enhanced reasoning (DeepSeek-R1) degrades agentic tool-use performance is a significant empirical signal for the field, suggesting that reasoning and agency are orthogonal capabilities requiring separate training signals. The MCP-native tool framework and the hallucination-in-tool-use analysis provide actionable diagnostics for agent developers. For anyone tracking LLM agent evaluation, SoMe fills an important gap at the intersection of social intelligence, tool use, and multi-task agentic reasoning.