Skip to content

Towards Developing a Multimodal Chat Assistant for University Stakeholders: RAG-based Approach

🕒 Published (v1): 2026-07-01 16:03 UTC · Source: Arxiv · link

Why this paper was selected

RAG-based multimodal university assistant; practical retrieval-augmented VLM deployment case study

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a RAG-based multimodal university chatbot for KUET (Bangladesh) that accepts text and image queries via LLaVA-1.5-7B, grounding responses in institutional documents stored in a ChromaDB vector store. The system reduces LLaVA-1.5-7B's baseline hallucination rate from 31.7% to 6.6% on domain-specific university queries. It is publicly deployed at chat.kuet.ac.bd on commodity hardware using quantized inference.

Problem

Rule-based and static-knowledge chatbots cannot handle complex, nuanced institutional queries or adapt to dynamic policy changes. University intelligent support systems are nearly absent in developing countries like Bangladesh, leaving students and staff without reliable self-service information access. General LLMs suffer from high hallucination rates (~31.7%) when queried on institution-specific content they were not trained on.

Method

The pipeline follows a standard RAG architecture with a multimodal extension: 1. Document ingestion: The KUET university handbook (PDF) is parsed page-by-line; regex cleans remove non-semantic tokens. 2. Chunking: RecursiveCharacterTextSplitter with chunk size 512 characters and 64-character overlap. 3. Embedding & indexing: all-MiniLM-L6-v2 produces 384-dimensional vectors stored in ChromaDB; persisted across sessions. 4. Retrieval: Cosine-similarity search returns top-3 chunks at query time. 5. Augmentation: Retrieved chunks are concatenated into a structured prompt with explicit instruction to answer only from context or refuse. 6. Inference: LLaVA-1.5-7B (quantized 4-bit) handles both text-only and image+text queries via a unified prompt. Text queries optionally skip retrieval if no image is present but still receive RAG context; image queries build a structured prompt combining the visual input with retrieved text context.

The backend is FastAPI (async); the frontend is Next.js/React. Deployment hardware: Intel i7-7700, 32 GB RAM, NVIDIA GTX 1080 (8 GB).

Key Contributions

  • RAG-augmented multimodal university chatbot using LLaVA-1.5-7B, supporting simultaneous text and image queries.
  • End-to-end retrieval pipeline: PDF ingestion → recursive chunking → MiniLM-L6-v2 embeddings → ChromaDB semantic search → context-fused prompting.
  • Quantized inference enabling deployment on a single consumer-grade GPU (GTX 1080, 8 GB).
  • Publicly accessible deployment (chat.kuet.ac.bd) with a feedback collection loop (CSV-logged).
  • Quantitative hallucination reduction from 31.7% to 6.6% (Cohen's \(\kappa = 0.84\) inter-annotator agreement).

Results

  • Hallucination rate: 31.7% (baseline LLaVA-1.5-7B) → 6.6% (RAG-enhanced), evaluated on 160 domain-specific queries.
  • Retrieval accuracy: 78.3% top-1, 92.1% top-3 (system uses top-3).
  • User satisfaction: 4.26/5 overall (52 trial participants); 4.56/5 for text-only queries; 3.95/5 for image-based queries.
  • Response latency: Mean 13.57 s (SD 10.87 s), range 1.23–44.49 s over 160 queries; image queries averaged 16.4 s (~4.6 s overhead vs. text-only).

Limitations

  • High and highly variable response latency (up to 44.49 s) due to 7B-parameter model on a GTX 1080; sequential CPU preprocessing further bottlenecks throughput.
  • Static knowledge base (single university handbook); does not handle dynamic document updates without re-indexing.
  • No multi-hop reasoning; complex queries requiring cross-document inference fail or produce incomplete answers.
  • Contextual integration problems acknowledged but not quantified.
  • Evaluation is limited to one institution's corpus (KUET) with 52 trial users; generalizability is untested.
  • Image understanding relies entirely on LLaVA-1.5-7B without specialized OCR or document-image parsing; complex visual layouts may degrade accuracy.

Relevance to Vision-Language Models

This work demonstrates a practical deployment pattern for VLMs in resource-constrained, domain-specific settings: a quantized LLaVA-1.5-7B serves as the backbone for both visual reasoning and text generation, with RAG grounding compensating for the model's lack of institutional knowledge. The hallucination reduction result (31.7% → 6.6%) is a concrete datapoint on how retrieval grounding interacts with VLM generation quality, relevant to ongoing work on grounded VLMs and multimodal RAG. The paper also highlights the latency cost of multimodal processing (~4.6 s overhead for image queries) as a practical deployment constraint for VLM-based systems on commodity hardware. For VLM researchers, this illustrates both the promise and the bottlenecks of deploying instruction-tuned VLMs in real-world, low-resource institutional settings.