Enterprise Internal AI Chatbot
TL;DR
Built an on-premise enterprise chatbot for HG Group — RAG-grounded answers over the company's internal document library. Hybrid dense + keyword retrieval with hierarchical chunking, multi-LLM routing across GPT and Gemini APIs, and a self-hosted observability stack on Langfuse + Prometheus + Grafana.
Problem
HG Group needed a chatbot that could answer internal questions over a sprawling document library — and it had to live on-premise for compliance.
- Heterogeneous source formats: PDFs, Word docs, PPT decks, Excel sheets, and scanned PNG images.
- On-premise constraint: most of the stack had to be self-hosted; only the LLM call could touch external APIs.
- Audience was non-technical: HR, recruiters, and business stakeholders — answers needed to be grounded with source citations, not hand-wavy summaries.
- Latency expectation: trusted answer with sources in seconds, not tens of seconds.
Architecture — On-Premise RAG
User Flow — How It Works
The user-facing loop. Employees ask via web or mobile chat; the AI brain searches PDF, Docx, PPT, Excel, and scanned images and returns a grounded answer with citations in seconds. Citations are non-negotiable — business stakeholders only trust answers they can verify against the source.
Data Ingestion Pipeline
The offline pipeline. Raw docs hit format-specific parsers (PDF, Docx, PPT, Excel); scanned images go through Gemini 2.5 Flash OCR. Parsed text is split via hierarchical chunking — large parents with small children, each contextualized by prepending its previous and next sibling so retrieval never loses surrounding meaning. BGE-M3 embeddings (multilingual, 1024-dim) served via Ray Serve land in a dual index: Qdrant for dense, MongoDB for keyword + metadata.
RAG Pipeline with Retrieval
The online path. A query is embedded (BGE-M3) and parsed into keywords in parallel. Hybrid search runs both: Qdrant dense + MongoDB keyword each return top-K children. Candidates are fused, deduped, and reranked with a Cross-Encoder (ms-marco-MiniLM-L-6-v2). An auto-merge layer promotes to parent chunk when many siblings hit, otherwise keeps children. The final context goes into a prompt, and a multi-LLM router picks GPT or Gemini by query type and cost — output is a grounded answer with citations.
On-Premise AIOps — Model Layer
Everything except external LLM calls lives on-prem. The app layer (Chatbot API + Ingestion Service), the model layer (BGE-M3 + ms-marco-MiniLM cross-encoder via Ray Serve), and the data stores (Qdrant + MongoDB) all run locally. Only GPT, Gemini, and Gemini 2.5 Flash OCR are cloud-only by necessity. The observability stack is unified: Langfuse for LLM traces and cost; Prometheus for CPU/GPU/latency metrics; Grafana for dashboards and alerts. The on-prem constraint forced disciplined resource budgeting and model versioning — patterns that translated cleanly into the AWS-native systems I later built.
Outcome
- Trusted answers in seconds — grounded responses with source citations across 5+ document formats.
- Hybrid retrieval (Qdrant dense + MongoDB keyword + cross-encoder rerank + auto-merge) handled both semantic and exact-phrase queries without query-routing overhead at the front door.
- On-premise deployment met the compliance bar without sacrificing model quality — external LLM calls were the only off-prem dependency.
- Multi-LLM routing (GPT + Gemini) gave the team a cost/quality dial per query type, and avoided single-vendor lock-in on the answering layer.
- Unified observability via Langfuse + Prometheus + Grafana meant every step — embedding latency, retrieval recall, LLM cost, GPU utilization — was visible from one pane.
- Patterns that translated forward: the hierarchical chunking, hybrid retrieval, and reranking patterns built here informed the production work I later did at FPT Software for Fortune 500 clients.