Case Study

Knowledge-Graph RAG for ESG Banking Reports

Client: Academic Research Role: AI Engineer / Researcher

TL;DR

GRI-aligned KG-RAG over Vietnamese bank sustainability reports. Neo4j knowledge graph + hybrid retrieval (BM25 + dense + cross-encoder) + claim-level LLM-as-judge fact-checking against GRI Universal Standards 200/300/400. 88.14% accuracy on 1,440 expert-annotated QA pairs across 18 banks.

Neo4j Knowledge Graph Qwen3-8B Llama 3.1-8B OLMoCR FAISS BM25 Cross-Encoder Reranker LLM-as-a-Judge FEVER GRI Standards PyMuPDF Cypher

Problem

ESG disclosure for Vietnamese banks has moved from a PR exercise to a hard compliance and capital-markets metric — but the reports themselves are brutal to query.

  • Heterogeneous PDFs: 28–80 page sustainability reports with multi-column layouts, embedded tables, mixed heading hierarchies, and scanned figures.
  • Vocabulary mismatch: local Vietnamese framing rarely aligns one-to-one with GRI Universal Standards (200 / 300 / 400 series), so lexical retrieval falls over.
  • Hallucination risk: standard RAG invents numbers or misattributes them across companies and years — unacceptable when an answer is going to a regulator or an analyst.
  • Multi-hop reasoning: real analyst questions cross sections (e.g. green credit balance × scope-1 emissions × board-diversity ratios) — single-shot vector search can’t compose the answer.

Phase 1 — Foundation

System Architecture — 5 Modules at a Glance

Two layers: offline (Modules 1–3) builds the Neo4j graph from GRI standards and bank PDFs; online (Modules 4–5) handles user questions with KG-driven retrieval and post-generation fact-checking. The rest of the page walks each module in order.

High-level system architecture — five modules covering metadata extraction, report processing, knowledge graph construction, KG-driven RAG, and automated fact-checking

Module 1 — GRI Metadata Extraction Pipeline

The reference layer. PyMuPDF parses official GRI standards (Universal + 200/300/400) into chunks; an LLM with few-shot ESG examples emits a 5-tuple per disclosure. This becomes the controlled vocabulary every downstream module aligns to.

GRI metadata extraction pipeline — PyMuPDF parsing followed by LLM-driven semantic extraction into a 5-tuple structured record

Module 2 — ESG Reports Preprocessing

The raw-PDF side. OLMoCR flattens chaotic multi-column layouts and embedded tables; RAP + ALIGN anchors the ToC; Qwen3-4B’s sliding window finds semantic breakpoints, not fixed-size chunks. Output: 4-level labeled disclosure segments ready for the graph.

ESG reports preprocessing pipeline — OLMoCR linearization, RAP and ALIGN ToC anchoring, Qwen3-4B sliding-window segmentation, hierarchical labeling

Phase 2 — Knowledge & Retrieval

Module 3 — ESG Knowledge Graph in Neo4j

Neo4j graph with 5 node types and explicit relational edges. Each statement node carries both its semantic vector embedding and its provenance metadata (company, year, page). This is the move that unlocks graph traversal and 100% source-traceable answers downstream.

ESG knowledge graph construction — 5 node types and relational edges including HAS_REPORT, CONTAINS, RELATED_TO, BELONGS_TO

Module 4 — KG-Driven Hybrid RAG

The online path. A user question becomes a structured Cypher query against the Neo4j graph — not the usual top-K vector lookup. A dual-stream scorer (semantic + keyword) fuses candidates; the top set goes to the LLM with full provenance attached, so every generated sentence is traceable to its source.

KG-driven retrieval-augmented generation pipeline — question understanding, Cypher candidate generation, dual-stream scoring, hybrid evidence fusion, provenance-grounded generation

Phase 3 — Reliability: Automated Fact-Checking

Module 5 — Fact-Checking Pipeline Overview

The reliability layer. Every answer is decomposed into atomic claims; each claim gets its own retrieval, reranking, and verdict pass. This flips RAG from “sometimes hallucinates” into something a compliance team can put in front of a regulator.

Six-stage automated fact-checking pipeline — atomic claim extraction, query generation, hybrid retrieval, rerank and distill, LLM-as-judge, aggregation

Hybrid Retrieval & Cross-Encoder Reranking

Two parallel retrieval channels: internal (BM25 + dense FAISS over the private ESG corpus) and external (Google-compatible web search). A cross-encoder catches what bi-encoders miss — numeric agreement, negation, unit mismatches — producing the final top-K evidence set.

Stage 3-4 evidence construction — parallel hybrid retrieval from internal BM25 + dense FAISS and external web search, then deduplication, cross-encoder reranking, and top-K distillation

Verdicts, Confidence Scoring, Review-Trigger

LLM-as-judge assigns each claim a FEVER-style label (SUPPORTS / REFUTES / NOT_ENOUGH_INFO). Labels aggregate to a signed confidence in [-1, 1]; below threshold, a manual-review flag fires. The system never silently ships a low-confidence answer — the bar for regulated workflows.

Verdict aggregation — LLM-as-a-judge assigns FEVER-style labels (SUPPORTS / NOT_ENOUGH_INFO / REFUTES), then signed and overall confidence is computed and a manual review flag fires below threshold

Phase 4 — Validation

Qualitative Case Studies — VPBank, ACB, BIDV

Three worked examples. VPBank green credit (+48.5%) — surfaces volume and reach. ACB female managers (51%) — adds board-level diversity context. BIDV electricity efficiency — preserves each year’s normalization basis so cross-year tracking stays honest.

Three qualitative case studies — VPBank 2024 green credit growth, ACB 2023 gender equality in management, BIDV cross-year electricity efficiency tracking

Outcome

  • 88.14% overall answer accuracy on the expert-annotated benchmark (KG_Enhanced + Qwen3-8B) — up from 69.57% on baseline KG and 53.65% on BM25.
  • +30 points Recall@3 vs BM25 baseline (83.0% vs 52.5%); KG-aware retrieval is the lever, cross-encoder reranking is the multiplier.
  • Expert-validated benchmark: 1,440 question-document pairs (120 questions × 12 reports) annotated by ESG domain experts. Balanced across Environmental (45.8%), Social (26.7%), and Governance (27.5%) — designed to stress-test the system, not flatter it.
  • Claim-level reliability layer: FEVER-style verdicts with signed confidence in [-1, 1] and a manual-review trigger below threshold — production-grade pattern for regulated domains where silent failure is unacceptable.
  • Provenance to the page: every answer traceable to company, year, page, and GRI indicator. Non-negotiable for ESG analytics, audit, and regulator-facing workflows.
  • Patterns that translated forward: KG + hybrid retrieval + cross-encoder reranking + LLM-as-judge fact-checking — the same patterns I later applied to production AWS RAG systems at FPT Software for Fortune 500 clients.

Research credit

Co-authors: Nguyen Minh Quang, Vuong Anh Binh, Dam Hong Phuc, Le Xuan Hieu. Supervisor: MS Le Dinh Huynh (FPT University).