Enterprise AI Contract Intelligence System
TL;DR
Upgraded legacy contract parsing into a skill-based framework where each skill is a self-contained Context-Funnel + Agentic-Loop (ReAct) workflow over Qdrant RAG — boosting extraction accuracy by +35%. Progressive disclosure keeps every LLM call focused; async batch jobs on AWS scale to thousands of documents per run with schema-validated outputs.
Problem
The legacy contract parser was a single mega-prompt over an LLM. Each layer of that simplicity had a cost.
- Hallucinations on unfamiliar clauses — the model invented language when it had no anchor.
- Inconsistent outputs across runs — the same contract returned different fields twice in a row.
- Per-task accuracy invisible — only aggregate scores existed; no way to pin a regression to “price extraction” vs “heading detection.”
- Couldn’t scale — sync REST timed out as contract volume grew.
- Unstructured output — downstream legal systems received free text, not validated schemas.
Architecture — Production
User Flow
The full user journey, framed as human-in-the-loop. A user uploads a contract; the system extracts structured fields through a chain of skills; the user reviews and corrects the output. Each correction is captured and used to refine future extractions. The product is the extraction plus the review experience — not just the LLM call.
High-Level Architecture
The whole stack at a glance. The API gateway routes uploads into an orchestrator, which selects and runs the right Skills. Skills pull grounded context from a Qdrant vector store, call Bedrock for inference, and emit schema-validated structured output. Results land in a result store that downstream legal workflows can consume directly.
Skill-Based Framework
Challenge A single mega-prompt doing price extraction, heading detection, and QA at once is unprompt-able. You can't tune it for one task without breaking another, and a regression is invisible until a downstream system breaks.
The core differentiator. Each capability — price extraction, heading detection, heading review, contract QA — is its own independent Skill. Each Skill is a fully self-contained workflow: it does not coordinate with the others and there is no shared orchestration chain — a request runs one Skill end-to-end. This is what unlocks the +35% accuracy: each Skill is small, focused, separately prompt-engineered, and individually evaluated. Adding a capability is a new Skill, not a rewrite.
Skill Structure (Anatomy)
What lives inside one Skill — an on-disk folder, not a single prompt. SKILL.md carries the base context: metadata, baseline rules, signal-conditional rules, the output schema, and few-shot examples. tools/ holds deterministic Python filters (parse, language detect, currency detect, Qdrant lookup) that run with no LLM cost. references/ holds pattern docs loaded conditionally by signal — only the refs that match the document. At runtime the Skill runs a Context Funnel (deterministic filters → focused XML prompt, ~5× fewer tokens via progressive disclosure), then an Agentic Loop (ReAct): the model extracts, self-evaluates confidence, calls a Qdrant search tool when unsure, and converges to Pydantic-validated output with citations.
RAG Pipeline
The retrieval side. Contract language varies by jurisdiction and template; a raw LLM call hallucinates on edge cases. Qdrant stores embeddings of a reference clause corpus, and inside the Agentic Loop a Skill retrieves the most relevant clauses on demand — when the model’s confidence on a detection is low — rather than on every call. Grounding the LLM in known patterns only when needed is what brings hallucination rates down on unfamiliar language without wasting tokens on the easy cases.
AWS Infrastructure
Production deploy is fully AWS-native. Contracts land in S3, which triggers Lambda to enqueue extraction jobs in SQS. Workers consume the queue, run Skills against Qdrant and Bedrock, and write structured results back to S3 and the result store. VPC and IAM hardening keep the data plane locked down. No vendor lock-in beyond AWS itself.
Async Batch Jobs
Challenge Sync REST cannot stream thousands of contracts through extraction without timing out. One failure cannot poison the whole batch, and a worker restart cannot double-bill Bedrock.
Sync REST cannot stream thousands of contracts through extraction. The async batch path uses SQS-driven workers with idempotent handlers and retries, so a failing contract doesn’t poison the batch and a worker restart doesn’t double-charge Bedrock. Progress is reported back to the user as the batch advances, not just at completion.
Outcome
- +35% extraction accuracy vs the legacy single-prompt system.
- Scales to thousands of contracts per batch via SQS + async workers.
- Independent, self-contained Skills unlocked per-task evaluation and surgical regression fixes — each Skill ships and is tuned on its own.
- Progressive disclosure (Context Funnel) cut each LLM call to ~5× fewer tokens — lower cost and higher accuracy from less prompt noise.
- On-demand RAG grounding cut hallucination rates on unfamiliar contract language without paying retrieval cost on the easy cases.
- AWS-native deploy on S3 + Lambda + SQS + Bedrock + Qdrant — no vendor lock-in beyond AWS.
- The framework is reusable for adjacent legal workflows: NDA review, clause comparison, obligation tracking.