The RAG Pattern That Stops LLMs From Inventing Fields That Don't Exist
Standard RAG retrieves documents and lets the LLM generate field names freely — which means it will invent fields your schema never had. AK-RAG (Attribute Knowledge RAG) indexes your governed attribute catalog instead, and the LLM can only emit attribute_ids that actually exist. Here's the architecture and why it matters for regulated-industry AI.
Table of Contents
Every team building RAG for a regulated enterprise hits the same failure mode eventually. The system retrieves the right context — clinical guidelines, product documentation, data dictionaries — and the LLM produces a confident, fluent response that references fields, thresholds, and identifiers that don’t exist in the actual data model. In a consumer app this is a nuisance. In a bank’s credit underwriting pipeline, a healthcare payer’s cohort identification system, or a compliance team’s AML query layer, it is a governance failure. The LLM invented a field. That field never went through data governance. The output is meaningless at best and misleading at worst.
Standard RAG was not designed to prevent this. It retrieves document chunks and hands them to the LLM as context. The LLM then generates free-form text, including any field names, identifiers, and thresholds it infers from that context. There is no guarantee that what the LLM generates corresponds to anything that actually exists in your data catalog. Prompt-level mitigations — instructions to only use fields from the provided context, few-shot examples, output validation — help at the margins but don’t solve the structural problem. The problem is that the retrieval unit is wrong.
Attribute Knowledge RAG (AK-RAG) is a reference architecture that fixes this at the retrieval unit level. Instead of embedding documents, it indexes your governed attribute catalog as individual knowledge objects — one embedding per attribute, not one embedding per document page. The LLM’s job is phrase extraction and clarification dialog, not field name generation. The only identifiers that can appear in the final output are attribute_id values that exist in the indexed catalog. The LLM cannot invent fields because field selection happens through a governed retrieval and classification pipeline, not through free generation.
Architecture
AK-RAG has two distinct pipelines: an ingestion pipeline that builds the attribute index, and a query pipeline that translates natural language into a governed DSL. Both are designed around a core principle: the attribute catalog is the knowledge layer, not the document corpus.
Ingestion Pipeline
The ingestion pipeline takes your enterprise attribute metadata — typically maintained in Excel, CSV, or exposed via an API — and transforms it into a searchable, versioned index:
Excel / CSV / API
|
v
Contract Validation & Normalization
- Split synonyms
- Parse ranges and thresholds
- Coerce types, standardize units/dates
|
v
NDJSON (One Attribute = One Document)
|
v
┌──────────────────────────────────────┐
│ Embedding Provider │
│ sentence-transformers (dev) │
│ openai · gemini · bedrock (prod) │
└──────────────────────────────────────┘
|
v
┌──────────────────────────────────────┐
│ Search / Index Backend │
│ local BM25 + token vector + RRF │ ← offline, no cluster (dev)
│ opensearch BM25 + HNSW kNN + RRF │ ← production
│ faiss / chroma │ ← dense vector alternatives
└──────────────────────────────────────┘Each attribute becomes a single NDJSON document. Here’s what that looks like for a clinical attribute in the healthcare domain:
{
"attribute_id": "clinical.hba1c",
"type": "numeric",
"business_name": "HbA1c Level",
"technical_field": "hba1c_pct",
"domain": "clinical",
"definition": "Most recent HbA1c percentage on record. Indicates glycemic control...",
"synonyms": ["glycated hemoglobin", "blood sugar control", "a1c", "hemoglobin a1c"],
"operators": [">", ">=", "<", "<=", "="],
"unit": "%",
"example_values": [7.0, 8.0, 9.0, 10.0],
"governance": {
"phi": true,
"hipaa_category": "clinical",
"de_identification_required": false,
"minimum_cell_size": 11,
"allowed_channels": ["care_management", "quality_reporting", "analytics"],
"consent_required": false
},
"embedding_text": "numeric | HbA1c Level | hba1c_pct | glycated hemoglobin | blood sugar control | a1c | glycemic control | diabetes | percentage | clinical"
}The embedding_text field is hand-composed to maximize retrieval signal: it concatenates the type, business name, technical field name, synonyms, definition keywords, and domain tags. This is not a document summary — it is a retrieval-optimized representation of what this attribute means in natural language.
Indexes are versioned by date and promoted via an atomic alias swap:
attributes_v2026_06_26 (previous)
attributes_v2026_07_06 (new)
attributes_current → attributes_v2026_07_06This means a new metadata release never partially updates a live index. If post-deployment smoke tests fail, rollback is a single alias pointer change.
Query / Conversation Pipeline
The query pipeline is a six-step process. The LLM participates in two of the six steps — phrase extraction and clarification dialog generation. Everything else is deterministic:
User Natural Language Input
|
v
┌──────────────────────────────────────┐
│ LLM (Step 1: Parse) │
│ Extract attribute phrases only │
│ "diabetic patients over 65 with │
│ elevated HbA1c" → │
│ ["diabetic", "over 65", │
│ "elevated HbA1c"] │
└──────────────────────────────────────┘
|
v For each phrase independently:
┌──────────────────────────────────────┐
│ Step 2: Hybrid Retrieval │
│ BM25 lexical ──┐ │
│ ├──► RRF fusion │
│ kNN vector ──┘ │
│ Top-k candidates per phrase │
└──────────────────────────────────────┘
|
v
┌──────────────────────────────────────┐
│ Step 3: Decision Policy │
│ │
│ score ≥ EXACT_THRESHOLD (0.92) │
│ → exact: use attribute as-is │
│ score ≥ NEAR_THRESHOLD (0.75) │
│ → near: show top options │
│ multiple near matches │
│ → ambiguous: ask user to choose │
│ no match above threshold │
│ → none: ask for alternative │
└──────────────────────────────────────┘
|
v
┌──────────────────────────────────────┐
│ Step 4: Clarify (LLM) │
│ Generate clarification dialog for │
│ near / ambiguous phrases │
│ User selects from governed options │
└──────────────────────────────────────┘
|
v
┌──────────────────────────────────────┐
│ Step 5: Governance Check │
│ PHI · HIPAA category │
│ consent_required │
│ allowed_channels │
│ minimum_cell_size │
│ Blocks any attribute that violates │
└──────────────────────────────────────┘
|
v
┌──────────────────────────────────────┐
│ Step 6: Emit Governed DSL │
│ Only selected attribute_ids │
│ No free-form field generation │
└──────────────────────────────────────┘The final DSL output contains only attribute_id values that exist in the catalog, plus provenance metadata:
{
"version": "1.0",
"type": "attribute_filter",
"filters": [
{
"attribute_id": "diagnosis.diabetes_type2",
"business_name": "Type 2 Diabetes Diagnosis",
"technical_field": "diabetes_t2_flag",
"source_phrase": "diabetic"
},
{
"attribute_id": "utilization.readmission_30d",
"business_name": "30-Day All-Cause Readmission",
"technical_field": "readmit_30d_flag",
"source_phrase": "readmitted in the last 30 days"
}
],
"unresolved": [
{
"phrase": "elevated HbA1c",
"outcome": "ambiguous",
"options": [
{ "attribute_id": "clinical.hba1c_gt_8", "business_name": "HbA1c Level > 8.0%" },
{ "attribute_id": "clinical.hba1c_gt_9", "business_name": "HbA1c Level > 9.0%" }
]
}
]
}unresolved phrases are surfaced back to the user — the system never silently picks one. This is not a UX courtesy; it is an accuracy requirement. “Elevated HbA1c” maps to two different governed clinical thresholds with different population implications. Guessing wrong has downstream consequences for care management targeting, quality reporting, and regulatory submissions.
The Governance Gap
The reason standard RAG fails for governed enterprise data is that it treats the retrieval problem as a document retrieval problem, when it is actually an attribute selection problem.
When an analyst asks “show me diabetic patients with high HbA1c who were readmitted,” the correct system behavior is to select the right diagnosis.diabetes_type2, clinical.hba1c_gt_9, and utilization.readmission_30d attribute_ids from the catalog, confirm any ambiguous thresholds with the user, and produce a filter expression using only those fields. The system does not need to retrieve paragraphs of clinical documentation to do this. It needs to retrieve governed attribute objects that match the analyst’s natural language phrases.
Standard RAG would embed the clinical documentation — HEDIS technical specifications, ICD-10 codebooks, clinical guideline PDFs — chunk them, and let the LLM synthesize field names from the retrieved text. This produces outputs that sound correct to a non-technical reader but are unreliable as machine-executable data operations. The LLM might generate hba1c_high instead of hba1c_gt_9. It might infer a 30-day readmission window from the retrieved text without knowing whether the enterprise system distinguishes all-cause from condition-specific readmissions. It might produce a filter expression that references fields which simply don’t exist in the target data model.
AK-RAG sidesteps this entirely because the retrieval unit is already a governed attribute. The system cannot generate hba1c_high because hba1c_high is not an attribute_id in the catalog. The system cannot guess at a readmission time window because the decision policy forces clarification when a phrase maps to multiple governed attributes. The governance layer catches any remaining violations — PHI fields in unauthorized channels, attributes below the minimum cohort size threshold — before the DSL is emitted.
Why Hybrid Retrieval (BM25 + kNN + RRF)?
The pattern uses hybrid retrieval because neither lexical nor semantic search alone is sufficient for enterprise attribute lookups.
BM25 excels at exact field name matches, acronyms, and clinical abbreviations. A query phrase of a1c should directly match the synonyms array on clinical.hba1c. Vector search would handle this too but degrades on rare clinical abbreviations without domain-specific embedding training.
kNN vector search handles paraphrase and semantic variation. “Readmitted to hospital last month” is semantically equivalent to “30-Day All-Cause Readmission” but shares almost no tokens. A pure BM25 search would miss this; semantic search finds it.
Reciprocal Rank Fusion (RRF) combines the two ranked lists without requiring the scores to be on the same scale. BM25 scores and cosine similarities cannot be directly averaged — they measure different things. RRF fuses by rank position, which is stable across query types, embedding model changes, and index rebuilds.
The decision thresholds (default: EXACT_THRESHOLD=0.92, NEAR_THRESHOLD=0.75) are configured in .env and should be calibrated on your specific attribute catalog with a labeled evaluation set.
What This Means for Developers
The AK-RAG reference implementation is a working Python vertical slice. You can run it locally without an OpenSearch cluster using the offline local search backend:
# Install with Claude LLM + sentence-transformers (dev default)
pip install -e ".[claude,sentence]"
# Validate your attribute contract
akrag validate data/sample_attributes.csv
# Convert to NDJSON
akrag to-ndjson data/sample_attributes.csv build/attributes.ndjson
# Run hybrid retrieval locally
akrag query build/attributes.ndjson \
"high HbA1c" \
"readmitted in the last 30 days"The pluggable provider matrix means you can start with local search and sentence-transformers during development and swap to OpenSearch + OpenAI embeddings for production without changing any application code — all three layers (LLM, embedding, search) are selected via environment variables:
# Dev: fully offline, no API costs
LLM_PROVIDER=ollama
EMBEDDING_PROVIDER=sentence
SEARCH_PROVIDER=local
# Production (AWS)
LLM_PROVIDER=bedrock
EMBEDDING_PROVIDER=bedrock
SEARCH_PROVIDER=opensearchThe same pipeline works for any enterprise domain. The healthcare cohort-definition use case in the demo is one instance. The pattern is domain-agnostic — the six steps (parse → retrieve → classify → clarify → govern → emit DSL) are identical whether the catalog is clinical attributes, banking risk factors, KYC fields, or insurance underwriting parameters. For banking specifically, the same governance layer that enforces phi, hipaa_category, and minimum_cell_size maps directly to PII/NPI masking, regulatory reporting eligibility, and fair-lending minimum-cohort-size rules:
| Healthcare Governance Field | Banking Equivalent |
|---|---|
phi: true | PII / NPI masked field |
hipaa_category: clinical | regulatory_category: credit_risk |
minimum_cell_size: 11 | Fair lending minimum cohort size |
allowed_channels: [care_management] | allowed_channels: [underwriting, analytics] |
consent_required: true | SAR reporting restriction |
The attribute document shape is the same. The decision policy is the same. The DSL emitter is the same. You swap the catalog.
The SuperML Take
The failure mode AK-RAG addresses is one of the most common and least discussed problems in production enterprise AI: the LLM that confidently generates field names that don’t exist. Everyone who has shipped a RAG system against a structured data model has seen this. It gets patched with prompt engineering, output validation, and retry logic — but it keeps surfacing because the root cause is architectural, not prompt-level.
The root cause is that standard RAG is optimized for document retrieval and free-form synthesis. That is excellent for knowledge-base chatbots, customer support systems, and research assistants. It is the wrong architecture for any system that needs to translate natural language into governed data operations — cohort definitions, compliance queries, credit policy evaluations, AML rule lookups. These systems need deterministic attribute selection, not probabilistic field name synthesis.
AK-RAG’s key insight is that the embedding unit should be the attribute, not the document. A clinical data dictionary has hundreds of attributes. Each attribute has a business name, synonyms, a technical field name, governance metadata, and valid operators. That is exactly what should be embedded and retrieved. When you embed at the attribute level, the retrieval result is always a set of candidate attribute objects. The LLM selects from that set — it cannot generate outside it. The governance layer checks the selected attributes before the DSL is emitted. The system is constrained end to end.
For teams building AI systems in banking, healthcare, or insurance, the practical question is not “should we use AK-RAG?” but “do we have an attribute catalog to index?” Most regulated enterprises already have the catalog — it lives in a data dictionary, a metadata management tool, or a governance platform. The ingestion pipeline accepts Excel, CSV, or API input. The attribute document schema is defined in the repo. Getting a working prototype running against your own catalog should take a day, not a sprint.
The open-source reference implementation at github.com/crazyaiml/attribute-knowledge-rag includes the full pipeline: contract validation, NDJSON generation, offline hybrid retrieval, decision policy, governance check, and DSL assembly. It runs without a cluster in dev mode. Production deployment swaps local search for OpenSearch BM25 + HNSW kNN and the embedded sentence-transformers for a production embedding provider. The architecture is provider-agnostic by design — the LLM, embedding model, and search backend are all pluggable.
If your enterprise AI system is producing field names your data model has never heard of, the fix is not a better prompt. It is a better retrieval unit.
Sources
Enterprise AI Architecture
Want more enterprise AI architecture breakdowns?
Subscribe to SuperML.
