Chapter 6: Integrating Your Vector Database: Permission-Aware Search and Data Pruning

Ingesting document chunks with sec:* security envelopes is only half the battle. When a user asks a question, the vector database retriever must ensure that restricted chunks and chunks from other tenants never leak into the model's prompt context. Vector search alone measures only semantic similarity, not authorization. Securing knowledge retrieval requires dynamic query pre-filtering to instruct the database engine to drop unauthorized records, coupled with in-memory post-retrieval verification to discard tampered entries and enforce clearance floors.

Two-Phase Retrieval Architecture

The contextual retrieval filter operates across two deterministic defense stages:

1. Phase 1: Dynamic Filter Push-Down (Pre-Filtering): Extracts SecurityIdentity from LangChain's RunnableConfig and translates caller attributes into a native vector database filter dictionary (e.g., {"$and": [...]}).
2. Phase 2: Post-Retrieval Pruning & Verification: Validates retrieved candidate chunks in memory against the caller's clearance floor, role memberships, and cryptographic HMAC signatures before handing documents to the LLM prompt assembler.

Dynamic AST Pre-Filter Translation

An Abstract Syntax Tree (AST: a hierarchical tree representation of structural code or query expressions) dynamically compiles the caller's identity claims into the vector store's native dialect. This offloads coarse access control directly to the database engine:

langchain_security_extension/security_adapter_langchain/retriever_wrapper.py

# langchain_security_extension/security_adapter_langchain/retriever_wrapper.py
def build_full_ast(self, identity: Optional[SecurityIdentity]) -> PrefilterAST:
    """Constructs unconstrained full pre-filter AST from caller SecurityIdentity."""
    if identity is None or identity.is_anonymous or identity.subject_id is None or str(identity.subject_id).upper() == ANONYMOUS_SUBJECT_ID:
        # Strict Fail-Closed: Unauthenticated callers receive a synthetic deny-all sentinel filter
        return PrefilterAST({SEC_TENANT_ID: "__DENY_ALL_UNAUTHENTICATED__"})

    and_clauses: List[Dict[str, Any]] = [
        {SEC_TENANT_ID: identity.tenant_id},
        {SEC_CLEARANCE_FLOOR: {"$lte": identity.clearance_level}},
    ]
    if identity.roles:
        and_clauses.append({SEC_ALLOWED_ROLES: {"$in": list(identity.roles)}})
    return PrefilterAST({"$and": and_clauses})

Explanation: If an unauthenticated caller issues a query, the method returns a synthetic deny-all filter (__DENY_ALL_UNAUTHENTICATED__), guaranteeing that no records match. For authenticated callers, it binds their tenant ID, clearance floor, and assigned roles into native database query filters.

Post-Retrieval Pruning, Top-K Oversampling, and In-Flight PII

When databases support only partial filtering or when metadata requires cryptographic verification, chunks must be pruned in memory. If a query requests 5 chunks (K=5) and 4 are discarded during clearance checks, the model suffers from Result Starvation (an information-loss condition where access pruning leaves too few documents for accurate reasoning).

To prevent result starvation, SecureRetriever calculates an inflated candidate fetch size before querying the vector store:

K_fetch = min(K_target × OversamplingFactor, K_max)

If post-retrieval pruning still leaves fewer than K_target chunks, the retriever executes adaptive secondary paging up to a bounded threshold (default: 2 attempts). To prevent irrelevant documents from being injected simply to satisfy K_target, candidate oversampling and secondary paging enforce a configurable similarity score cutoff floor (similarity_cutoff, default: 0.0), discarding any candidate whose similarity score falls below the relevance floor.

Post-retrieval verification also serves as an active in-flight PII enforcement point: when full in-flight sanitization is enabled, retrieved document chunks are intercepted at stage PII_STAGE_RETRIEVAL_DOCUMENT and redacted before entering model prompt context (see Chapter 9: Sanitizing Sensitive Payloads for detailed redaction policies).

Restricted Title Pruning & Telemetry Redaction

When candidate documents are discarded during post-retrieval clearance evaluation, SecureRetriever records the title/header of each unauthorized document into config["configurable"]["sec:pruned_doc_titles"]. Downstream agent telemetry finalizers (such as finalize_telemetry() in app/agents/guardrail_tests_telemetry_helper.py) inspect this list to redact restricted document names from user responses and logs with [REDACTED_RESTRICTED_DOC], preventing indirect data leakage via document titles.

Auditing and Telemetry Events

During query execution and candidate pruning, SecureRetriever emits standardized audit events to the Policy Decision Point (PDP) audit pipeline:

• EVENT_POLICY_TAMPER_DETECTED (Severity: CRITICAL_ALERT): Emitted when a document chunk's sec:envelope_hash fails cryptographic HMAC verification, dropping the tampered record immediately.
• EVENT_RAG_UNAUTHORIZED_PRUNING (Severity: NOTICE): Emitted when candidate chunks are pruned in memory due to missing mandatory security envelope tags, insufficient clearance level, or role mismatch.
• EVENT_RAG_RESULT_STARVATION (Severity: WARNING): Emitted when final authorized candidates returned to the caller are fewer than the requested target top-K despite oversampling and secondary paging attempts.

Vector Store Metadata Type Constraints

Embedded stores like ChromaDB require primitive scalar metadata types (strings, integers, floats, booleans). Complex attributes such as sec:allowed_roles must be flattened into delimited strings during ingestion, and reconstructed during retrieval:

app/agents/knowledge_base_store.py

def to_chroma_metadata(meta: Dict[str, Any]) -> Dict[str, Any]:
    """Normalizes metadata values into ChromaDB-compatible primitive scalar types."""
    chroma_meta: Dict[str, Any] = {}
    for k, v in meta.items():
        if isinstance(v, (str, int, float, bool)):
            chroma_meta[k] = v
        elif isinstance(v, (list, tuple, set)):
            chroma_meta[k] = ",".join(str(item) for item in v)
        elif v is None:
            chroma_meta[k] = ""
        else:
            chroma_meta[k] = str(v)
    return chroma_meta

Explanation: Complex Python types like lists or sets are flattened into comma-delimited strings to comply with ChromaDB's scalar storage requirements, preserving all security envelope attributes without serialization errors.

Code Hardening: Securing Runbook Retrieval

When hardening knowledge retrieval in your enterprise app, wrap your vector retriever with SecureRetriever.wrap() and forward caller identity via RunnableConfig into _retriever.invoke(). In the reference assistant, this transformation is implemented in app/agents/knowledge_base_store.py across two steps:

Step 1: Wrapping the Base Retriever

Wrap the underlying vector store retriever with SecureRetriever.wrap(), configuring candidate oversampling, storage capability profile, and the secondary fetch callback:

app/agents/knowledge_base_store.py

@@ -25,12 +25,18 @@ class KnowledgeBaseStore:
-        # INSECURE: Returns raw retriever without tenant boundaries or clearance checks
-        self._retriever = self._vector_store.as_retriever(search_kwargs={"k": TOP_K_DEFAULT})
+        pdp_engine = get_pdp_engine()
+        base_retriever = self._vector_store.as_retriever(search_kwargs={"k": TOP_K_DEFAULT})
+        self._retriever = SecureRetriever.wrap(
+            base_retriever,
+            pdp_engine=pdp_engine,
+            oversampling_factor=DEFAULT_OVERSAMPLING_FACTOR,
+            secret_key=config.security_hmac_secret,
+            store_profile=STORE_PROFILE_FULL_AST,
+            secondary_fetch_fn=self._secondary_fetch_chroma_page,
+        )

Explanation: Wrapping the base retriever attaches AST filter compilation, cryptographic HMAC validation, and secondary paging. If unauthorized chunks are pruned post-retrieval, _secondary_fetch_chroma_page fetches subsequent batches to satisfy target K.

Step 2: Propagating Caller Identity into Search

Ensure that search methods pass caller identity to _retriever.invoke() inside RunnableConfig:

app/agents/knowledge_base_store.py

@@ -410,12 +425,15 @@ class KnowledgeBaseStore:
     def search(self, query: str, *args: Any, **kwargs: Any) -> List[TextSegment]:
+        run_config: Optional[Dict[str, Any]] = args[1] if len(args) > 1 else kwargs.get("config")
+        if run_config is None:
+            from langchain_security_extension.security_core.identity import IdentityBridge
+            resolved_identity = IdentityBridge.resolve_identity(kwargs.get("identity"))
+            run_config = IdentityBridge.create_runnable_config(resolved_identity)
+
+        # CRITICAL: Caller identity must be passed via RunnableConfig into retriever invoke
+        pruned_docs = self._retriever.invoke(query, config=run_config, top_k=top_k)

Explanation: SecureRetriever.invoke() extracts caller identity from run_config["configurable"]["security_identity"], compiles the dynamic AST pre-filter, queries the database, and verifies candidate envelopes. See app/agents/knowledge_base_store.py in secured branch for the complete search implementation.

Storage Engine Capability Profiles

Vector databases differ in their native filtering capabilities. SecureRetriever negotiates filter predicates based on the underlying storage profile:

Note on ChromaDB and Embedded Stores: While the demo application specifies STORE_PROFILE_FULL_AST for ChromaDB to evaluate basic equality and numeric ranges, ChromaDB does not natively support array containment on unflattened lists or complex boolean operators like $exists. The security adapter applies _sanitize_chroma_filter() to strip incompatible clauses before query execution, relying on Phase 2 post-retrieval verification to evaluate any unsanitized predicates.

Verification Tests (Guardrail Tests)

Validate permission-aware retrieval, AST compilation, oversampling, and cryptographic tamper detection using the project's standardized guardrail suite:

• Test 06: HMAC Envelope Tamper Detection (06-hmac-envelope-tamper-detection.sh)
  Verifies that SecureRetriever re-computes the SHA-256 HMAC (Hash-based Message Authentication Code: a cryptographic signature verifying data integrity and authenticity) signature on chunks post-retrieval, dropping any record whose metadata envelope was tampered with in storage.
  BASH

  ./internal-security-guardrail-tests/06-hmac-envelope-tamper-detection.sh

• Test 15: RAG Clearance Data Leakage Prevented (15-rag-clearance-data-leakage-prevented.sh)
  Confirms post-retrieval clearance gating by asserting that low-clearance callers are blocked from receiving classified runbook chunks into prompt context.
  BASH

  ./internal-security-guardrail-tests/15-rag-clearance-data-leakage-prevented.sh

• Test 16: RAG Top-K Predictive Oversampling and Paging (16-rag-topk-predictive-oversampling-and-paging.sh)
  Validates candidate oversampling multipliers and adaptive secondary paging fallbacks to guarantee result sets meet target K when unauthorized chunks are pruned.
  BASH

  ./internal-security-guardrail-tests/16-rag-topk-predictive-oversampling-and-paging.sh

• Test 17: RAG VectorStore AST Prefilter Translation (17-rag-vectorstore-ast-prefilter-translation.sh)
  Tests the dynamic translation of SecurityIdentity attributes into native vector store metadata filter dictionaries to enforce push-down pre-filtering.
  BASH

  ./internal-security-guardrail-tests/17-rag-vectorstore-ast-prefilter-translation.sh