Chapter 5: Security Labeling and Admission Control in the Ingestion Pipeline

Retrieval-Augmented Generation (RAG) is only as secure as the metadata stored inside the vector index. If document chunks are embedded without cryptographic boundaries, role tags, and multi-tenancy keys, downstream retrievers cannot reliably enforce authorization. Security labeling and admission control in the ingestion pipeline ensure that during the ETL (Extract, Transform, Load: a data processing pipeline that extracts raw data, transforms its structure, and loads it into a storage target) lifecycle, every chunk is stamped with a tamper-evident security envelope and validated against strict admission policies before embeddings are generated and stored.

The sec:* Chunk Metadata Envelope

Vector databases index flat key-value pairs alongside embeddings. To prevent namespace collisions with arbitrary source metadata (such as page_number or author), all access boundaries are unified under the reserved sec:* prefix:

• sec:schema_version: Semantic schema version string (e.g., "1.0").
• sec:tenant_id: Mandatory partition identifier enforcing multi-tenant isolation.
• sec:document_id: Unique identifier linking chunks to their root parent source.
• sec:chunk_index: Zero-based sequential integer denoting the chunk's position in the document.
• sec:classification_level: Hierarchical data classification tier (PUBLIC, INTERNAL, CONFIDENTIAL, RESTRICTED).
• sec:clearance_floor: Numeric clearance rank (0 ≤ PUBLIC < 1 ≤ INTERNAL < 2 ≤ CONFIDENTIAL < 3 ≤ RESTRICTED) used for fast numeric range pre-filtering.
• sec:allowed_roles: List or delimited collection of RBAC roles entitled to read the chunk.
• sec:envelope_hash: Cryptographic HMAC (Hash-based Message Authentication Code: a cryptographic signature verifying data integrity and authenticity) digest ensuring metadata fields cannot be altered inside the vector database.

Ingestion Lifecycle: Labeling, Enrichment & Admission Control

The security labeling and admission pipeline executes across four deterministic stages implemented as LangChain BaseDocumentTransformer primitives:

1. Baseline Validation (DocumentSecurityTransformer): Inspects the raw document for mandatory baseline attributes (e.g., tenant_id, baseline clearance) before text splitting.
2. Document Chunking: Uses standard LangChain text splitters (e.g., RecursiveCharacterTextSplitter) to divide the document into chunks while inheriting parent metadata.
3. Chunk Enrichment (SecureDocumentChunkEnricher): Scans chunk text for structural markdown headers (e.g., ## [CONFIDENTIAL]) or invokes an external SPI (Service Provider Interface: an extensibility pattern allowing custom classes to plug into framework pipelines) to elevate classification tiers.
4. Envelope Validation & Signing (SecurityEnvelopeValidator): Validates all mandatory sec:* tags against the active policy schema and calculates the SHA-256 sec:envelope_hash signature.

Enrichment SPI and Monotonic Elevation

The enrichment stage supports custom DLP (Data Loss Prevention: automated software tools that detect and prevent the exposure of sensitive data) scanners, regex matchers, and structural header parsers via the ChunkSecurityEnricherSPI. All enrichers must adhere to the Monotonic Classification Invariant: an enricher can elevate a chunk to a higher classification tier, but it can never weaken the parent document's baseline classification.

langchain_security_extension/security_adapter_langchain/ingestion.py

# langchain_security_extension/security_adapter_langchain/ingestion.py
from typing import Dict, Any
from langchain_security_extension.security_adapter_langchain.ingestion import ChunkSecurityEnricherSPI, ChunkEnrichmentContext
from langchain_security_extension.security_core.constants import (
    SEC_CLASSIFICATION_LEVEL,
    SEC_CLEARANCE_FLOOR,
    CLASSIFICATION_CLEARANCE_MAP,
)

class HeadingClassificationEnricher(ChunkSecurityEnricherSPI):
    """Enricher that elevates chunk classification based on structural Markdown headers."""

    def enrich_chunk(self, context: ChunkEnrichmentContext) -> Dict[str, Any]:
        envelope = dict(context.metadata)
        parent_clearance = int(envelope.get(SEC_CLEARANCE_FLOOR, 0))

        detected_tier = self._extract_heading_classification(context.text)
        if detected_tier:
            detected_clearance = CLASSIFICATION_CLEARANCE_MAP.get(detected_tier, 0)
            # Monotonic Elevation Invariant: Never downgrade below parent baseline
            if detected_clearance > parent_clearance:
                envelope[SEC_CLASSIFICATION_LEVEL] = detected_tier
                envelope[SEC_CLEARANCE_FLOOR] = detected_clearance

        return envelope

Code Hardening: Enforcing Security Labeling and Admission Control

When hardening document ingestion in your enterprise application, replace unvalidated chunk splitters with an orchestrated SecureVectorStoreIngestor. In the reference assistant, this transformation is implemented in app/agents/knowledge_base_store.py across two focused stages:

Stage 1: Configuring Transformers and SPI Enrichers

Configure the baseline document metadata defaults and register SPI enrichers to automatically scan chunks for classification headings and sensitive credentials:

app/agents/knowledge_base_store.py

@@ -143,15 +143,17 @@ class KnowledgeBaseStore:
+        transformer = DocumentSecurityTransformer.create(
+            default_classification=DEFAULT_CLASSIFICATION_PARAM,
+            default_roles=DEFAULT_ROLES_PARAM,
+            default_tenant=DEFAULT_TENANT_PARAM,
+            default_department=DEFAULT_DEPARTMENT_PARAM,
+            rules=[kb_security_rule],
+        )
+        spi_enrichers = [
+            HeadingClassificationEnricher.create(),
+            DlpSecretDetectionEnricher.create(),
+        ]
+        enricher = SecureDocumentChunkEnricher.create(
+            spi_enrichers=spi_enrichers,
+            secret_key=config.security_hmac_secret,
+            audit_manager=audit_mgr,
+        )

Explanation: DocumentSecurityTransformer stamps mandatory baseline tags (tenant, roles, clearance) onto raw documents. SecureDocumentChunkEnricher executes registered SPI plugins—such as HeadingClassificationEnricher and DlpSecretDetectionEnricher—to detect inline sensitivity markers and elevate chunk classification monotonically.

Stage 2: Assembling the Ingestor with Cryptographic Validation

Next, instantiate the SecurityEnvelopeValidator to enforce schema compliance and sign chunks with HMAC digests, binding everything into SecureVectorStoreIngestor:

app/agents/knowledge_base_store.py

@@ -160,13 +162,14 @@ class KnowledgeBaseStore:
+        validator = SecurityEnvelopeValidator.create(secret_key=config.security_hmac_secret)
+        text_splitter = SecureTextSplitter.create(
+            chunk_size=DEFAULT_CHUNK_SIZE,
+            chunk_overlap=DEFAULT_CHUNK_OVERLAP,
+        )
+        return SecureVectorStoreIngestor.create(
+            transformer=transformer,
+            enricher=enricher,
+            validator=validator,
+            ingestion_mode=ingestion_mode,
+            text_splitter=text_splitter,
+            audit_manager=audit_mgr,
+        )

Explanation: The validator verifies that all required sec:* fields are present, verifies envelope integrity, and calculates sec:envelope_hash using HMAC-SHA256 before chunks enter vector storage. Any chunk failing schema checks is immediately rejected under strict fail-closed mode.

Admission Control & Ingestion Policy Modes

When an unlabelled or malformed chunk reaches the validation boundary, the admission controller in SecurityEnvelopeValidator can be set to enforce one of three operational fallback policies configured via SecureVectorStoreIngestor.create(..., ingestion_mode=...) or the security.ingestion.mode application property. These modes are defined in langchain_security_extension.security_core.constants:

Verification Tests (Guardrail Tests)

Validate chunk classification, header parsing, monotonic elevation, and quarantine handling using the project's standardized guardrail suite:

• Test 01: Ingestion Envelope Validation Enforced (01-ingestion-envelope-validation-enforced.sh)
  Verifies schema validation rules ensuring all required sec:* fields exist prior to embedding generation.
  BASH

  ./internal-security-guardrail-tests/01-ingestion-envelope-validation-enforced.sh

• Test 02: Ingestion Header Classification Elevation (02-ingestion-header-classification-elevation.sh)
  Tests the structural heading parser elevating chunk classification tiers when sensitive markdown headers (e.g., ## [CONFIDENTIAL]) are detected.
  BASH

  ./internal-security-guardrail-tests/02-ingestion-header-classification-elevation.sh

• Test 05: Ingestion Monotonic Classification Elevation (05-ingestion-monotonic-classification-elevation.sh)
  Validates monotonic classification invariants, ensuring enrichers can elevate security tiers but cannot weaken parent document baselines.
  BASH

  ./internal-security-guardrail-tests/05-ingestion-monotonic-classification-elevation.sh

• Test 07: Ingestion Unlabelled Chunk Quarantine Policy (07-ingestion-unlabelled-chunk-quarantine-policy.sh)
  Verifies that chunks missing explicit classification tags are properly processed under quarantine rules during ETL.
  BASH

  ./internal-security-guardrail-tests/07-ingestion-unlabelled-chunk-quarantine-policy.sh