Chaos is the new Normal.
Yesterday, we talked about moving vectorization upstream. Today, we need to talk about what we are actually vectorizing.
For the last 40 years of database history, the Schema was King. If data didn't fit into VARCHAR(255) or INT, it was rejected.
In the AI age, 80% of the valuable data is unstructured. It’s PDFs, Slack threads, call recordings, and images.
As Data Engineers, we are no longer just "Table Architects." We are "Blob Wranglers."
Let’s dive in.
1. THE CORE: The "Schema-on-Read" Renaissance
Why the "Medallion Architecture" (Bronze/Silver/Gold) breaks for AI.
The classic Databricks/Delta pattern assumes you can refine data into perfect tables.
Bronze: Raw JSON.
Silver: Cleaned Tables.
Gold: Aggregated Metrics.
But you cannot aggregate a PDF. You cannot "clean" a webinar recording into a row and column format without losing the semantic meaning.
The New Architecture: The Metadata Sidecar
For AI pipelines, we don't try to force the content into a schema. Instead, we force a schema on the metadata.
When you ingest unstructured data (like a contract PDF), your pipeline should split:
The Blob: Goes to Object Storage (S3/GCS) $\to$ Vector Database (Chunks).
The Sidecar: A rigid JSON object that travels with it.
The Sidecar Schema:
{
"file_id": "uuid",
"data_source": "sharepoint",
"permissions_group": "hr_access_only",
"creation_date": "2026-01-02",
"version": "1.0",
"entity_tags": ["contract", "vendor_x", "urgent"]
}The Engineering Shift:
Stop trying to parse the PDF text into columns. It’s futile.
Instead, spend your engineering effort ensuring the Sidecar Metadata is 100% accurate.
Why? Because when your RAG system fails, it won't be because the vector similarity search failed. It will be because the LLM retrieved an outdated contract because your metadata didn't have a valid version or date tag for filtering.
In 2026, the Metadata is the Schema.
2. THE TOOLKIT: Enforcing Structure with Pydantic
If we rely on metadata, we can't let it be messy. We need strict validation. Python’s Pydantic is the industry standard for this.
Here is how to enforce a "Sidecar Schema" in your ingestion pipeline so bad metadata never hits your Vector DB.
from pydantic import BaseModel, Field, field_validator
from typing import List, Optional
from datetime import datetime
# Define the Strict Schema for your Metadata Sidecar
class DocumentMetadata(BaseModel):
doc_id: str
source: str = Field(..., pattern="^(slack|sharepoint|email)$")
tags: List[str]
created_at: datetime
is_sensitive: bool = False
# Custom Validator: Enforce business logic
@field_validator('tags')
def check_tags(cls, v):
if len(v) < 1:
raise ValueError('Every document must have at least one tag for RAG retrieval')
return v
# The Ingestion Function
def validate_ingest(raw_payload: dict):
try:
# This acts as a firewall.
# If the data doesn't match the schema, it crashes here, not in production.
meta = DocumentMetadata(**raw_payload)
# Proceed to Vector DB insertion
print(f"Ingesting valid document: {meta.doc_id}")
return meta.model_dump()
except ValueError as e:
print(f"Ingestion Rejected: {e}")
# Send to Dead Letter Queue
# Example Payload
payload = {
"doc_id": "101",
"source": "slack",
"tags": [], # This will fail validation!
"created_at": "2026-01-02T08:00:00",
"is_sensitive": True
}
validate_ingest(payload)3. THE PULSE: Headlines
Azure AI Search Update: Microsoft just rolled out "hybrid retrieval" by default. If you aren't using keyword + vector search combined, you are now officially behind the platform standard.
Python 3.14 Alpha: The "No-GIL" (Global Interpreter Lock) removal is becoming stable. This is huge for Data Engineers running heavy compute tasks on single nodes.
The "Context Window" Trap: A new paper shows that filling a 1M token context window degrades reasoning by 40%. Lesson: RAG is not dead; you still need to retrieve small, relevant chunks.
4. THE CEREBRAL GYM: Solution & New Puzzle
Yesterday's Solution (SQL Deduplication)
The Challenge: Delete duplicates from a streaming table, keeping only the latest timestamp, without doing a slow self-join.
The Elegant Solution: QUALIFY If you are using Snowflake, BigQuery, or Databricks, stop using JOIN for this. Use the Window Function filter.
-- The Modern Way
DELETE FROM user_clicks
WHERE click_id IN (
SELECT click_id
FROM user_clicks
-- Assign a rank to every row based on time (1 = Newest)
QUALIFY ROW_NUMBER() OVER (
PARTITION BY click_id
ORDER BY clicked_at DESC
) > 1
);Why? It scans the table once. It’s cleaner to read. It handles the logic in memory.
Today's Puzzle (Python Logic)
We are building a cache for our API. Look at this Python function carefully.
def add_to_cache(item, cache=[]):
cache.append(item)
return cache
print(add_to_cache("User A"))
print(add_to_cache("User B"))
The Question: You expect the output to be ['User A'] and then ['User B']. But it actually outputs ['User A'] and then ['User A', 'User B']. User B just saw User A's data! 😱
Why does this happen and how do you fix it? (Reply with the fix!)
5. THE LATENT SPACE
"A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable."
In 2026, AI pipelines are the ultimate distributed system. An API change in OpenAI, a rate limit in Pinecone, or a schema drift in Salesforce can break your bot.
Build defensively. Assume everything outside your code will fail.
Thank you for reading today’s edition. That’s all for today’s issue.
💡 Help me get better and suggest new ideas at [email protected]
😊 New reader? Subscribe here
See you tomorrow.
— Harsh Kathiriya