Tuition is expensive.
In engineering, we don't call them "failures." We call them "tuition."
Sometimes tuition costs you a late night. Sometimes, it costs the company $12,000 in OpenAI credits in a single weekend.
Today, I’m sharing a real story from the trenches of AI Engineering. It’s a mistake I’ve seen teams make when moving from batch processing to real-time streams.
Let’s ensure you never pay this tuition bill.
1. THE HORROR STORY: The Cyclical Commit
The Setup: A team wanted to build a "Real-Time User Search." The Architecture:
Source: A PostgreSQL database with a
userstable.Trigger: A Change Data Capture (CDC) stream (Debezium) listening for updates.
Consumer: A Lambda function that takes the text, hits OpenAI for an embedding, and writes the vector back to the
userstable so it can be queried.
The Logic:
Python
def process_stream_event(event):
user_id = event['id']
bio_text = event['bio']
# 1. Generate Vector
vector = openai.embeddings.create(input=bio_text)
# 2. Write Vector back to DB
db.execute(
"UPDATE users SET embedding = %s WHERE id = %s",
(vector, user_id)
)
The Catastrophe: Can you spot the bug?
The Lambda function updated the users table. The UPDATE command counts as a "Change Event." The CDC stream saw the change and fired the Lambda function again. The Lambda function embedded the bio (again) and updated the table (again).
The Result: Infinite recursion. They deployed on Friday at 5:00 PM. By Monday morning, they had processed the same 5,000 users millions of times. Bill: ~$12,000 in API credits. Database: Locks everywhere.
2. THE FIX: The "Idempotency Check" & Separation
There are two ways to fix this. One is code, one is architecture. Architecture is better.
Fix A: The Hash Check (Code)
Before calling the API, check if the content has actually changed.
import hashlib
def get_hash(text):
return hashlib.md5(text.encode()).hexdigest()
def process_stream_event(event):
new_hash = get_hash(event['bio'])
current_hash = event.get('content_hash')
# CIRCUIT BREAKER: Stop if content hasn't changed
if new_hash == current_hash:
print("Skipping: No content change.")
return
vector = openai.embeddings.create(input=event['bio'])
# Update with the new hash to prevent next run
db.execute(
"UPDATE users SET embedding = %s, content_hash = %s WHERE id = %s",
(vector, new_hash, event['id'])
)
Fix B: Separation of Concerns (Architecture)
This is the "Senior Engineer" solution.
Never write derived data (vectors) back to the source table if that table is the trigger for the pipeline. Isolate the Source of Truth from the Index.
Table 1:
users(Raw Data). Triggers the event.Table 2:
users_vectors(Derived Data). The Lambda writes here.
Since the Lambda writes to a different table, the CDC stream on Table 1 never fires a second time. The loop is physically impossible.
3. THE CEREBRAL GYM: Solution & New Puzzle
Yesterday's Solution (The Mutable Default Trap)
The Puzzle: Why did def add_to_cache(item, cache=[]) share data between users?
The Answer: In Python, default arguments are evaluated only once—at function definition time, not at execution time. The list [] is created once in memory. Every time you call the function without a second argument, it points to that exact same list object.
The Fix: Use None as the default.
# The Correct Pattern
def add_to_cache(item, cache=None):
if cache is None:
cache = [] # Creates a NEW list every time
cache.append(item)
return cache
Today's Puzzle (Docker Networking)
You are running a Python script inside a Docker container. You want to connect to a PostgreSQL database running on your host machine (your laptop), not inside another container.
You try: db_host = "localhost" or db_host = "127.0.0.1"
The Error: Connection Refused.
The Question: Why does "localhost" fail inside the container, and what is the special DNS name (on Docker Desktop) required to break out of the container and hit the host?
(Reply with the DNS name!)
4. THE PULSE: Industry Signals
LangChain "LangGraph": LangChain has realized that "chains" (DAGs) are too rigid for agents. They just released LangGraph to support cyclic, stateful loops. This is the future of agentic workflows.
AWS "Q" Developer: Amazon's answer to Copilot is rolling out. The killer feature? It can analyze your AWS account resources to suggest infrastructure code (IaC).
Vercel's "AI SDK 3.0": They are pushing "Generative UI"—where the LLM returns a React component instead of just text. Imagine asking a chatbot for "stock prices" and getting a live rendered chart.
5. THE LATENT SPACE
"Experience is simply the name we give our mistakes."
The difference between a Junior and a Senior engineer isn't that the Senior writes bug-free code. It's that the Senior has already made the $12,000 mistake and has the scars to prove it.
If you broke something this week: Document it. Share it. Learn from it. Then it’s not a failure. It’s an asset.
Please share your some production experiences in the comments so that others can learn from it.
Until tomorrow,
Harsh Kathiriya - Query & Context