The Tool Trap
If you Google "How to become a Data Engineer," you get hit with a wall of logos. Spark / Kafka / Airflow/ Snowflake / Databricks / Flink / dbt
It is overwhelming. It is also a trap.
Most juniors try to memorize the tools. "I need to get a certification in Spark." "I need to watch a 10-hour Kafka tutorial."
But tools change. Five years ago, everyone used Hadoop. Now it's dead. Five years from now, we might not use Airflow.
The architecture, however, never changes.
For the month of February, we are ignoring the tools. I am launching the Data Engineering 101 Series. We will focus on the First Principles of moving data.
The Mental Model: The 5 Stages of Data
To build a production data platform, you don't need to know 50 tools. You need to solve 5 specific problems. This is our syllabus for the next 4 weeks:
1. Ingestion (Getting it in) Data lives in messy places (APIs, logs, databases). How do we extract it without crashing the source system?
The Debate: Batch (Once a day) vs. Streaming (Real-time).
2. Storage (Keeping it safe) Why can't we just dump everything into a Postgres database?
The Concepts: Data Warehouse vs. Data Lake vs. Data Lakehouse.
The Formats: Why
Parquetis faster thanCSV.
3. Processing (Cleaning it up) Raw data is useless. We need to transform JSON logs into financial tables.
The Method: ETL (Extract-Transform-Load) vs. ELT.
The Engine: Distributed Computing (how to process a petabyte when your laptop only has 16GB of RAM).
4. Orchestration (The Traffic Controller) Step 2 must happen after Step 1. If Step 3 fails, who creates the alert?
The Concept: The DAG (Directed Acyclic Graph).
5. Serving (The Value) Data is worthless until it is consumed.
The Output: Dashboards (BI), ML Models, or Reverse ETL (sending data back to Salesforce).
and much more…
The Library: Required Reading
Every issue in this series will include high-value resources. To start, these are the only two links you need to bookmark for this month.
Fundamentals of Data Engineering (The Bible for Data Engineering) By Joe Reis and Matt Housley. This book is the inspiration for this series. It explicitly avoids tool-hype and focuses on the lifecycle of data.
The Google MapReduce Paper (2004) This 10-page PDF started the entire industry. It explains why we need distributed systems. If you can read and understand this paper, you know more than 90% of boot camp grads. Link:https://static.googleusercontent.com/media/research.google.com/en//archive/mapreduce-osdi04.pdf
The Takeaway
Amateurs talk about Tools ("I know Spark"). Professionals talk about Architecture ("I know how to handle late-arriving data in a distributed stream").
This month, we become professionals. Class starts tomorrow.
See you tomorrow.
Harsh Kathiriya - Query & Context