Data Engineer / GenAI & Agentic AI Builder
AkshayThakare
Building scalable data pipelines, exploring intelligent AI systems, and sharing practical engineering knowledge.
Watch the tutorials on YouTubeData engineering
Pipelines, warehouses and the reliability that makes data trustworthy.
Python, SQL, Apache Spark, Kafka, Apache Airflow, Snowflake, Databricks, AWS Glue
AI engineering
LLM apps, retrieval and agents that do useful work with that data.
LLMs, RAG, LangChain, LangGraph, MCP, Claude, Hugging Face
Start here
Pick the path that matches where you are today.
New to data engineering
Watch the roadmap, then learn SQL, Spark and the cloud in order.
StartBuild a real project
Follow the Walmart data engineering project end to end, with code and failure handling.
StartMove into GenAI & agents
Understand LLMs, RAG and tool-calling agents with animated explanations.
StartPrepare for interviews
Practise answers that explain the decision, the evidence and the trade-off.
StartFind your starting point
Pick a goal and where you are today. You get a short reading list, in order, that you can start right now.
Your reading list
- 1CTEs vs Subqueries vs Temp Tablesbeginner2 min
- 2NULL in SQL: The Rules That Break Queriesbeginner2 min
- 3Reading and Writing JSON, CSV and Parquet in Pythonbeginner2 min
- 4Type Hints and Dataclasses for Cleaner Pipeline Codebeginner2 min
- 5Normalization in Plain English: 1NF, 2NF, 3NFbeginner2 min
About 10 minutes of reading in total.All topics
The tools behind every project
From ingestion and warehousing to retrieval and agents — taught with the same stack used in production.
- Python
- SQL
- Apache Spark
- AWS Glue
- Amazon S3
- AWS Lambda
- Snowflake
- Databricks
- Apache Airflow
- Kafka
- PostgreSQL
- MongoDB
- Delta Lake
- Step Functions
- Power BI
- Docker
- LLMs
- RAG
- LangChain
- LangGraph
- MCP
- Claude
- Hugging Face
- Ollama
- Qdrant
- Vector search
- AI agents
- CrewAI
- MLflow
- FastAPI
- PyTorch
- LoRA / QLoRA
Flagship case study
One project documented end to end — architecture, code, failure handling and an honest record of what has and hasn’t been run.
🚀 Walmart Data Engineering Project | End-to-End AWS & Snowflake Pipeline
An end-to-end retail pipeline built on synthetic data. Point-of-sale transactions in PostgreSQL and inventory in MongoDB are extracted by arrival time. Every store-day is reconciled against the store's close report, and daily sales and inventory KPIs are MERGEd into Snowflake. The local reference is verified by 12 passing tests; the AWS and Snowflake code is supplied but not yet run in the cloud.
DynamoDB3
Tested locally
9
Code supplied
1
Planned
10 chapters
- 01Overview & the retail problem
- 02Architecture & why each service
- 03Source systems & synthetic data
- 04Incremental extract & raw validation
- 05Reconciliation, late data & transformation
- 06Snowflake COPY INTO & MERGE
- 07Orchestration & alerting
- 08Run it yourself
- 09Verified vs planned & next steps
- 10Power BI dashboard & data usage
Interview question of the day
One question a day. Think about your answer first, then check it.
- Data Engineering System Design
Data Pipeline Observability: Freshness, Volume and Quality
What to monitor so you find broken data before your users do: freshness, volume, schema, quality checks, lineage and alert design.
Intermediate 2 min - Data Engineering System Design
Exactly-Once Is a Myth; Idempotency Is the Answer
Why distributed systems deliver at-least-once, what 'exactly-once' really means in Kafka and Spark, and how idempotent writes give you correct results.
Advanced 2 min - Data Engineering System Design
System Design: A Daily Batch Pipeline, Step by Step
A repeatable answer for the most common data engineering design question: requirements, architecture, data model, failure handling and monitoring.
Intermediate 2 min - SQL
NULL in SQL: The Rules That Break Queries
Three-valued logic, why NOT IN returns nothing, COUNT(*) vs COUNT(col), and how NULLs change joins and aggregates.
Beginner 2 min - SQL
Indexes and Query Plans: Why a Query Is Slow
Read an EXPLAIN plan, understand when an index helps, and the common reasons a database ignores the index you created.
Intermediate 2 min - SQL
CTEs vs Subqueries vs Temp Tables
When to use each, what the optimizer does with them, and how to write long transformations that stay readable.
Beginner 2 min
What’s new
The latest notes, videos and projects, newest first.
- Video: PHP2Shell: The PHP Bug
- Note: Data Pipeline Observability: Freshness, Volume and Qualitysystem-design
- Note: Exactly-Once Is a Myth; Idempotency Is the Answersystem-design
- Note: System Design: A Daily Batch Pipeline, Step by Stepsystem-design
- Note: NULL in SQL: The Rules That Break Queriessql
- Note: Indexes and Query Plans: Why a Query Is Slowsql
Learning paths
Follow a sequence instead of a pile of links. Two tracks, one engineer: each step opens a note you can read today.
Interview preparation
Answers that explain the decision, the reason, the evidence and the limit — not one-liners.
Choose a question
Swipe to browseInterview answer
01 / 06
Why can a JOIN silently inflate your totals?
Joining two child tables of the same parent multiplies rows.
One order line joined to two payment rows becomes two rows, so revenue doubles.
State the grain of every table and aggregate to a shared grain before joining.
Visual walkthrough
| O101 | line 1 | ₹300 |
| O101 | P101 | FAILED |
| O101 | P102 | SUCCESS |
| line 1 | P101 | ₹300 |
| line 1 | P102 | ₹300 |
Wrong: the single ₹300 line is counted once per payment attempt.
Fix: aggregate payments to one row per order first
line 1 · paid ✓ · ₹300 → SUM = ₹300
About me
Senior Data Engineer with experience of designing and delivering scalable data platforms, real-time streaming pipelines and production-grade AI systems. I work across PySpark, Databricks Lakehouse, Delta Lake, Apache Kafka and AWS, and I build with Generative AI — fine-tuning open LLMs with LoRA/QLoRA and shipping LangChain + FAISS RAG pipelines. I created Datainteg.io, an open-source data engineering lab, and I teach everything I build on my YouTube channel.
Get new projects and notes by email
I email when something substantial ships — by hand, not on a schedule. No tracking pixels, and every email has an unsubscribe link.