Mike's Notes
Great read here. It takes humility to write this. Thanks, Ananth.
Data Engineering Weekly is an excellent newsletter to subscribe to.
Resources
References
Repository
- Home > Ajabbi Research > Library > Subscriptions > Data Engineering Weekly
-
Home > Handbook >
Last Updated
30/10/2025
Thinking Like a Data Engineer
By: Ananth Packkildurai
Data Engineering Weekly: 23/10/2025
Ananth Packkildurai is a data engineering leader, writer, and author of Data Engineering Weekly, sharing insights on modern data platforms, large-scale pipelines, and AI-driven architectures.
I thought becoming a data engineer meant mastering tools. Instead, it meant learning how to see. I thought the hardest part would be learning the tools — Hadoop, Spark, SQL optimization, and distributed processing. Over time, I realized the real challenge wasn’t technical. It was learning how to think.
Learning to think like a data engineer — to see patterns in chaos, to connect systems to human behavior, to balance simplicity and scale — is a slow process of unlearning, observing, and reimagining. I didn’t get there through courses or certifications. I got there through people.
Four mentors, in four different moments of my life, unknowingly gave me lessons that shaped how I approach engineering, leadership, and even life. Each taught me something not about data, but about thinking systems.
What follows isn’t a tutorial. It’s a map of how four people — and their lessons — rewired how I think.
#1. “Chasing Knowledge.”
One of my friends recently asked why you are constantly reading and writing. It all started with an internship. A family friend of mine helped me find an internship. He was the person who taught me Java — patiently explaining not just syntax, but how to think through logic, abstraction, and design.
When I called him after getting my first full-time job, I expected congratulations or career advice. Instead, he said something that I only understood years later:
“Don’t chase money. Chase knowledge. Money will follow.”
The advice struck a chord with me forever. In technology, everything changes — languages, frameworks, stacks, even paradigms. But curiosity compounds. The more you learn, the faster you learn. The more you focus on mastering fundamentals, the easier it becomes to adapt when the next wave arrives.
That advice became a quiet compass throughout my career. Every time I faced a decision — whether to take a higher-paying role or a role that stretched my skills — I thought back to his words. And every single time I chose learning, the money eventually caught up.
It taught me that data engineering is not a sprint to expertise, but a lifelong apprenticeship in curiosity.
#2. “Modeling the World.”
It was in my early days of career as a data engineer that I met another mentor — an industry veteran, family friend, and one of the most grounded data people I’ve ever known.
One day, I asked him a question I thought was simple:
“How do I become a data engineer?”
He didn’t answer directly. He just said, “Go to Walmart and tell me what you see.”
I didn’t get it. But I went.
I walked through aisles, looked at shelves, and bought a few things — shampoo, snacks, and batteries. I came back and said, “Okay, I went. I bought x, y, z. Now what?”
He smiled and said, “Now tell me — how would Walmart model this data?”
That’s when I stopped seeing stores — and started seeing systems. I started describing the data model — a shelves table, a products table, a users table, and an events table. I started seeing not a store, but a database. Every product placement was a joint. Every checkout was an event stream.
That exercise reshaped how I saw the world. Data engineering wasn’t about ETL jobs or pipelines. It was about modeling reality.
When you can take something as ordinary as a store and translate it into entities, relationships, and flows, you start to think like a data engineer.
That mental model helped me immensely later. During one of my interviews at Slack, I was asked to design a data warehouse for Netflix. I didn’t start from schemas or metrics. I started by thinking: what’s the store? What are the products? What are the customers?
That grounding in observation and modeling helped me move fast and think clearly.
That day at Walmart was my real introduction to data modeling — not through theory, but through the physical world.
#3. “Thinking in Systems.”
When I started my career in the U.S., I worked on a loyalty analytics system — one of my first end-to-end data design projects.
I spent weeks sketching, refining, changing, and rebuilding the system. Every week, it looked different. One day, as I stepped into the elevator, my boss turned to me and said,
“I saw your design. You changed it completely from last week. That’s good. It means you’re thinking. Keep it going.”
That short elevator ride changed the way I viewed iteration. Until then, I thought redesigning meant I’d made a mistake. But his words reframed it — evolution is the process of engineering.
I started seeing every design review not as a checkpoint, but as a conversation.
Every new diagram was not a failure of the previous one, but an iteration toward clarity.
As we worked together more, he became my go-to mentor — not just for technical guidance, but for how to think. One day, over coffee, I asked him a question that had been on my mind:
“How do you cultivate architectural thinking?”
He smiled and gave me an unusual exercise.
He said, “Every day you walk from Montgomery Street to the Caltrain. It’s what — fifteen minutes? During that walk, observe how the city works. Watch how people who don’t know each other coordinate. Notice how traffic lights, signs, and signals function without central control. You’ll start to see the same patterns in our system design from emergent behaviour to the leader-follower model.”
It sounded odd, but I did it.
Each evening, I walked through downtown San Francisco — watching the rhythm of people, cars, and crosswalks. I noticed how one signal turning green in one intersection triggered waves of motion down the street. I saw bottlenecks at corners, retries when people crossed late, failovers when lights malfunctioned, and officers took control.
And suddenly, distributed systems weren’t abstract anymore. They were everywhere.
Humans, too, were part of a system — loosely coordinated, mostly independent, but connected through protocols, feedback, and flow.
That was the moment I truly understood system thinking — the ability to look beyond components and see interactions, dependencies, and evolution.
It also taught me humility: a good architect doesn’t impose order; they design for emergence. They build for change, not for perfection.
To this day, when I review a data platform design or a data flow diagram, I still imagine that walk from Montgomery Street to Caltrain. Systems are just cities — made of processes instead of people. Once you see it, you can’t unsee it.
#4. “Believing You Belong”
After I moved to the USA, I spent more time doubting myself than writing code.
Every meeting felt like an exam I hadn’t studied for. Every question felt like a test of belonging. I carried a quiet imposter syndrome — that whisper in your head that says you just got lucky.
One afternoon, my boss noticed my hesitation during a design review. After the meeting, he stopped me and said something simple that stayed with me forever:
“If you’re sitting here, you’re already good enough.”
It was a short sentence, but it landed deeply. There was no lecture, no performance review, no pep talk — just a fact. If you’re in the room, it’s because you earned it.
That moment changed how I carried myself. I realized confidence doesn’t come from knowing everything; it comes from recognizing that you’ve earned your seat — and you can grow from there.
I’ve repeated that same line to dozens of people since then — junior engineers, interns, even peers who were struggling to see their worth. Because the truth is, the data world can be intimidating. There’s always a new framework, a new paper, a new “modern” stack. You’ll never feel fully caught up.
But you don’t need to.
You need to keep showing up, keep learning, and keep thinking.
That sentence became the foundation for something else — it taught me that data engineering, at its core, is a confidence game. You’re constantly making decisions under uncertainty: what schema to use, how to partition data, how to handle scale. Doubt will paralyze you faster than a bad design.
Good engineers are not fearless; they just keep thinking despite fear.
Looking Back
When I look back now, I realize that everything I learned about data engineering started with a mindset shift, not a technical one.
- Don’t chase money — chase knowledge. → Focus on curiosity; mastery compounds faster than rewards.
- You’re thinking — keep it going. → Iterate relentlessly and observe systems beyond code.
- Go to Walmart. → Learn to model the world in data.
- If you’re sitting here, you’re good enough. → Believe in your ability to learn and grow.
The tools, frameworks, and clouds will keep changing. The mental models won’t.
Those mentors taught me that thinking like a data engineer is not about syntax or pipelines; it’s about curiosity, observation, and humility.
It’s about realizing that the best engineers don’t just build systems — they listen to them.
They watch how systems behave, evolve, and sometimes break — and they design with that reality in mind.
These lessons stayed with me — not as quotes to remember, but as ways of seeing the world.
So, if you’re early in your career or doubting your place in the data world, remember this:
You don’t need to know everything to think like a data engineer. You just need to keep observing, iterating, and thinking.
Because in the end, data engineering is less about moving data — and more about understanding how the world moves.
Closing Thoughts
If I had to summarize years of learning into one sentence, it would be this:
“Thinking like a data engineer is not about data. It’s about systems — human, technical, and everything in between.”
The world around you is the best classroom. Every store, street, and signal is a distributed system waiting to be understood. Every mistake is a feedback loop waiting to be improved. Every mentor is a node in your network of thought.
Keep learning. Keep thinking. And most importantly — keep observing.
Once you start seeing the world as data, you realize you've always been engineering it.
