The Data Engineering Lifecycle and Undercurrents, 4 Years Later
And what I'd change today in Fundamentals of Data Engineering
Four years ago yesterday (July 26, 2022), Matt Housley and I shipped Fundamentals of Data Engineering into the world with basically no idea how it would land. Turns out, it landed quite well.
Four years is long enough that I’ve started asking myself what if I would write this book the same way today.
The short answer is mostly yes. With some caveats.
The Data Engineering Lifecycle, One More Time
The core of FoDE was always two things stacked on top of each other. The data engineering lifecycle:
Generate
Store
Ingest
Transform
Serve
And the undercurrents running underneath all five stages, which is the stuff that touches every single one of them anyway:
Security
Data management
DataOps
Data architecture
Orchestration
Software engineering
That undercurrents weren’t originally in the cards, by the way. Matt and I kept hitting the same wall while writing the book - where does stuff like security go? Where does data management go? It doesn’t live in one stage. It runs through all of them, all the time. That’s the entire idea of an undercurrent, and I hadn’t seen anyone lay it out that way before. It turned out to be the most useful part of the book, IMO. It all seems super simple and “easy” in retrospect.
What I’d Add If I Were Writing It Today
Two things come to mind.
More depth on data modeling, although I’m addressing this elsewhere. It’s the whole reason Mixed Model Arts exists. Although this has been the data industry’s GTA 6, it’s due in August 2026 (confirmed date pending), and it’s a deeper treatment of exactly the gap I’d point to in FoDE if you asked me what’s missing. The book series is several hundred pages, so probably not a fit for FoDE.
The other candidate is FinOps as a seventh undercurrent. I’m not fully sold yet, but the case is there. Cost has become its own discipline that cuts across every stage the same way security does.
Does AI/ML Deserve a Seat at the Undercurrents Table?
This is the one I genuinely go back and forth on, and I’m not going to pretend I’ve landed somewhere just to sound decisive. I’m quite indecisive on this, and the answer is “it depends.”
The case for yes: AI/ML pervades everything now, the same way security or orchestration does. It touches generation, it touches serving, it touches every stage in between.
The case for no: AI amplifies the existing undercurrents more than it’s becoming its own. Orchestration is going agentic. Software engineering increasingly means directing a model instead of hand-writing every line. It’s improving how the work gets done, but does this make it an undercurrent?
I understand both sides of that argument. I don’t have a settled answer. AI/ML will feature prominently in data engineering no matter what.
Why This Still Matters
Not a day goes by that I don’t get a message from someone about how the book, the lifecycle, and undercurrents changed how they think about their work, their team, or their entire data org. Small companies. Fortune 10 companies. Governments. Universities and colleges. The US military. Everything in between. It’s always cool to see this, and it never gets old.
The actual test of whether a mental model works is whether people are still reaching for it four years and one AI boom later. We didn’t want to write a technology-centric book, as that goes stale quickly. Plus, it’s easier to write. We apparently like punishment. Explaining Airflow, or picking a tool, is relatively simple by comparison. The actual hard part of writing FDE was the abstraction: pulling out what data engineers had actually been doing for decades, stripping away the specific technology, and depicting it simply enough that it becomes an operating framework instead of describing tools that go out of fashion by the time the book is published. It’s not the first lifecycle anyone’s ever drawn. But drawing one simple enough to operate from is a fundamentally different kind of work than a tools guide, and it’s a big part of why there aren’t many books that even attempt it.
Whether there’s a second edition of FDE at some point…there’s probably some appetite for it. I’m not ruling it out. I’m also very busy at the moment, so don’t hold your breath.
For now, happy fourth birthday to Fundamentals of Data Engineering. Thanks to everyone who read it, argued with it, and told me where it actually helped.
Back to the new one now. August. Stay tuned.
Listen to this as a podcast
If you haven’t read Fundamentals of Data Engineering for whatever reason, order it today.
How are you thinking about knowledge management and context engineering? In some ways it seems to parallel the data engineering lifecycle, but also it diverges and sometimes calls for more governance/permission management so that meaning reflects decision rights to create meaning and isn’t just defined by whoever takes the time to engage with definition work.
FoDE was one of the first reads that really helped me conceptualize my career path, what I had been doing and what I wanted to do in the future. Will forever be grateful to you and Matt 🙏