What Does AI Do to The Craft of Software and Data Engineering?
Joe's Nerdy Rants #77 - Weekend reads, podcasts, and other stuff
As I finish Mixed Model Arts, my upcoming data modeling book, I often reflect on a deeper theme that might be my next book: how craftsmanship, depth, and expertise evolve in an AI-first world. My writing process involves a ton of research and conversations with industry leaders. Fundamentals of Data Engineering was written pre-ChatGPT (I think you’ll see this as an ongoing delimiter in discussions about content). For my Mixed Model Arts book, I use AI as a sounding board and an “editor” to poke holes in my thinking and refine my writing. I am not “vibe writing” nor letting the AI write for me, as that cheats me and my readers. But I could churn out a book a day if I chose to. The book wouldn’t be copyrightable and would very likely suck, but it’s a book nonetheless. Amazon is getting flooded with these types of “books”, and I’ve seen over a dozen different AI-generated titles on Amazon called “Fundamentals of Data Engineering.” Imitation is flatter or something…The cost of creating content (writing, code, etc.) is zero. The cost to produce high-quality content with depth is still as high as ever. I don’t see the demand for high-quality content disappearing anytime soon. I think it will grow in our era of AI slop. But what about the impacts on the practitioners who read my books?
Particularly amid the current vibe coding and AI Agent hype cycle, I’ve been thinking a lot about what happens to the craftsmanship, depth, and expertise of devs and engineers. When I talk with engineers, most seem excited by how AI accelerates their work. In what appears to be one of the greatest bouts of cognitive dissonance I’ve ever seen, this excitement also reveals an existential dread that AI will eliminate their jobs. “We’ve probably got a few years left,” is the common refrain. And of course, vendors and VCs are promising agents will usher in an era of unbounded abundance and more productivity (e.g., fewer people). Just vibe code and automate your business with agents and apps. No overpaid and pesky developers required.
There’s a lot of buzz in the AI-generated app space. Databricks just bought Neon in anticipation of these AI apps needing a database backend. IBM recently “estimates that over one billion apps will emerge by 2028” due to genAI. This seems like a great headline, but the number of apps is meaningless without context. This is where I take issue with the notion of letting everyone build apps with AI. On the surface, this is a good idea as it helps non-technical people create apps to solve their problems. I saw the same arguments during the citizen data scientist, data democratization, and no-code tooling waves of the past. A big reason these trends never achieved critical mass is pretty simple. People generally suck at understanding and communicating the problem they’re trying to solve. It’s hard to go deep into something and understand its contours, especially when translating it to code and data. Add in poor data and technical literacy, crappy data, organizational dysfunction, and you start to understand the problem has never been about tools and technology.
While some say that citizen developers will develop apps for a prototype and “local work”, things rarely stay prototypes or local for long. Most people don’t know the difference between a program and a product. One is a toy, the other is, well, a product. Most toys are prototypes, and prototypes have a weird habit of becoming a part of production. Think of how many random scripts, stored procedures, and apps you’ve seen that started off as “this is temporary” to “I guess we need to update this eyesore since it runs a mission-critical process”. Even with AI agents coordinating to manage these systems, I predict a Great Pacific Garbage Patch of AI Slopware will ensue. More apps, more problems. But there’s hope if we take the right approach to our tools and our craft.
Back to craftsmanship. The dichotomy of AI or humans isn’t a zero-sum game. In fact, I’m highly optimistic in the long term. I’ve spent a lot of time thinking about this disconnect between the promise of AI and the dread I hear from the same people using these tools. The misnomer is that engineers are only paid to write code. Code is a means to an end, and is pretty worthless on its own. While writing code can be cathartic, it can also be toil. Think of the death slog of code or data migration. Yuck! We can do better things with our time.
To be useful, code must be directed to solve problems and add value. Engineers are problem solvers. AI will enable engineers to move toward higher-value work of being more business-facing and architecting systems and products. If AI handles the grunt work of writing code, you can shift your focus toward higher-level thinking - improving your craft and business literacy, and architecting better systems. In capable hands, AI is an accelerant providing almost God-like powers to build amazing and useful things. Part of me thinks we’ll need more engineers and technically literate people, not fewer.
AI makes respecting and investing in your craft more important than ever. The future belongs to those who pair AI with mastery, not to those who abandon mastery altogether.
Please listen to the audio above or on Spotify (or your podcast platform of choice).
Have a wonderful weekend,
Joe
Join dbt Labs May 28 for the dbt Launch Showcase to hear from executives and product leaders about the latest features landing in dbt. See firsthand how features will empower data practitioners and organizations in the age of AI.
Thanks to dbt Labs for sponsoring this newsletter.
Cool Weekend Reads
Here are some cool articles I read this week. Enjoy!
Feelings, Facts, and Our Crisis of Truth
From Output to Outcome: Why Defining Outcomes in Data & AI Products Is So Challenging
Airlines prepare for nuclear war
Republicans Try to Cram Ban on AI Regulation Into Budget Reconciliation Bill
Executives Are Pouring Money Into AI. So Why Are They Saying It's Not Paying Off?
Curl project founder snaps over deluge of time-sucking AI slop bug reports
Upcoming Events
I’m excited to be going to London very soon. Join me for the European data streaming event of the year with Confluent!
I’ll discuss the implications of AI-generated content and a digital ecosystem drowning in bad data and software, and how to avoid this!
Join me at #Current25 London and use my discount code L-PRM-JOEREIS for 50% off!
See you there! https://cnfl.io/438XnTV
Nexla’s 2025 Data + AI Integration Summit on May 21 Register here
Analytics and data engineering used to live in separate worlds—different teams, different tools, different goals. But the lines are blurring fast. As modern data products demand speed, scale, and seamless integration, the best teams are embracing engineering principles and best practices.
In this no-BS conversation, Ryan Dolley, Matt Housley, and I, dive into how engineering principles are transforming the way analytics is built, delivered, and scaled.
📆 May 27, 2025 🕘 9:00 AM PDT, 12:00 PM EDT, 6:00 PM CEST 🔗 Register here!
Brought to you by GoodData
Snowflake and/or Databricks - June TBA
Iceland - Global Data Summit, June 23-24. Register here
Australia (Sydney, Melbourne) - Data Eng Bytes, July 24-30. Register here
UK - Big Data London, September 24-25. Register here
Helsinki Data Week - October TBA
More to be announced soon…
Podcasts
Freestyle Fridays - What Does AI Do to The Craft of Dev and Engineering? (Spotify)
Laura McDonald - Navigating the Complex World of Enterprise Sales (Spotify)
Freestyle Friday - Navigating Data Strategy in the Age of AI w/ Dia Adams & Gordon Wong (Spotify)
Michael Drogalis - Building a Company in Public (Spotify)
John Giles - The Data Elephant in the Board Room, Data Modeling, and More (Spotify)
Zhamak Dehghani - Autonomous Data Products, Data Mesh, and NextData - Q&A (Spotify)
Freestyle Friday - Advice for 2025 Graduates (Spotify)
Jessica Talisman - Libraries, Knowledge, Shitty Tech Jobs, and More (Spotify)
Freestyle Fridays - “I don’t need to learn anything anymore.” (Spotify)
Juan Sequeda & Jesus Barrasa - Unlocking Knowledge with Graphs (Spotify)
Freestyle Fridays - Wartime Data Teams (Spotify)
Tim Berglund - The Art of Developer Relations, Hardware Hacking, and More (Spotify)
Freestyle Fridays - Coding is Fun (Again) (Spotify)
Juhani Vanhatapio - From the Arctic Circle to AI (Spotify)
Freestyle Fridays - Shifting Left AND Right. The Data Engineering Lifecycle in 2025. - (Spotify)
Mark Freeman - Shifting Left in Data, Startup Rocket Ships, and More (Spotify)
Vaibhav Gupta - BAML and AI-First Tools (Spotify)
Freestyle Fridays - Figuring Out Your Next Move (Spotify)
There are way more episodes over at the Joe Reis Show, available on Spotify, Apple Podcasts, or wherever you get your podcasts. Also available on YouTube.
Thanks! If you want to support this newsletter
The Data Engineering Professional Certificate is one of the most popular courses on Coursera! Learn practical data engineering with lots of challenging hands-on examples. Shoutout to the fantastic people at Deeplearning.ai and AWS, who helped make this a reality over the last year. Enroll here.
Practical Data Modeling. Great discussions about data modeling with data practitioners. This is also where early drafts of my new data modeling book will be published.
Fundamentals of Data Engineering by Matt Housley and I, available at Amazon, O’Reilly, and wherever you get your books.
The Data Therapy Session calendar is posted here. It’s an incredible group where you can share your experiences with data - good and bad - in a judgment-free place with other data professionals. If you’re interested in regularly attending, add it to your calendar.
My other show is The Joe Reis Show (Spotify and wherever you get your podcasts). I interview guests on it, and it’s unscripted, always fun, and free of shilling.
Would you like me to speak at your event? Please submit a speaking request if you want me to give a workshop or talk at your event.
If you’d like to sponsor my newsletter or podcast, please get in touch with me.
Could you be sure to leave a lovely review if you like the content?
Thanks!
Joe Reis
Similar issues are creeping into the professional engineering sector with regard to designing physical infrastructure. I agree in general with your view, those that can augment their thinking with AI tools can reach further than those that are simply regurgitating AI outputs.
It will be interesting to see how regulators adopt to licensed professions and how they use AI.