Playing Not to Lose
Joe's Nerdy Rants #53 - Why Data Teams need to move from defense to offense, plus weekend reads and other stuff
Last week, I posted my Field Notes newsletter. Among many other ideas in that post, these four sentences somehow grabbed much attention. Here’s one example.
Data’s still a mess. Most data initiatives fail. Data teams are seen as a cost center and not getting the support they deserve. Same as it ever was.
There’s not too much in that statement that should be sensational. But after I pondered the reactions, I wanted to understand the underlying factors.
I think a root cause of my statement is that data teams mostly play not to lose. In a conversation yesterday with Jason Hare and Winfried Etzel, Jason brought up a key point I hadn’t considered. He said in most companies, there’s no incentive to add value. It’s easier to pass the buck to the people downstream from you. You play defense and make the next person pay.
This manifests itself in data in quite a few ways. A familiar one is when data teams get data from software developers, who are notorious for lobbing “data” over to data teams. In this case, there’s no incentive for the upstream data producers to make the data useful for analytics. So, the data’s often a mess, and the data team has to patch and duct tape the data together so it’s useful for analytics. Then, the data team creates dashboards for people who probably don’t even look at them. Rinse and repeat every day.
This is a common situation for data teams, and it’s an example of playing not to lose. The data team is just going through the motions, receiving messy data, doing “data stuff” like making dashboards and staying tucked away from the business. Their budget is stagnant because the data team is seen as a cost center.
Start playing to win. This is easier said than done. What does that look like? It means integrating with the business and tying data to critical initiatives. This is surprisingly foreign to quite a few practitioners I talk with. They’d rather passively provide reports and dashboards. Then, they wonder why the business does not recognize their data team’s work. Or they’re confused when the data team’s headcount is reduced. It all comes down to stopping the buck-passing and becoming actively involved in the business. Again, it's easy for me to sit in my fancy Aeron chair and type this motivational message to you. The reality is most of you won’t do this. It’s hard. And I’ll probably write the same thing in five years. Same as it ever was.
If you haven’t checked it out, the Data Engineering Professional Certificate is available on Coursera! Learn practical data engineering with lots of challenging hands-on examples. Shoutout to the fantastic people at Deeplearning.ai and AWS, who helped make this a reality over the last year. Enroll here.
On another note, the popular Data Therapy Session calendar is posted here. It’s an incredible group where you can share your experiences with data - good and bad - in a judgment-free place with other data professionals. If you’re interested in regularly attending, add it to your calendar.
Finally, I’m traveling and speaking a lot. My schedule is at the end of this newsletter. I hope to see you somewhere in the world!
Hope you have a fun weekend!
Thanks,
Joe
P.S. If you haven’t done so, please sign up for Practical Data Modeling. There are lots of great discussions on data modeling, and I’ll also be releasing early drafts of chapters for my new data modeling book here. Thanks!
Cool Weekend Reads
The Meta Grid is nuclear architecture (Ole Olesen-Bagneux)
DJs are debating whether AI can replace them (Semafor)
From bustling hub to ghost town: heart of China’s e-commerce sector feels price war pain (SCMP)
AI Neocloud Playbook and Anatomy (Semi Analysis)
We’re Entering Uncharted Territory for Math (The Atlantic)
Making Uber’s ExperimentEvaluation Engine 100x Faster (Uber Blog)
Slow Deployment Causes Meetings (Kent Beck)
Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?
Practices of Reliable Software Design (Entropic Thoughts)
I’m Running Out of Ways to Explain How Bad This Is (The Atlantic)
New Show & Upcoming Events
The Joe Reis Show
5 Minute Friday - Playing Not to Lose (Spotify)
Navnit Shukla - Data Wrangling and Architecting Solutions on AWS, Writing Books, and More (Spotify)
5 Minute Friday - Notes from the Field, Early Fall 2024 Edition (Spotify)
Ilya Reznik - How to Lead New and Existing ML Teams and More (Spotify)
Jordan Morrow - How to Write Amazing Books (Spotify)
Venkat Subramaniam - Moving Beyond Agile as a Buzzword, Learning to do Less, and more (Spotify)
Paco Nathan - Hacker Culture, Cyberpunk, AI, and More (Spotify)
Bethany Lyons - Disrupting the Recruitment Industry, Startups, and the Future of Work (Spotify)
5 Minute Friday - How Good Do You Need To Be? (Spotify)
Jordan Tigani - Why Small Data is Awesome, DuckDB, and More (Spotify)
Demetrios Brinkmann - AI Hype vs Reality, Building a Global Community, and More (Spotify)
5 Minute Friday - Zero-Sum vs Positive Sum Games (Spotify)
Bill Inmon - History Lessons of the Data Industry. This is a real treat and a very rare conversation with the godfather himself (Spotify) - PINNED HERE.
Monday Morning Data Chat
Matthew Mullins - (Spotify, YouTube)
Ricky Thomas and Paul Dudley - (Spotify, YouTube)
Andrew Ng - Why Data Engineering is Critical to Data-Centric AI (Spotify, YouTube)
Tevje Olin - What Should Data Engineers Focus On? (Spotify, YouTube)
Rob Harmon - Small Data, Efficiency, and Data Modeling (Spotify, YouTube)
Joe Reis & Matt Housley - The Return of the Show! (Spotify, YouTube)
Nick Schrock & Wes McKinney - Composable Data Stacks and more (Spotify, YouTube)
Zhamak Dehghani + Summer Break Special (Spotify, YouTube)
Chris Tabb - Platform Gravity (YouTube)
Ghalib Suleiman - The Zero-Interest Hangover in Data and AI (Spotify, YouTube)
Events I’m At
This event ^^^ is going to be great. Matthew Scullion (CEO of Matillion), Mark Balkenende (VP of Product Marketing at Matillion), and I are chatting about the post-modern data stack. In a chat earlier this week with these guys, I realized that should’ve just been a podcast! It was that good. This discussion will be excellent, so please register!
Matillion Deep Dish Data (virtual event) - October 23. Register here
Other events…
Helsinki Data Week - Helsinki, Finland. October 28 - November 1. Register here
Austin - TBA. November 7-8.
NYC - Data Galaxy Event. November 13.
Amsterdam - TBA. November 21.
Forward Data Conference - Paris, France. November 25. Register here
AWS ReInvent - Las Vegas. Early December. Doing the after-conference scene. Let’s meet up.
Seoul, Korea - TBA. Mid December.
CES - Las Vegas. Early January 2025.
Data Day Texas - Austin, TX. January 25, 2025. Register here
Data Modeling Zone - Arizona. March 4, 2025. Register here
Winter Data Conference - Austria. March 7, 2025. Register here
Netherlands - TBA. April 2025
Much more to be announced soon…
Thanks! If you want to help out…
Thanks for supporting my content. If you aren’t a subscriber, please consider subscribing to this Substack.
Would you like me to speak at your event? Submit a speaking request here.
You can also find me here:
Monday Morning Data Chat (YouTube / Spotify and wherever you get your podcasts). Matt Housely and I interview the top people in the field. Live and unscripted.
My other show is The Joe Reis Show (Spotify and wherever you get your podcasts). I interview guests on it, and it’s unscripted and free of shilling.
Practical Data Modeling. Great discussions about data modeling with data practitioners. This is also where early drafts of my new data modeling book will be published.
Fundamentals of Data Engineering by Matt Housley and I, available at Amazon, O’Reilly, and wherever you get your books.
Be sure to leave a lovely review if you like the content.
Thanks!
Joe Reis
Dude....I literally wrote a post about this exact topic. It's like we're one dysfunctional data family https://thewongway.co/are-you-playing-to-win/
okay I will byte. ;)
And you may find yourself living in a data swamp
And you may find yourself in another part of the cloud
And you may ask yourself, "How do I work this?"
And you may ask yourself, "Where is that large conceptual model?"
And you may tell yourself, "This is not my beautiful data warehouse"
And you may tell yourself, "This is not my beautiful data model"
Same as it ever was, same as it ever was
Same as it ever was, same as it ever was
LOL.
As you mentioned, my feeling is it is part practitioner knowledge that is really not taught well, but really learned by experience of several roles in an application and data environment. The forethought of a good foundation and getting a head start on down stream BI and Analytical needs. The enterprise data architecture forethought and planning of where the data fits in the conceptual model, and how it will be combined and harmonized in a data platform. Every project needs that data planning up front, understanding reporting and analytical needs upfront. In many companies there is push back about talking data needs, saying well, we will get to reporting and data needs in phase 2, or phase 3. I have found too many times, the quality improves and defects decrease when the business analyzes reporting data up front.
PS - Joe, what was the book you mentioned? - "How Big Things Fail", I could not find that one, I did see a well liked book called "How Big Things Get Done" that looks interesting.