The Dev and Data Divide, Redux

Joe's Nerdy Rants #13 - Weekend reads and other stuff

The Dev and Data Divide, Redux

The cool thing about what I do these days is traveling and talking with people about their experiences. Given the variety and sheer number of people I meet, I’d like to think I’ve got a global perspective on the state of data. No matter where I am in the world, recurring themes keep popping up. I hate rehashing topics, like the dev and data divide, but this week was a good reminder that the divide is all too real. Data teams often depends upon dev for their data. When dev holds all the power in the relationship, data teams can often feel rightfully helpless.

On Tuesday, I did an impromptu talk/town hall at the Utah Engineering Leadership Meetup with a group of devs and engineering managers/directors/VPs/Cs. The big takeaway was that even if data teams depend on dev for their data, the devs were largely unaware of the needs of data people, and frankly didn’t care. One dev put it perfectly - he’d probably help the data people at his company if 1) the data people were crystal clear on what they wanted from him and 2) he had the incentive and bandwidth to help. But since he had no time or incentive to help anyone but his immediate team, the requests from the data team were ignored. As a result, the data team worked around him, and I’m unsure what happened after that. Nor I don’t get the sense this dev cared.

Thursday night in Atlanta, I spoke at my roadshow with dbt Labs, this time to a group of data engineers. When I showed them the above image, there were unanimous agonized groans from people immediately getting hit with the pain of dealing with dev teams who don’t care about them. The conversation moved toward better ways dev and data can work together. Communication and collaboration were big solutions that kept coming up over and over. I hope at some of the people in the audience take steps to start communicating with the dev teams.

As I wrote earlier this summer on this same topic, “The world has moved on from data teams providing basic reporting. It did that a while ago. Beyond simple CRUD transactional use cases, data is becoming integrated into applications. The ubiquity of ML/AI, streaming data, reverse ETL (I wish this term would disappear), and other feedback loops back to the application mean data isn’t something that lives outside the application. Data is the application. Other forces like data products, data mesh, and data-centric models will nudge this along too. Inevitably, the software and data divide will disappear.”

I still think this divide will shrink over time. To move this along (and assuming the dev team is a dependency), I urge data teams to build a bridge with the dev teams they depend upon. It doesn’t need to be fancy. Do lunch and learns with the dev team. Educate them on what you do, and why it matters to the company. Simple things like this will build empathy, and a little empathy goes a long ways.

Listen to the audio clip above on this topic, which is also my 5 Minute Friday on Spotify.

---

Cool Weekend Reads

Hope you all had a great week.

Here are some cool things I read this week…

Also…I’m holding off on any posts on superconductors yet, as it’s unclear if this is the real thing or total BS (which has happened a lot in the past with this sort of thing). But if superconductors are the new AI or crypto, cool ;)

Tech, AI & Data

Squeeze the hell out of the system you have (Dan Slimmon)

“When complexity leaps are on the table, there’s usually also an opportunity to squeeze some extra juice out of the system you have. By tweaking the workload, tuning performance, or supplementing the system in some way, you may be able to add months or even years of runway. When viable, these options are always preferable to building out a next-gen system.”

This is refreshing article that sometimes the best option isn’t to chase shiny new tools, but better leverage the ones you have.

A good thought on requirements (Dax)

Google launches Project IDX, a new AI-enabled browser-based development environment (Techcrunch)

In typical Google fashion, Google enters the AI-enabled IDE game.

The Last 1% (Jared Ramsey )

The last 1% of stuff in releasing a feature is often the most important stuff.

Business & Startups

I Would Rather See My Books Get Pirated Than This (Or: Why Goodreads and
Amazon Are Becoming Dumpster Fires) (Jane Friedman)

As an author, this article resonates. Fun short story - a week before Fundamentals of Data Engineering was released, a summary of our book appeared on Amazon. The summary’s copy had very little to do with our book, or data engineering in general, and it looked like a bot wrote it. Upon further inspection, the publisher also had these odd summaries of other books from authors like Alex Jones and similar. F*cking weird! Since then, we seen other books with identical titles to ours on Amazon, and Amazon won’t do a damn thing about it. Amazon is a necessary evil and total dumpster fire for authors.

Should I change job? (The Engineering Manager)

“Are you learning? Are you earning? And if you’re not, what are you going to do about it?”

I don’t know what’s in the air these days, but a lot of people I meet are unhappy at work and want to change jobs. This is a good article if you’re in that boat right now.

AI Bubble Bursting into AI Winter – yes or no? (Turing Post)

Really good read about…whatever the hell you call AI right now. Bubble? Inflection point? Unstoppable? You decide.

Hopin, once valued at $7.6 billion, makes some major changes (Axios)

During COVID (remember that?) Hopin was all the rage. It was EVERYWHERE. Now, it’s sold for scrap. Goes to show just how quickly valuations can change on a dime.

---

New Content, Events, and Upcoming Stuff

Monday Morning Data Chat

Coming up…

Streaming Data Processing Deep Dive w/ David Yaffe and Johnny Graettinger (co-founders of Estuary) LinkedIn Live, YouTube

In case you missed it…

The Rise and Importance of Business Language w/ John O'Gorman (Spotify, YouTube)

Why Apache Iceberg Won the Table Format War w/ Brian Olsen (Spotify, YouTube)

Why Your BI Team is Your Best Bet for Data Science w/ Dave Langer (Spotify and YouTube)

The Joe Reis Show

Coming up…

Kevin Hu, Gordon Wong, and many more….

In case you missed it…

5 Minute Friday - The Dev and Data Divide (Spotify)

Vin Vashishta - From Data to Profit, Writing a Book, and more (Spotify)

Scott Taylor - Scott Taylor - Being a Storyteller, Speaker, Creator, and Influencer in the Data Space (Spotify)

Ryan Boyd - Small Databases are Motherducking Awesome! (Spotify)

Kai Zenner - The Evolution, Challenges, and Potential of the EU AI Act (Spotify)

Events

August

US - Utah Data Engineering Meetup (8/16) - me + Mage.ai! register here

Australia - DataEngBytes - I’ll be on the continental tour in Perth, Brisbane, Melbourne, and Sydney for a couple of weeks. August 2023 more info and registration

September

Joe Reis + dbt roadshow - Bellevue, WA (9/7) - register here!

Big Data London- 9/20

Europe - TBA

October

India - 10/12. Details TBA

Dubai - 10/16-10/19. Details TBA

Chicago - 10/26 - Details TBA

November

Canada - DAMA Toronto. Details TBA

Las Vegas - ReInvent - got a massive special announcement in store :)

2024 - lots of stuff. Stay tuned :)

Thanks! If you mind helping out…

Thanks for supporting my content. If you aren’t a subscriber, please consider subscribing to this Substack.

You can also find me here:

Monday Morning Data Chat (YouTube / Spotify and wherever you get your podcasts). Matt Housely and I interview the top people in the field. Live and unscripted. Zero shilling tolerated.

The Joe Reis Show (Spotify and wherever you get your podcasts). My other show. I interview guests, and it’s totally unscripted with no shilling.

Fundamentals of Data Engineering (Amazon, O’Reilly, and wherever you get your books)

Be sure to leave a nice review if you like the content.

Thanks! - Joe Reis