The Politics of Data
On a hike with a good friend yesterday, we chatted about the politics of data. Companies and people say they’re “data-driven.” This is a nice platitude, and often furthest from the politics that actually drive organizational behavior. The politics of data often takes the form of companies and people saying they are “data-driven”, while in reality using data to push a pre-determined agenda.
This is something you might have experienced in your own career and observations. I’ve seen this countless times. Initiatives are pursued because it looks good for some executive’s resumes, or because “the board” wants to pursue something like “AI” (whatever the hell that is). Data is used and sometimes massaged to justify these initiatives.
The impact on software and data teams can be massive. If you’re an individual contributor or in a non-decision-making role, know that you’re often not in control of your destiny. The real decisions in your organization are likely made in a waterfall way. The ironic thing is while you can practice Agile all you want, your boss’s boss already decided what she wants to do, your boss decided what he wants to do, and so on. There are different motivations, but you must know that technical excellence and “data” are not what drive most people. What drives most people are ego-driven - climbing the career ladder, outshining others, and money.
Politics has an inescapable gravity on organizations, and the sooner you can learn to get a good sense of the political dynamics, the better. Know the reasons behind the reasons and the questions behind the questions. Understand the Golden Rule of Value. Above all, with the number of layoffs happening in tech, stay sharp and try to get ahead of any political things that might impact you.
Listen to the audio clip above on this topic, which is also my 5 Minute Friday on Spotify.
Cool Weekend Reads
Hope you all had a great week.
Here are some cool things I read this week…
Tech, AI & Data
Building and operating a pretty big storage system called S3 (All Things
Distributed)
If you read one technical article this week, let it be this one. S3 is one of the true wonders of the universe. Crazy that S3 is almost old enough to be an adult in the US.
AI and the automation of work (Benedict Evans)
The 2023 version of when John Maynard Keynes thought we’d all be working 15-hour weeks by now.
Cleaning Up ChatGPT Takes Heavy Toll on Human Workers (WSJ)
“My experience in those four months was the worst experience I’ve ever had in working in a company,” Alex Kairu, one of the Kenya workers, said in an interview.
There’s the AI ethics we typically talk about, then there’s the other AI ethics - the impacts on the “blood diamond diggers” of the AI economy (and also the social media moderation economy) - who deal with the horror show of the Internet. Reading this article made me physically upset.
ChatGPT Out-scores Medical Students on Complex Clinical Care Exam Questions (Stanford)
I’ve been chatting with quite a few professors about how education changes now that large language models are out of the bag. Working alongside machines will be a new reality for students and schools. I think LLMs and “AI” invert the entire way we teach and learn, and demonstration of mastery will come from truly diagnosing a problem from first principles, not just rote regurgitation of information.
Business & Startups
An epilogue to my time working at Twitter ( Esther Crawford)
Good read on what it’s like to get acquired by Twitter and deal with the rollercoaster/rocketship/shitshow.
Worldcoin isn’t as bad as it sounds: It’s worse (Blockworks)
“Imagine that your digital identity has been lost in some way — shut down by authorities for non-compliance, or otherwise blocked. With traditional cash — and other cryptocurrencies — you can always make a new wallet and stash some fresh coins in it. But this isn’t Minority Report, and you can’t get a new iris from your neighborhood surgeon.
When your immutable digital identity is locked — imagine merchants who won’t take your coins from you without a digital signature announcing your World ID — it’s over for you. No old account. No new account. No soup for you. You just lost your digital personhood.”
I’m skeptical enough of using Clear and refuse to use stuff like 23 and me. No damn way I’m using Worldcoin.
Americans Are Moving Toward Climate Danger in Search of Cheaper Homes
Everything about America in 2023 is in that sentence ^^^^
New Content, Events, and Upcoming Stuff
This week
Monday Morning Data Chat - Why Your BI Team is Your Best Bet for Data Science w/ Dave Langer (Spotify and YouTube)
In case you missed it…
Monday Morning Data Chat - Dataframe Deep Dive w/ Devin Petersohn (Spotify, YouTube)
Monday Morning Data Chat - #134 - Should Your Business Chase Generative AI? w/ Andreas Welsch (Spotify, YouTube)
Monday Morning Data Chat - #133 - Intro to Data Contracts w/ Andrew Jones (Spotify, YouTube)
The Joe Reis Show
Ryan Boyd - Small Databases are Motherducking Awesome! (Spotify)
Kai Zenner - The Evolution, Challenges, and Potential of the EU AI Act (Spotify)
In case you missed it…
Benny Benford - Elevating Data to a Profession (Spotify)
Joshua Bowles - A Wide-Ranging Chat on ML and AI (Spotify)
Upcoming
Monday Morning Data Chat - Why Apache Iceberg Won the Table Format War w/ Brian Olsen (LinkedIn, YouTube)
The Joe Reis Show - Lots coming up! Scott Taylor, Vin Vashista, and many more….
Events
Joe Reis + dbt roadshow
Atlanta (8/10) - register here
Seattle (9/7) - details soon
DataEngByes. I’ll be on the continental tour in Perth, Brisbane, Melbourne, and Sydney for a couple of weeks. August 2023 (more info and registration)
Big Data London - I’m keynoting. Big up the London Massive. September 2023.
Europe - September 2023 TBA
India, Dubai - October 2023
Canada - November 2023
Vegas - ReInvent, November 2023
More to come…
Thanks! If you mind helping out…
Thanks for supporting my content. If you aren’t a subscriber, please consider subscribing to this Substack.
You can also find me here:
Monday Morning Data Chat (YouTube / Spotify and wherever you get your podcasts). Matt Housely and I interview the top people in the field. Live and unscripted. Zero shilling tolerated.
The Joe Reis Show (Spotify and wherever you get your podcasts). My other show. I interview guests, and it’s totally unscripted with no shilling.
Fundamentals of Data Engineering (Amazon, O’Reilly, and wherever you get your books)
Be sure to leave a nice review if you like the content.
Thanks! - Joe Reis
“ The politics of data often takes the form of companies and people saying they are “data-driven”, while in reality using data to push a pre-determined agenda.” - oh man do I ever feel this statement lately. It also falls nicely in with the Twitter read. These companies that started as elite engineering orgs devolve over time because they were never built to last, there was no incentive to do things sustainably.
I’d forgotten about the golden rule of value post. This chain of value sometimes has weak links, then you’re managing up trying to get it all in line. I guess an extreme version is someone like Steve Jobs who relentlessly ignored Wall St., if he listened to them, Apple would be selling Blackberry clones. Bezos did it too, 20 years ago there were tons of questions about when Amazon would be profitable and why were they doing th this S3 thing? Sometimes the company has to manage its investors. As a data engineer you’re almost always in a state of nebulous value, you’re supposed to be the feedback part of this flywheel (especially if any kind of recommendations or personalization is important to your business) but if that breaks down it might take a long time to figure that out.