Building a Global Data Engineering Community

Joe's Nerdy Rants #19 - Weekend reads and other stuff

Not much new to say this week. I’m in the middle of traveling in India and the Middle East, and the world’s turned into a very different place from last week. People have things to process, so I’ll keep it pretty light. The big thing on my mind right now is how data engineers can come together to build a strong global community.

Thanks,

Joe Reis

P.S. Want me to speak at your event in the first half of 2024? Please reach out to me by 11/6/2023. No matter who you are, I won’t be accepting talk or event inquiries after this date. Thanks.

Building a Global Data Engineering Community

I’ve probably been to most of the significant data engineering conferences and events this year, and I’m amazed at the enthusiasm of each community. That said, things seem very fractured. I keep hearing the same pain points from practitioners around the world. Here are some questions I often encounter.

“What stack or tools should I use to address this problem?”

“How should I model my data?”

“How can I level up my skills?”

“How do you think I can get the business to understand what our team does?”

These are just a few themes I get asked about. Whenever someone asks a question like this, I wonder why this person can’t access a community of other practitioners or a body of knowledge to get a quick answer. It seems like every company embarks on data engineering on its own, and it’s a cold-start problem every single time. My big question is, if data engineering is to grow as a profession, why don’t we have a way to help standardize our practices?

Case in point - I traveled this week to speak in Bangalore, India. Before I start, I’d like to give a shout out to Ananth Packkaldurai and Aswin James Christy Nayagam for the excellent DEWcon data engineering conference in Bangalore! I believe this is the first proper community-led data engineering in Bangalore, which is crazy considering how many engineers there are in the city. By most accounts, there are more tech engineers in Bangalore than anywhere else on the planet. This is the “manufacturing hub” of the software world. Data engineers at DEWcon seemed beyond excited to meet fellow data engineers and swap stories. I hope this enthusiasm continues and grows into something much bigger.

I’ve traveled the globe, and I have a good sense of where data engineering is on a macro level. We’ve got a lot of work to do. For data engineering to mature as a profession, we’ll need to start thinking globally.

Listen to the audio clip above on this topic, which is also my 5-Minute Friday on Spotify.

---

Cool Weekend Reads

Here are some cool things I read this week. Enjoy!

Tech, AI & Data

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning (Anthropic)

The challenge with neural nets is interpretability. This is a good - and dense (no pun intended) - read about some research that might lead to solving this problem.

Why AI is teetering on the edge of a disillusionment cliff (Venturebeat)

“The bottom line is that AI may have incredible positive potential for humanity’s future, but I don’t think companies are doing a great job of communicating what that is. Where is the “why” — as in, why are we going through all the angst of building all of this? What is the current and future value of generative AI to individuals, workers, enterprises, and society at large? How do the benefits outweigh the risks?”

“We’ll call it AI to Sell it, Machine Learning to Build it” (The AI Underwriter)

Same as it ever was…back in the day, it was selling it as ML, with linear regression under the hood. Now, it’s calling it AI, with deep learning under the hood.

Business & Startups

‘Start-up Nation’ goes to war (FT)

The consequences of the war on the startup ecosystem will be very interesting (and tragic) to see.

The perfect pricing model doesn't exist (Wraptext)

“Make peace with the fact that your pricing will never be perfect.”

Unbundling AI (Ben Evans)

Really good analysis from Ben Evans on the “science” and “product” problems of AI.

How ChatGPT and other AI tools could disrupt scientific publishing (Nature)

As with books and articles, I feel like LLMs are a blessing and the worst curse imaginable. On one hand, it’s easier than ever to create content (I actually refrain from using LLMs for my writing, except for ideation). On the other hand, the world of content is about to be flooded with a mess beyond imagination.

---

New Content, Events, and Upcoming Stuff

Monday Morning Data Chat

Coming up…

No Monday Morning Data Chat on 10/16. We’ll resume the show on 10/23 with our favorite nerd, Kevin Hu, from Metaplane.

In case you missed it…

Data Warehouses and Semantics Deep Dive, SDF, and more w/ Lukas Schulte - (Spotify, YouTube)

Improving Your Health and Wellness - Techie Edition w/ Colleen Fotsch (Spotify, YouTube)

Data Engineering AMA w/ Matt Housley & Joe Reis (Spotify, Youtube)

The Joe Reis Show

5 Minute Friday - Building a Global Data Engineering Community (Spotify)

Bob Muglia - Moving Beyond SQL, Knowledge Graphs, and More (Spotify)

In case you missed it…

5 Minute Friday - Skills to Pay the Bills (Spotify)

Egor Gryaznov - The "Non-Modern Data Stack", and Getting Out of Our Data Bubble (Spotify)

Jason Taylor - The Divides in the Data Space, Fighting Dumpster Fires, and More (Spotify)

Michel Tricot - The Impact of AI on the Modern Data Stack (Spotify)

Events

October

Dubai - 10/16-10/19. GITEX - register here

November

Canada - DAMA Toronto - register here

Finland - Agile Data Engine Summit - register here

San Jose, CA - TBA

Las Vegas - ReInvent

2024

Data Day Texas (Austin) - register here

Data Modeling Zone (Arizona) - register here

Skiers in Data (Switzerland) - TBA

Spain - TBA

Want me to speak at your event in the first half of 2024? Get your inquiries to me by 11/6/2023. I won’t be accepting talk or event inquiries after this date. Thanks.

Thanks! If you want to help out…

Thanks for supporting my content. If you aren’t a subscriber, please consider subscribing to this Substack.

You can also find me here:

Monday Morning Data Chat (YouTube / Spotify and wherever you get your podcasts). Matt Housely and I interview the top people in the field. Live and unscripted. Zero shilling tolerated.

The Joe Reis Show (Spotify and wherever you get your podcasts). My other show. I interview guests, and it’s totally unscripted with no shilling.

Fundamentals of Data Engineering (Amazon, O’Reilly, and wherever you get your books)

Be sure to leave a nice review if you like the content.

Thanks! - Joe Reis