No Extra Credit For Complexity

What is simplicity in data engineering?

You don't get any extra points for the fact that something's very hard to do. - Warren Buffett

In a recent LinkedIn post, I left two short sentences.

Pretty simple. That’s the point.

What is simplicity in data engineering

¹

? This is not a straightforward question to answer. Everything you build is different, and simplicity depends on the context in which it’s determined.

Though the question of simplicity isn't straightforward, the answer to simplicity is relatively straightforward. You know simplicity when you see it. Simplicity means designing things with the fewest moving parts, making each part easy to understand and work with. A simple system is elegant and probably brings you joy.

My litmus test for simplicity in data engineering - Is there anything that can be removed without sacrificing your needed form and function? If nothing else can be removed, then it’s probably simple enough for you.

Simple isn’t always perfect, but it’s always good enough.

Simplicity isn’t static. Over time, simplicity evolves to match your new requirements. New requirements are often where complexity introduces itself. Beware of complexity. It’s the killer of simplicity. Sadly, data engineers are often addicted to complexity. It tempts a data engineer to engineer things, sometimes sacrificing simplicity. The data engineer eventually realizes their life is too complicated. Then they seek simplicity. Rinse and repeat.

There’s no extra credit for unnecessary complexity. Always strive for simplicity.

1. This also applies to nearly every other field except law, government, health care, and related.