No Extra Credit For Complexity
What is simplicity in data engineering?
You don't get any extra points for the fact that something's very hard to do. - Warren Buffett
In a recent LinkedIn post, I left two short sentences.
Pretty simple. That’s the point.
What is simplicity in data engineering? This is not a straightforward question to answer. Everything you build is different, and simplicity depends on the context in which it’s determined.
Though the question of simplicity isn't straightforward, the answer to simplicity is relatively straightforward. You know simplicity when you see it. Simplicity means designing things with the fewest moving parts, making each part easy to understand and work with. A simple system is elegant and probably brings you joy.
My litmus test for simplicity in data engineering - Is there anything that can be removed without sacrificing your needed form and function? If nothing else can be removed, then it’s probably simple enough for you.
Simple isn’t always perfect, but it’s always good enough.
Simplicity isn’t static. Over time, simplicity evolves to match your new requirements. New requirements are often where complexity introduces itself. Beware of complexity. It’s the killer of simplicity. Sadly, data engineers are often addicted to complexity. It tempts a data engineer to engineer things, sometimes sacrificing simplicity. The data engineer eventually realizes their life is too complicated. Then they seek simplicity. Rinse and repeat.
There’s no extra credit for unnecessary complexity. Always strive for simplicity.
This also applies to nearly every other field except law, government, health care, and related.
To your footnote: Law and government also suffer from unnecessary/counterproductive complexity! (even if other times they're too simplistic and lack needed nuance)
US tax code was the first example that came to mind. "Tax law" isn't exactly what comes to mind when I think of fun and interesting topics, but this book was a surprisingly fun read: https://www.goodreads.com/book/show/32301945-a-fine-mess