You know how sometimes you're building a thing and everything keeps going wrong? Edge case after edge case, workaround after workaround. Things are just hard 😩
You may have the wrong abstraction my friend.
I'm reading System Design and the Cost of Architectural Complexity, an MIT PhD thesis from 2013 by Daniel J. Sturtevant. It found empirical proof that engineering decisions matter 🤩. You can read my scribbled-over version at /pdfs/system-design-and-the-cost-of-architectural-complexity-with-swiz-notes.pdf. Right now I'm about a third through.
Why architectural complexity matters
Sturtevant's study found that architectural complexity:
- Decreases productivity by up to 50%
- Increases bug density by 3x
- Causes an order of magnitude increase in staff turnover
That's a huge effect. The author developed an empirical measure of architectural complexity and was able to show its effect on engineers by comparing different files in the same real world project. Working in areas with higher complexity had worse outcomes than poking areas with lower complexity.
What is architectural complexity
Sturtevant compares different academic measures of complexity and concludes that:
- Measuring complexity is hard
- Every engineer knows it when they see it
Academics use engineers as a benchmark to judge whether a metric they develop correlates with experts' subjective feelings. This is in line with other research I've read saying that ethnographic approaches to studying software engineering are the most useful. Seeing what experts do and turning it into theory, not the other way around.
Out of available metrics, Sturtevant concludes that cyclomatic complexity is the best metric to answer "Is this piece of code hard to understand?". The more indents you have in a code, the more complex it is.
But cyclomatic complexity doesn't work for systems.
Modern software isn't hard because the code is hard, it's hard because of emergent behaviors and spooky systemic behaviors. Because no one engineer knows how everything works.
Your code is a network
Instead Sturtevant's research hinges on a metric he calls architectural complexity. This one measures how many connections exist between parts of a system.
Using this approach lets you analyze a codebase as a network graph and identify modules with tight or loose coupling, connections between modules, notice abstractions, choke points, layers, etc. Any big mess immediately stands out.
He points out that senior engineers "see" this structure when working with code. As opposed to more junior team members who rely on reading the code itself.
Interestingly that means good naming of functions, variables, etc is less important to seniors because they don't really read the code. The name is just a symbol.
The impact of a wrong abstraction
Abstractions are key to taming architectural complexity.
You can think of an abstraction as a boundary between tight internal coupling and loose external coupling. A choke point through which unrelated modules talk to a piece of functionality.
A good abstraction hides details and makes them unimportant. You call a function, it does the thing, and you don't care how. The opposite of lasagna or minestrone code.
A bad abstraction ... that feels like work. Like Sturtevant's thesis I'll quote from Daniel Jackson's Software abstractions; logic, language and analysis because it's so good:
Software is built on abstractions. Pick the right ones and programming will flow naturally from design; modules will have small and simple interfaces; and new functionality will more likely fit in without extensive reorganization. Pick the wrong ones, and programming will be a series of nasty surprises: Interfaces will become baroque and clumsy as they are forced to accommodate unanticipated interactions, and even the simplest of changes will be hard to make. No amount of refactoring, bar starting again from scratch, can rescue a system built on flawed concepts.
When good abstractions are missing from the design, or erode as the system evolves, the resulting program grows barnacles of complexity. The user is then forced to master a mass of spurious details, to develop workarounds, and to accept frequent inexplicable failures
An abstraction is a structure, pure and simple – an idea reduced to its essential form.
So, when everything feels hard, you may be doing it wrong. Time to step back and rethink your approach. The sooner the better.
Next article in series: Two types of complexity and their impact
Cheers,
~Swizec
Continue reading about You can't fix the wrong abstraction
Semantically similar articles hand-picked by GPT-4
- Two types of complexity and their impact
- Why taming architectural complexity is paramount
- Forget complicated code, focus on the system
- Finding modules in a Big Ball of Mud with ChatGPT
- Complex vs. complicated
Learned something new?
Read more Software Engineering Lessons from Production
I write articles with real insight into the career and skills of a modern software engineer. "Raw and honest from the heart!" as one reader described them. Fueled by lessons learned over 20 years of building production code for side-projects, small businesses, and hyper growth startups. Both successful and not.
Subscribe below 👇
Software Engineering Lessons from Production
Join Swizec's Newsletter and get insightful emails 💌 on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.
"Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️