Swizec Teller - a geek with a hatswizec.com

Senior Mindset Book

Get promoted, earn a bigger salary, work for top companies

Senior Engineer Mindset cover
Learn more

    You can't fix the wrong abstraction

    You know how sometimes you're building a thing and everything keeps going wrong? Edge case after edge case, workaround after workaround. Things are just hard 😩

    You may have the wrong abstraction my friend.

    I'm reading System Design and the Cost of Architectural Complexity, an MIT PhD thesis from 2013 by Daniel J. Sturtevant. It found empirical proof that engineering decisions matter 🤩. You can read my scribbled-over version at /pdfs/system-design-and-the-cost-of-architectural-complexity-with-swiz-notes.pdf. Right now I'm about a third through.

    Why architectural complexity matters

    Sturtevant's study found that architectural complexity:

    1. Decreases productivity by up to 50%
    2. Increases bug density by 3x
    3. Causes an order of magnitude increase in staff turnover

    That's a huge effect. The author developed an empirical measure of architectural complexity and was able to show its effect on engineers by comparing different files in the same real world project. Working in areas with higher complexity had worse outcomes than poking areas with lower complexity.

    What is architectural complexity

    Sturtevant compares different academic measures of complexity and concludes that:

    1. Measuring complexity is hard
    2. Every engineer knows it when they see it

    Academics use engineers as a benchmark to judge whether a metric they develop correlates with experts' subjective feelings. This is in line with other research I've read saying that ethnographic approaches to studying software engineering are the most useful. Seeing what experts do and turning it into theory, not the other way around.

    Out of available metrics, Sturtevant concludes that cyclomatic complexity is the best metric to answer "Is this piece of code hard to understand?". The more indents you have in a code, the more complex it is.

    But cyclomatic complexity doesn't work for systems.

    Modern software isn't hard because the code is hard, it's hard because of emergent behaviors and spooky systemic behaviors. Because no one engineer knows how everything works.

    Your code is a network

    Instead Sturtevant's research hinges on a metric he calls architectural complexity. This one measures how many connections exist between parts of a system.

    Using this approach lets you analyze a codebase as a network graph and identify modules with tight or loose coupling, connections between modules, notice abstractions, choke points, layers, etc. Any big mess immediately stands out.

    Code as network graph
    Code as network graph

    He points out that senior engineers "see" this structure when working with code. As opposed to more junior team members who rely on reading the code itself.

    Interestingly that means good naming of functions, variables, etc is less important to seniors because they don't really read the code. The name is just a symbol.

    The impact of a wrong abstraction

    Abstractions are key to taming architectural complexity.

    You can think of an abstraction as a boundary between tight internal coupling and loose external coupling. A choke point through which unrelated modules talk to a piece of functionality.

    Abstraction boundary between internal and external coupling
    Abstraction boundary between internal and external coupling

    A good abstraction hides details and makes them unimportant. You call a function, it does the thing, and you don't care how. The opposite of lasagna or minestrone code.

    A bad abstraction ... that feels like work. Like Sturtevant's thesis I'll quote from Daniel Jackson's Software abstractions; logic, language and analysis because it's so good:

    Software is built on abstractions. Pick the right ones and programming will flow naturally from design; modules will have small and simple interfaces; and new functionality will more likely fit in without extensive reorganization. Pick the wrong ones, and programming will be a series of nasty surprises: Interfaces will become baroque and clumsy as they are forced to accommodate unanticipated interactions, and even the simplest of changes will be hard to make. No amount of refactoring, bar starting again from scratch, can rescue a system built on flawed concepts.

    When good abstractions are missing from the design, or erode as the system evolves, the resulting program grows barnacles of complexity. The user is then forced to master a mass of spurious details, to develop workarounds, and to accept frequent inexplicable failures

    An abstraction is a structure, pure and simple – an idea reduced to its essential form.

    So, when everything feels hard, you may be doing it wrong. Time to step back and rethink your approach. The sooner the better.

    Next article in series: Two types of complexity and their impact


    Published on May 11th, 2023 in Software Architecture, Software Engineering, Papers, Complexity

    Did you enjoy this article?

    Continue reading about You can't fix the wrong abstraction

    Semantically similar articles hand-picked by GPT-4

    Senior Mindset Book

    Get promoted, earn a bigger salary, work for top companies

    Learn more

    Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.

    Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.

    Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.

    Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev

    Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization

    Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections

    Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog

    Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️

    Created by Swizec with ❤️