Swizec Teller - a geek with a hatswizec.com

Senior Mindset Book

Get promoted, earn a bigger salary, work for top companies

Senior Engineer Mindset cover
Learn more

    Clever technical hackery can't solve the wrong design

    Do you ever get stuck on a project that the more you look at it, the harder it gets? We faced that the other week and with 1 tweak to requirements it became easy, almost trivial. Here's the story.

    The project

    We're building a new electronic health record, a way for patients and providers to collaborate on medical data in a standards compliant way that can integrate with other health systems. It is the gnarliest domain modeling project I've seen.

    How do you store data such that it is flexible enough to model complex reality, but structured enough to be analyzable? The FHIR standard.

    FHIR architecture
    FHIR architecture

    We decided implementing full FHIR is too hard, but took lots of inspiration. Lean hard into the idea of resources and actors, have a proper REST API, and use a gaggle of database tables to store structured data. Great.

    The problems explode in our face

    After months of planning and implementing backend services, the time came for my team to build our first patient-facing UI. A way to enter your past and current conditions.

    Together with design and product, we envisioned a UI like this:

    UI mockup for a page
    UI mockup for a page

    You have a page with questions. Each question can store multiple entries. Then you hit save.

    Can you spot the problem?

    Yeah we thought it would be easy, too. It's a form, how hard can it be, we've built thousands ...


    Validations were first to blow up.

    You have to validate the <thing, year> pair before adding to the list of values for each question. No future years, for example.

    Then you have to validate the form as a whole before enabling the big fat Submit button. Are all questions answered? Do they have conflicts?

    We re-learned that you can't use nested forms. We failed to use field arrays. We then created a monster piece of logic that re-implemented half of react-hook-form.

    It almost worked.

    Distributed transactions

    Next we faced a distributed transactions problem. How do you ensure everything does or doesn't save together?

    Each entry for a question is considered a resource. The backend expects an API call to save or update each.

    User adds 5 conditions, 3 of them save, 2 fail because of a network error. How does the user resolve this? How do we even tell the user what happened?

    You could retry failures for a while, but there's no guarantee you'll succeed. And you can't clean up the 3 that saved either. Hard to send DELETE requests, if you can't talk to the server 💩

    I set a meeting with the backend team and our architect and after a collective set of groans we agreed: Maybe we can build bulk APIs. Let the UI save multiple resources at once.

    That solves atomicity per question.

    What about atomicity per page? Each page may have questions from different resources. You don't want a single bulk API to save a few active conditions, a couple genetic mutations, and perhaps an allergy or two.

    You could build a backend-for-frontend as an intermediary 🤔 but you're just smearing the problem around ...

    The change in perspective

    The fundamental issue is that we had the wrong design. The UI and API disagree on what counts as 1 atomic write.

    You can hide that with clever technical hackery, but you can't solve it.

    I wrote up my findings in a long Slack message that should've been a meeting and asked our product owner what we can do to meet in the middle with API constraints. We can do bulk APIs per resource, but not per page.

    "What about save as you go?", our manager came in with a celeb shot.

    Brilliant! 💡

    Product owner loved it, design said okay after we explained. Saving every answer as soon as it's added solved all our problems 😍

    Input fields, click add, save to backend. 1 atomic operation.

    Validation runs on change and enables the Add button. Hit add, show loading state in button. If it works, clear fields and add to list. If it doesn't, keep fields and show an error same as you would for validations.

    By the time users get through a page everything is saved. The big fat submit button becomes purely navigational. Heck yeah!

    After 2 days of struggling with the old approach, we built the new way in 2 hours. But we needed the struggle to think of it.

    The lesson

    What I like about this story is that it shows the leverage you can have as a senior+ engineer. By talking to other teams and involving the product owner you make a whole class of technical problems vanish and save time on related future stories.

    Plus the UX is better and users are happy. 🤘

    And remember: If everything's a struggle, your approach is wrong.


    Published on March 7th, 2023 in Lessons, REST API, architecture, data modeling, UX

    Did you enjoy this article?

    Continue reading about Clever technical hackery can't solve the wrong design

    Semantically similar articles hand-picked by GPT-4

    Senior Mindset Book

    Get promoted, earn a bigger salary, work for top companies

    Learn more

    Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.

    Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.

    Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.

    Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev

    Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization

    Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections

    Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog

    Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️

    Created by Swizec with ❤️