A long design phase without shipping kills many software projects. Here's a story from production I haven't shared before.
This happened early 2023 altho it feels like a lifetime ago. The story is part of the book I'm writing. Figured that's better than trying to conjure something out of reference material.
https://x.com/Swizec/status/1831352994539188449
A big design
At Tia we once faced a big scary project, the biggest we'd ever done. It was going to take multiple teams, represent a huge investment, and a massive challenge for everyone involved. Leadership said the project's goals were critical to company success.
The project felt too important for our typical chaotic approach so we started planning. Meetings were had, requirements collected, consensus built, detailed plans planned. But not much scope negotiation or alternatives proposed 🤔
6 weeks later we had a plan: The project would take 2 quarters (nevermind the half quarter we spent planning), two teams working full-time, and after many tough discussions we managed to negotiate 1 huge release down to 3 milestones.
The project touched clinical data in a way we couldn't feature flag without causing confusion for doctors. Reading stale data because they clicked the wrong button could be bad. Milestones were our compromise to cram at least some iteration into the project.
Our plan was to migrate data section by section. That way portions of it change for all doctors and patients together, but we don't have to launch everything all at once.
One team would build the server-side and set up new relational data structures based on the FHIR standard, an international health data standard the industry is moving towards. If you're curious, you can see your own data in this format using your phone's native Health app. At least on Apple.
Another team (mine) would build the client-side and create a new system for building repetitive UIs that intake FHIR data. We'd create a bunch of atoms as new data types show up and use them to build forms faster and faster. React is great at that.
The first iteration
Our first milestone baaaarely launched at the end of that quarter. Right on schedule but way behind what the business had appetite for.
Business folk wanted the project done in 1 quarter, begrudgingly accepted 2, and didn't care that even deciding what we're building took half a quarter. That's our problem. Business always wants to have finished a project more than it wants to do a project.
This is the first big challenge with BUFD – big upfront design. Business appetites change fast and the suits like to over-estimate their commitment to a plan or strategy. It's hard when your investors expect quarterly results.
And our release was a disaster [name|].
The carefully created plan was too detailed and constraining in parts that didn't matter and not detailed enough in parts that did. At the last minute we found that our code barely fits together.
Contrary to expectation, we all had a different understanding of the plan. Integrating our work took a week, if not more. A huge pain nobody planned for. We even got dangerously close to building a translation layer, which sometimes happens when you can't get teams to work together.
Once the milestone hit production, we had bugs. So many bugs. Bugs because we couldn't integrate sooner. Bugs because we misunderstood the plan. Bugs because we missed requirements. Bugs because we didn't understand the data model. Bugs because the data model couldn't model what the business wanted. Bugs because the business didn't know what they wanted. Bugs because even our feature flags caused issues.
Naturally these 2 weeks of tough bug fixing were not part of the plan. The plan was supposed to have prevented this.
That's the other big challenge with BUFD – plans always work great in theory. But reality doesn't care.
Aaaaand we're behind
Seeing our progress, the business changed its mind and swapped priorities. The critical thing was no longer critical and our secondary project goals became more important.
This tends to happen as new information comes in. You realize that given the cost and complexity of what you're doing, it's not worth doing after all. Small fires and all that.
The important part is to always keep your software in a working state so you can push to prod, abandon course, and shelve a project for later. The longer you keep code un-executed, the worse it gets.
Apply lessons to the next iteration
We abandoned our plan and applied lessons learned to the next 2 milestones. The change in priorities meant our plan wasn't much good anyway.
First we re-assessed our approach to the data. Digging through the FHIR spec we found an easier way to hold it – like a document store. This simplified our backend as dozens of intricately connected tables turned into a flat table with a JSON field.
This helped us simplify the client. Every data save became a single JSON blob upsert instead of a pile of carefully coordinated RESTful API calls. This meant we could build faster and with more confidence because the client-server integration work became repetitive.
If you buy me a beer, I can go on a long rant about why that simpler design wasn't our original plan.
The next 2 milestones juuust managed to ship by the end of the quarter. Note that's double the milestones that we shipped in the first quarter. But we had to abandon every single nice-to-have from the original plan.
The final experience was janky for users, frustrating for us to build and use, difficult to rollout, and one of the most personally unsatisfying projects I've ever worked on. Nobody felt proud of that work.
But we shipped. Delivered value. Learned our lessons. Despite all the big planning, iteration saved the day.
And we learned that you can't throw code over the fence. Teams have to work together.
What about "You do have time to build it twice"
We deferred a lot in that project. Put it on the backlog only to be forgotten forever.
But it wasn't!
A year later we got back to migrating our data over to FHIR. Except this time the industry had caught up. There was a new SaaS vendor that offered a hosted FHIR store solution. Instead of doing lots of cumbersome work, we could swipe a credit card.
It was beautiful. Worked like a charm.
Our first iteration on that project was super small (we learned our lesson) and we shipped to a small subset of our users (another lesson) 2 weeks ahead of schedule 💪
Cheers,
~Swizec
Continue reading about How big up-front design fails
Semantically similar articles hand-picked by GPT-4
- Clever technical hackery can't solve the wrong design
- Coding is the easy part
- Validate your assumptions early
- Why great engineers hack The Process
- A better roadmap solves many issues
Learned something new?
Read more Software Engineering Lessons from Production
I write articles with real insight into the career and skills of a modern software engineer. "Raw and honest from the heart!" as one reader described them. Fueled by lessons learned over 20 years of building production code for side-projects, small businesses, and hyper growth startups. Both successful and not.
Subscribe below 👇
Software Engineering Lessons from Production
Join Swizec's Newsletter and get insightful emails 💌 on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.
"Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️