Software engineering isn't all about building products. Often it's about automating business processes. Even your own! But there's a trap you can fall into.
How automation goes wrong
One of my favorite XKCDs talks about opportunity cost: how much time can you invest before building the automation takes more time than it saves? Infrequent small tasks are almost never worth it unless you can make them go away completely.
Another favorite talks about the maintenance cost of automation. First you stop doing the original task while you build the automation and then ... you spend all eternity fiddling with and improving your work because it can always be even better. Now you don't have time for the original task at all.
Feynman, a famous physicist, called this the trouble with computers – The trouble with computers is you play with them. You get so sucked into making the computer do the work that you forget there was a bigger goal to reach.
It's okay to just do the work. What's the easiest way to cancel a user account? Click the button you already have!
Once you've built a bulk account cancellation feature, you'll have to keep it working. Any future story in this part of your codebase will have to take the bulk cancellation workflow into account. Especially if bulk cancellation catches on and people start using the feature in a regular workflow as part of a standard operating procedure.
The big danger
Be aware of opportunity cost but don't sweat. When it's big enough to care it will be obvious.
The bigger danger of spurious automation is automating a task you don't yet understand. This goes double when building a mini framework for "a common code pattern". When you automate a process, it ossifies and becomes hard to change.
This is fantastic when you need to scale.
A standard process with strictly defined steps ensures that every burger at a fast food chain tastes the same. Regardless of who's cooking that day. Plus it's faster to make because your chef is not reinventing steps for every order.
That same efficiency gets in the way when you're not sure if your customers want burgers, pasta, or pizza. You'll need a chef that can think on their feet and respond to feedback.
The less structure and automation you have while figuring things out, the faster you'll move and the easier it will be to respond to feedback.
That's why so many of your company's business processes run on random spreadsheets, half forgotten documents, and chat threads. Spreadsheets are easy to change and optimize for every unique situation. Once engineering gets involved, the process solidifies and there's no more give. Operators can't use their judgment to make a tweak.
Like that time a machine rejected my suitcase at the airport for being too heavy. Lady at the counter saw the 600 gram delta and scanned me through without mentioning the weight.
How to automate anything
When automating business processes (as opposed to building product), here's an algorithm you can use. I've found this to be effective in anything from building internal tools to libraries and frameworks for engineering teams to use.
It's a 3 step process and you're encouraged to give up at any stage and say "good enough". Only keep going if the challenge persists and you're getting annoyed. Annoyance is a good signal that it's worth the effort.
- Do the work
- Note your steps
- Write the code
1. Do the work
Try to do the work manually at first. This forces you to figure out every step in detail and make sure you know what's involved.
If the same problem never occurs again, you've saved yourself and the team all the overhead of productization and maintenance. Congratz!
If you know right away that N is too large to do by hand, try to do 1 or 2 cases. The clarity you get from doing the work is well worth the effort.
2. Note your steps
While you're doing the work, it's a good idea to write down the steps you're taking.
For tasks that happen infrequently, this helps you remember what to do in 6 months. Notes make the work go faster and feel less distracting.
Write the steps as a checklist and try to follow the list every time. Checklists have been shown to reduce surgical process mistakes by up to 40 percentage points.
The document this creates is called an SOP – standard operating procedure – or runbook. Once the steps are written down, anyone can run the process because the know-how is not stuck in your head. This frees you up for more interesting work.
With a good SOP even surprisingly frequent manual tasks can be no big deal. You do the steps almost mindlessly when you need a break from hard thinking work.
PS: always attach an SOP to every production alert or warning. We'll talk more about telemetry and monitoring later.
3. Write the code
Update your SOP every time you go through. Encourage others to contribute. If any step changes or lacks detail, add.
When the SOP stops changing and starts to feel annoying because you're doing it so much, turn the document into code. You now know exactly what needs to happen and when. You've also documented or dealt with every exception that comes up.
Feeling annoyed that you have to do something so often is a good indicator both that it's worth automating and that it has become rote enough to be easy to automate.
At Tia, for example, we went years with engineers "manually" (via migrations) managing bookable appointment types in the database. One day I'd had enough and added a new form to our admin tool that let anyone add a new appointment type.
This saved engineering an estimated month worth of work every year and empowered the business operations team to iterate on their own without waiting for engineers. They loved it.
Over time that page evolved into a full CMS and became part of our business processes. Now it gets occasional tweaks and updates as new needs arise. That's the ongoing maintenance/improvement part of the cycle. ✌️
Cheers,
~Swizec
Continue reading about The dangers of spurious automation and how to automate anything
Semantically similar articles hand-picked by GPT-4
- You don't need a big team to automate your work
- Don't neglect your upgrades
- Build better software with The Theory of Constraints
- You can't side-quest a product
- What I learned from Software Engineering at Google
Learned something new?
Read more Software Engineering Lessons from Production
I write articles with real insight into the career and skills of a modern software engineer. "Raw and honest from the heart!" as one reader described them. Fueled by lessons learned over 20 years of building production code for side-projects, small businesses, and hyper growth startups. Both successful and not.
Subscribe below 👇
Software Engineering Lessons from Production
Join Swizec's Newsletter and get insightful emails 💌 on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.
"Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️