Here's a fun exercise: what's wrong with this code?
// consider this pseudocode
async function runsEveryHour() {
const items = await db.fetchUnprocessedItems()
let successCount = 0
try {
await db.transaction(async trx => {
for (let item of items) {
const result = await doSomethingFancy(item, trx)
if (result) {
successCount += 1
}
}
})
console.log(`Processed ${successCount} items`)
} catch (err) {
console.error("Error processing items", err)
}
}
This is a simplified version of an hourly cronjob we had at work. Wakes up, fetches unprocessed data from the database, starts a new transaction, iterates over the items, and runs doSomethingFancy on every element. Reports progress or error at the end.
Looks great, works great.
Until one day we notice this code hasn't done anything in 5 days. It ran, but nothing happened. 🤨
Consider partial success
The code above is written in an all-or-nothing style. Either you process every item, or none of them.
Using a try/catch ensures clean error handling and db.transaction turns the complex database interactions inside doSomethingFancy into an atomic operation. If a query fails, the database rolls back. As if nothing ever happened.
You want to use this approach for atomic operations. Like when you're doing a specific task for a specific user. Imagine if charging a credit card failed separately from creating the order. You'd charge the user and never send the item 😬
Background processing tasks are different.
In cases like this, you're often fulfilling specific tasks for different users. Or multiple tasks for the same user. They should fail independently.
If 1 item out of 200 fails, should the other 199 suffer? Doubt it.
How to enable partial success
You can overcomplicate this for massive scale with a fan-out approach like I wrote about in the Serverless Handbook chapter on data pipelines. Split the work into small chunks (like 1 item), process each chunk on its own request.
An easier approach when the data is small and time isn't crucial – a background task that takes 10min to run is fine – is to invert the code to allow partial failures.
Like this:
// consider this pseudocode
async function runsEveryHour() {
const items = await db.fetchUnprocessedItems()
let successCount = 0
let errorCount = 0
for (let item of items) {
try {
await db.transaction(async trx => {
const result = await doSomethingFancy(item, trx)
if (result) {
successCount += 1
}
})
} catch (err) {
console.error("Error processing item", item, err)
errorCount += 1
}
}
console.log(`Processed ${successCount} items; got ${errorCount} errors`)
}
We inverted the code to put our loop on the outside.
Each item on its own runs in a database transaction because the internals of each operation may be complex and need to stay atomic. This allows items to fail separately.
And we got better error reporting as a bonus.
Instead of seeing "These 200 items failed" we get "This specific item failed". Much easier to debug. ✌️
Cheers,
~Swizec
PS: even if you know that anything can and will fail on the backend, it's easy to miss issues like this in code review. And this was the most obvious case I've seen.
Continue reading about A quick lesson in writing resilient code
Semantically similar articles hand-picked by GPT-4
- Promise.allSettled, a wonderful tool for resilient code
- Bad excuses to want a rewrite
- Waiting for Godot with callbacks, promises, and async
- Finding unresolved promises in JavaScript
- Async, await, catch – error handling that won't drive you crazy
Learned something new?
Read more Software Engineering Lessons from Production
I write articles with real insight into the career and skills of a modern software engineer. "Raw and honest from the heart!" as one reader described them. Fueled by lessons learned over 20 years of building production code for side-projects, small businesses, and hyper growth startups. Both successful and not.
Subscribe below 👇
Software Engineering Lessons from Production
Join Swizec's Newsletter and get insightful emails 💌 on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.
"Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️