A while back you got A Quick Lesson in Writing Resilient Code and Dominic, a reader, asked "Why not use Promise.allSettled?"
The goal of that lesson was to show how you might consider writing code when anything can and will fail. Unless you're doing an atomic operation, you have to let things fail separately.
For example: When processing payments and User A's credit card gets rejected, you still have to charge and renew User B's account.
Failing separately, approach 1
The easiest approach to isolating failures is when you're using a cron job that executes on a schedule. Fetch items to process, iterate, catch and handle errors.
// consider this pseudocode
async function runsEveryHour() {
const items = await db.fetchUnprocessedItems();
let successCount = 0;
let errorCount = 0;
for (let item of items) {
try {
await db.transaction(async (trx) => {
const result = await doSomethingFancy(item, trx);
if (result) {
successCount += 1;
}
});
} catch (err) {
console.error("Error processing item", item, err);
errorCount += 1;
}
}
console.log(`Processed ${successCount} items; got ${errorCount} errors`);
}
This approach is easy to read and quick to understand.
Fantastic for cron jobs and manual scripts because items are processed one by one. Any errors and debugging logs come out in sequence, which makes this approach easier to debug.
Failing separately, approach 2
Another way to write the above code is using Promise.allSettled, which waits until an array of promises succeeds or fails. It returns the result of each promise.
// consider this pseudocode
async function runsEveryHour() {
const items = await db.fetchUnprocessedItems();
const result = await Promise.allSettled(
items.map(async (item) =>
db.transaction(async (trx) => {
const result = await doSomethingFancy(item, trx);
if (result) {
successCount += 1;
}
})
)
);
const errors = result.filter((r) => r.status === "rejected");
const successes = result.filter((r) => r.status === "fulfilled");
console.log(
`Processed ${items.length} items with ${successes.length} success and ${
errors.length
} errors. The errors are ${JSON.stringify(errors)}`
);
}
We changed the loop to a map that creates an array of promises, fed that into Promise.allSettled, and filtered the result to find errors and successes.
Our processing now runs in parallel, which improves performance, but we lost the ease of debugging. Logs get mixed up between function calls and even though errors contain the Error object, is that enough to know exactly which item failed? Maybe 🤷♀️
Failing separately, approach 3
The ultimate approach, when you have lots of data, is distributed map-reduce processing. I talk about this in the Lambda Pipelines chapter of Serverless Handbook.
This is the same as above – map over an array of items, fan them out to individual processors (like an AWS Lambda), then recombine the result for a final tally.
You'd use this approach when you have massive amounts of data or each individual item takes a long time to process. If you add some queues with built-in retry mechanisms you get a superbly resilient architecture.
But it's a beast to debug end-to-end and overkill for most cases. You do get the benefit of "Item 5 failed, retry?" however so that's nice.
I'd start with the first approach and beef up the machine running my background tasks ✌️
Cheers,
~Swizec
Continue reading about Promise.allSettled, a wonderful tool for resilient code
Semantically similar articles hand-picked by GPT-4
- A quick lesson in writing resilient code
- Finding unresolved promises in JavaScript
- Waiting for Godot with callbacks, promises, and async
- A promises gotcha that will catch you out
- You don't *have to* build it sloppy to go fast
Want to become a JavaScript expert?
Learning from tutorials is great! You follow some steps, learn a smol lesson, and feel like you got this. Then you go into an interview, get a question from the boss, or encounter a new situation and o-oh.
Shit, how does this work again? 😅
That's the problem with tutorials. They're not how the world works. Real software is a mess. A best-effort pile of duct tape and chewing gum. You need deep understanding, not recipes.
Leave your email and get the JavaScript Essays series - a series of curated essays and experiments on modern JavaScript. Lessons learned from practice building production software.
Curated JavaScript Essays
Get a series of curated essays on JavaScript. Lessons and insights from building software for production. No bullshit.
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️