Good observability is like a safety net. You get to move fast, write fewer tests, and know that if production blows up you'll find out quick.
But in practice I've noticed that most alerts we set up are kinda useless. They sit there, trigger once in a while, and we go "Eh we should fix that one day".
Yes it's great to know when a background process with automatic retries and error handling hits a hiccup. Fixing those makes your system run smoother.
And it's hard to know in advance what will make an effective alert. You have to deeply understand your system and usage patterns. It comes with experience and getting it wrong a lot?
That's why engineers should own their own systems. None of that separate SRE nonsense. If you get paged a bunch of times for useless errors, you'll fix it quick.
Here's a few alerts I've found unreasonably effective over time:
- SQL Errors – a few errors are a warning, a spike pages someone. This has caught so many production bugs it's embarrassing.
- High latency – when your p95 suddenly goes 3x above baseline, something is struggling. Great way to find a database that's about to keel over, for example.
- High error rate – lots of errors (404 and 400 count) between services means you shipped a bug. On public-facing it means you shipped a bug or there's a bot that needs to get blocked.
Honestly for inter-service communication you should treat any 400 error as a definite bug and send a warning (but not a page). You control both sides of the system. If they can't talk to each other, fix it. Great way to find issues that only happen in production. You could even use those payloads to then write an integration test.
Cheers,
~Swizec
Continue reading about These 3 alerts catch the most issues
Semantically similar articles hand-picked by GPT-4
- Why you need observability more than tests
- 90% of performance is data access patterns
- Make mistakes easy to fix
- What helps you ship confidently?
- Let small fires burn
Learned something new?
Read more Software Engineering Lessons from Production
I write articles with real insight into the career and skills of a modern software engineer. "Raw and honest from the heart!" as one reader described them. Fueled by lessons learned over 20 years of building production code for side-projects, small businesses, and hyper growth startups. Both successful and not.
Subscribe below 👇
Software Engineering Lessons from Production
Join Swizec's Newsletter and get insightful emails 💌 on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.
"Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️