You may be way ahead of me – AI writes good tests actually. It's super handy for this use-case.
A while back I wrote that using AI to derive tests from code is bad because it ruins what tests are for: Validation.
You need two sources of truth to line up. If they don't, something's wrong. If you write code first, then make your tests match – there's no validation.
But I've been doing it more (using Cursor) and it works surprisingly well. Here's how.
Code first, tests AI
A while back I had to fix a function that deals with datetime math. Datetime math breaks my brain. This code broke my brain even worse because it dealt with packages across timezones.
Super leetcodey problem – get a list of packages that should have arrived but haven't. Courier can pick up the package on any day of their pickup schedule even if they missed the first opportunity. What packages are we expecting today?
We knew the code was broken, but not how and we couldn't reproduce.
After playing whack-a-mole for a while I thought "This is dumb, I need tests".
Ask AI to write tests for the function
Highlighted the function and told Cursor "Write tests for this". Hoping to get some boilerplate I can work with.
Cursor wrote fantastic tests! 😳
It was able to read the comments and write tests based on intent, not implementation. They immediately highlighted 2 bugs.
But I didn't believe the AI so I spent the next few hours debugging those tests thinking "How the hell is this failing, the test makes complete sense, I've been through every line a dozen times and even wrote this all out on paper".
The code was wrong, not the tests. 🤦♂️
Ask AI to fix its tests
There were a couple instances where the tests were wrong. Small details in how mocks were used or when the AI wrote extra code instead of reusing a function it didn't know about.
You can ask AI to iterate on those things. "Hey you got this wrong because blah blah". A lot like working with a junior.
Use AI for input combination salad
Varied inputs are hard to validate. Iterating through a representative sample of valid inputs is the hardest problem in testing.
Add state and you're doomed. Sure, your code works with a blank database, what about with 3 years of user history? What if there's a full moon and you just farted?
These rare cases become super common as your app grows. 1 in a million becomes every day.
If you're gonna test, test
In theory there are no guarantees that your code works on all valid inputs! You say every number is okay, but have you tried? Strings are fine, but have you verified? Users are okay, but have you tried every user? Even that dude who signed up 5 years ago, never used the app, forgot their account existed, and came back yesterday?
The combinatorial explosion is too great. You'll never test everything. That's why you need observability more than tests.
BUT! AI can help you test more. It's great at turning sentences like "Write a test for when ..." into tests. Now your job is to come up with input combinations that might break the code, not the schlep of writing those tests.
That's much more fun! And rewarding. And it's fine if the tests are slop. Nobody reads those.
Cheers,
~Swizec
PS: AI shines at tasks where answers are annoying to generate but quick to validate. Like writing database migrations.
PPS: the function that inspired this works like a charm now. Having unit tests was invaluable for self-contained algorithmic coding like that. The function is a typical S-program – purely mapping inputs to outputs
Continue reading about AI writes good tests, actually
Semantically similar articles hand-picked by GPT-4
- Coaching AI to write your code
- Why you shouldn't use AI to write your tests
- Why a coding AI like Github Copilot won't take your job
- Livecoding #38 - A faux AI that writes JavaScript
- Programming in Markdown
Learned something new?
Read more Software Engineering Lessons from Production
I write articles with real insight into the career and skills of a modern software engineer. "Raw and honest from the heart!" as one reader described them. Fueled by lessons learned over 20 years of building production code for side-projects, small businesses, and hyper growth startups. Both successful and not.
Subscribe below 👇
Software Engineering Lessons from Production
Join Swizec's Newsletter and get insightful emails 💌 on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.
"Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️