This is a Livecoding Recap – an almost-weekly post about interesting things discovered while livecoding. Usually shorter than 500 words. Often with pictures. Livecoding happens almost every Sunday at 2pm PDT on multiple channels. You should follow My Youtube channel to catch me live.
Really what we need is AI. We write the tests, AI builds code that passes.
— Swizec Teller (@Swizec) May 18, 2017
True test driven development!
Think about how a product manager writes code.
They define some features, come up with constraints, and unleash a team of engineers. A few days pass while stuff they don't understand goes on, and voilà! New features with working code and business metrics twitching to and fro as users respond to new goodies.
Great, huh?
Now think about how a software engineer writes code.
They talk to the product manager and get a "This is what it should do" spec, then they agree on "This is how much it should cost in time/resources to build". Then the engineer looks at existing code and gets a "This is the code the new stuff should fit into". Then they come up with a set of tests, automated or otherwise, that answer the question, "Does it work yet?"
With tests in hand, the engineer sets to work. Painstakingly manipulating characters on screen, typing and typing and running and testing and eventually they have code that is syntactically correct and doesn't break existing stuff and implements the new stuff. Hooray!
Great, huh?… kinda tedious 🤔
Your core value add as an engineer is to help the product manager come up with sensible constraints. To help them include the new stuff into the existing stuff. To suggest constraints they didn't think of. And to add constraints they shouldn't have to worry about.
All that code writing stuff… meh.
I know, it's not just typing. It takes smarts and experience, and it's really hard to do well. But if you had good tests and infinite memory and infinite time, you could keep generating random strings until one of them happened to make the tests pass. You'd call that string The Code.
Similar idea as the monkeys with typewriters thought experiment and the random sort algorithm. Do random stuff until requirements are met.
Expected running time: Infinity.
Because you're smart and experienced and a grizzled veteran or an eager newbie, it takes you less than infinity to write that code. You use smart heuristics like "not typing random stuff" and "actually thinking about what to do".
Although I often just keep trying random things when things get really tough. 😇
Wouldn't it be cool if someone could do that for you? I mean, if computers can write news, why not the code that makes computers write news?
Here's the idea: You write the tests, AI makes them pass.
Experiment 1 - failed
I'm calling this first experiment “faux AI” because it doesn't know what it's doing. The current approach is at the "Paste random characters and fingers crossed" stage.
We tried an evolutionary algorithm based on manipulating code as character strings. This did not work.
Our population kept getting stuck at a maximum far away from the result. Once every member of the population had the same fitness score, they stopped changing and a solution was never reached.
The situation improved once we tweaked our fitness function and added constraint weights. Instead of having a series of true/false conditions, we weighted them by importance.
// We aim to maximize this value
function fitness(code, conditions) {
const max = conditions.reduce(
(sum, { condition, weight }) => sum + weight,
0
),
fit = conditions
.map(({ condition, weight }) => Number(condition(code)) * weight)
.reduce((sum, fit) => sum + fit, 0);
return fit / max;
}
const Conditions = [
{
condition: (code) => !(run(code) instanceof Error),
weight: 10,
},
{
condition: (code) => run(code) === 4,
weight: 8,
},
{
condition: (code) => code !== "4",
weight: 2,
},
{
condition: (code) => code !== 4,
weight: 2,
},
{
condition: (code) => (code.length > 4 ? 4 / code.length : 1),
weight: 6,
},
];
The fitness function takes a piece of code and walks through a set of conditions. We test each condition and multiply the result with its weight. A syntactically valid piece of code gets 10, an invalid one gets 0. One that produces the correct result gets8, an invalid result gets 0, etc.
We're looking for code that is shorter than 4 characters, isn't 4 or "4", and returns 4. A good solution would be, say, 2+2, or 1+3. Even a = 4 works. Or maybe "afwaelfjlkwefj"; 4.
The tricky part of evolutionary algorithms is that even when they work, their results are weird. Best things in the world at finding loopholes in your conditions.
Our evolutionary algorithm looks like this:
function* generation({ population, fitness, N }) {
while (true) {
population = rank(population, fitness);
// pairwise breed top 50% of population
population = population.concat(
breedPairs(_.chunk(_.take(population, population.length / 2), 2))
);
population = rank(population, fitness);
population = _.take(population, BIGGEST_POPULATION);
yield {
fitness: fitness(population[0], Conditions),
code: population[0],
fitnessLast: fitness(population[population.length - 1], Conditions),
codeLast: population[population.length - 1],
size: population.length,
};
}
}
That's a generator, by the way. Call .next() to get the next generation from the current population. Call it as many times as you want; it keeps working :)
Each generation starts with an existing population, which is ranked by the fitness function. We make pairs out of the best 50% of the specimens, breed them, and add their offspring to the population.
The new population is again ranked, and if there's more specimen than we have room for, the bottom few are thrown away. They're useless.
You can see how the breedPairs and rank functions work on Github.
This current approach did not produce a single working piece of code. This was the best solution with a score of 0.5535714285714286
" pci s |ePke "
At least it's syntactically valid 🤔
This field of research is known as genetic programming. I don't think anyone has gotten it to the point of replacing software engineers yet, but I remember a cool project from my high school days by some Slovenian person called Critticall. It was able to invent novel sorting algorithms.
Next steps
The experiment failed, and the code is slow.
Maybe we should switch to Python? Or parallelize the generation generator. Evolution is, after all, highly parallelizable and there are 4 cores in my computer…
Fundamentally, we're gonna have to change the level of abstraction. You cannot evolve code as if it was a random series of characters. Maybe if our algorithm could understand syntax, or at least JavaScript symbols, and evolve an AST, then it might work.
I think there's also room for neural nets and deep learning in the evolution step. 🤔
Ideas very welcome. I will continue to dabble with this as time allows.
Continue reading about Livecoding #38 - A faux AI that writes JavaScript
Semantically similar articles hand-picked by GPT-4
- Livecoding #39: Towards an AI that writes JavaScript
- Evolving a poem with an hour of python hacking
- Haskell and randomness
- Programming in Markdown
- Why you shouldn't use AI to write your tests
Learned something new?
Read more Software Engineering Lessons from Production
I write articles with real insight into the career and skills of a modern software engineer. "Raw and honest from the heart!" as one reader described them. Fueled by lessons learned over 20 years of building production code for side-projects, small businesses, and hyper growth startups. Both successful and not.
Subscribe below 👇
Software Engineering Lessons from Production
Join Swizec's Newsletter and get insightful emails 💌 on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.
"Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. 👌"
Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.
Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.
Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help 👉 swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.
Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers 👉 ServerlessHandbook.dev
Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization
Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections
Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog
Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️