Swizec Teller - a geek with a hatswizec.com

Senior Mindset Book

Get promoted, earn a bigger salary, work for top companies

Senior Engineer Mindset cover
Learn more

    Stable Diffusion on an iPad ๐Ÿคฏ

    When's the last time a piece of tech blew your mind?

    For me it's the new M2 iPad Pro. I got one this weekend to replace the old iPad from when I was looking for the perfect light work device. The iPad + pencil + magic keyboard combo has become my main daily driver. It's where I do all my writing, emails, light doomscrolling, business management, and gentle examples coding.

    The new iPad has more processing power than my work laptop ๐Ÿคฏ

    8-core CPU, 10-core GPU, and 16-core "neural engine" which I think is a GPU optimized for neural networks. In a 494 gram package with no active cooling, half the thickness of my pinky finger, and a battery that feels like it lasts forever.

    Sure, the big fat laptop has more GPU and bigger RAM, but the fans make lots of noise and the laptop weighs 2177 grams. Cosplaying as a desktop machine most of its days.

    Stress testing the new iPad with stable diffusion

    The iPad's biggest holdback is its operating system. Optimized for mobile devices and machines with far less power, the OS won't let you push the limits.

    Thomas on twitter suggested I try Draw Things, an AI drawing tool powered by on-device Stable Diffusion, a generative art model.

    And ... wow. For 2 hours, I couldn't stop. The iPad even started getting warm ๐Ÿ˜

    First I tried a silly prompt: "young racing driver winning the championship by van gogh"

    Pretty impressive. Then the prompt engineering started. You have to tell these models what you want in a lot of detail. Simple english, lots of adjectives, strong descriptions. Vibes don't work. The more explicit you can be, the better.

    Eventually I got a decent picture of a Porsche 911 racing in the desert.

    an old porsche 911 rally racing in the sand dunes going very fast during sunset and raising lots of dust photorealistic action shot frontal view
    an old porsche 911 rally racing in the sand dunes going very fast during sunset and raising lots of dust photorealistic action shot frontal view

    The trick is to iterate fast on low settings โ€“ this is 30 "iterations" โ€“ then when you like the prompt, punch up the values.

    Same prompt, 70 iterations
    Same prompt, 70 iterations

    The pre-trained Stable Diffusion 1.5 model that I'm using is great with cars. You can try any make and model and get a recognizable result. Descriptive car names don't work. Better to say "BMW M3" than "sporty saloon car".

    I think that's down to the training data.

    Animals are more of a struggle. Species don't work, describing the features you want is better.

    sun conure parrot with red and yellow head sitting on a wooden perch indoors and eating seeds from a bowl by rembrandt
    sun conure parrot with red and yellow head sitting on a wooden perch indoors and eating seeds from a bowl by rembrandt

    Playing with different styles is nice. This one is a "by rembrandt", but when I tried another artist, the AI instead decided to replace the bowl of food with another bird ๐Ÿคจ

    sun conure parrot with red and yellow head sitting on a wooden perch indoors and eating seeds from a bowl abstract style by jeff koons
    sun conure parrot with red and yellow head sitting on a wooden perch indoors and eating seeds from a bowl abstract style by jeff koons

    But that's to be expected. The AI has no idea what it's doing. These images are created by using an image of pure noise then removing noise step by step and saying "Does this look like what I want?".

    The very essence of Michelangelo's "Oh statue of david? Easy. Take a block of marble and remove anything that isn't the statue of david"

    What can you do with this?

    I don't know. But here's a passable bowl of chinese food

    digital painting of a chicken chow mein on a porcelain plate with chopsticks on the table
    digital painting of a chicken chow mein on a porcelain plate with chopsticks on the table

    Right now the big problem is that these images require lots of cherry picking and iteration. You can't trust the AI to run on its own. And that's limiting.

    Cheers,
    ~Swizec

    PS: I wanna try running this on an AWS Lambda, that would be fun

    Did you enjoy this article?

    Published on November 22nd, 2022 in stable diffusion, ai, experiments

    Senior Mindset Book

    Get promoted, earn a bigger salary, work for top companies

    Learn more

    Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.

    Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.

    Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help ๐Ÿ‘‰ swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.

    Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers ๐Ÿ‘‰ ServerlessHandbook.dev

    Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization

    Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections

    Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog

    Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    By the way, just in case no one has told you it yet today: I love and appreciate you for who you are โค๏ธ

    Created by Swizec with โค๏ธ