Swizec Teller - a geek with a hatswizec.com

Senior Mindset Book

Get promoted, earn a bigger salary, work for top companies

Senior Engineer Mindset cover
Learn more

    What happens when you push AWS credentials to GitHub

    A couple weeks ago I was preparing for a workshop. Making life easy for students on their fun day of hacking, GraphQL, Serverless, React, and Gatsby.

    Click through for source
    Click through for source

    The workshop was great and then my AWS account got suspended πŸ˜…

    You see, a brilliant idea I had was to do file uploads. How hard can it be? A bit of JavaScript, a dash of S3, and voila you've got files on the server.

    Not so.

    Click through for source
    Click through for source

    Uploading files to S3 from the browser is really hard. You need an S3 bucket, that's easy, then you need a server that accepts a file from your submitted form and uploads it to S3 with the SDK and gets back the URL and ...

    Security by default

    You can't upload files straight to S3 from the browser because Amazon believes in security by default. Thou shalt not have an app out in the wild that is not secure.

    Great in theory, annoying when you're tight against a deadline for a toy app you intend to kill 3 days later.

    You can make an S3 bucket public. You can make files public too. You need an S3 access key to upload anyway.

    Yes even if it's a public bucket and a public file. You have to upload securely.

    Were this not the case, anyone could upload a bunch of nefarious things to your S3 and report you to the FBI. You don't want that. πŸ˜‰

    Uploading a file with the S3 SDK looks something like this:

    Click through for source
    Click through for source

    Looks fine, right?

    Notice this part:

    Click through for source
    Click through for source

    If you send that to the browser, anyone can see your credentials and upload nefarious things.

    ugh giphy

    You're supposed to run that code on a server.

    That keeps your credentials secure (unless your server is hacked) and means you now have to deal with file uploads in two places:

    • browser submits file upload form to server
    • server uploads file to S3
    • server returns URL

    But I didn't want a server.

    The whole point of my workshop is that there's no servers and everything runs on AWS Lambda. Plus data transfer on AWS can get expensive.

    You don't want to transfer each file twice. First to your lambda, then to your S3.

    Click through for source
    Click through for source

    Presigned URLs – a secure way to upload

    AWS offers a secure solution πŸ‘‰ pre-signed URLs.

    With a pre-signed URL, your server creates a unique URL with credentials, filenames, and other params baked in. Your browser code makes a fetch() request with some data and a file shows up on your S3 bucket.

    You request a new pre-signed URL for the next file.

    I'll write more about secure S3 file uploads from the browser soon, but here's the server code in a nutshell:

    Click through for source
    Click through for source

    You still need a server, but you can run this on a Lambda. It's a small request that's quick to calculate. A perfect fit ☺️

    Returns an upload URL that expires in 5 minutes with a pre-baked filename and read URL.

    So how did Swizec's workshop get his AWS account suspended?

    See this part of my code:

    Click through for source
    Click through for source

    Before I cleaned my GitHub that contained actual access keys. I knew it was bad, I knew I shouldn't, I did it anyway. Correct secrets management just felt too complicated in this case.

    First I created a special S3 user with access to a specific bucket. Then I copy pasted those credentials into my code and pushed away.

    brave giphy

    You might steal my credentials but it's okay. They're limited!

    And boy oh boy did that make the workshop wonderful. Having access keys baked into my code made everything so easy. Pull my code and get started right away.

    We built the whole UI before students had to even think about AWS. It was fire. πŸ”₯

    AWS saw what I did πŸ˜…

    Click through for source
    Click through for source

    A few days before the workshop, almost as soon as I "leaked" my credentials, Amazon sent me an email.

    Hey we saw your credentials got leaked. We get it, shit happens, no biggie, please fix it ... ... AND IF YOU DON'T WE WILL KILL YOUR ACCOUNT

    Whoa wait what? They can't be serious.

    So I ignored the warning and thought to myself Pfft, paranoid ops people, it's an example account. Who cares

    They kept emailing me. First every day. Then every couple hours. Like yo, you should really fix this.

    After the workshop I though fine fine, if you insist, I'll fix it.

    What to do when you leak AWS credentials

    Once your AWS credentials are out there, there's only one thing you can do: change everything

    1. You have to change your AWS password
    2. You have to go into the IAM app and delete all existing keys for all sub-users
    3. Generate new keys
    4. Scrub GitHub clean

    First 3 are easy. Go into your AWS console and click the things.

    The 4th ... how the hell do you scrub your entire GitHub history of a specific string? BFG

    Click through for source
    Click through for source

    Put the strings you want to replace in passwords.txt and BFG rewrites your entire history. Each password becomes a accessKeyId: "***REMOVED***", ... but you can still see the diff. Lol

    xQFcArr

    I guess that's part of why you also delete the keys. Deactivating isn't enough Amazon says, you have to delete.

    You might still get banned

    Once you've finished all the steps tell Amazon. They might be omniscient in detecting that you did something bad, but they don't see that you fixed it. I don't know why.

    I forgot to tell them and my account got suspended.

    Click through for source
    Click through for source

    swizec.com looked like a brutalist website, techletter.app didn't work so I couldn't write newsletters, everything died. You don't know how much of your life relies on AWS until it dies.

    cry giphy

    Amazon support was wonderful however. A little aloof when scolding me for leaking the credentials, but absolutely wonderful once I got suspended and started crying about it on Twitter.

    Click through for source
    Click through for source

    Click through for source
    Click through for source

    What surprised me most was that I could use Support with a suspended account. Can't log into AWS, can't see any of the services, but you can see support and get help.

    Top notch UX πŸ‘Œ

    Cheers,
    ~Swizec

    PS: everything works now ^_^

    Did you enjoy this article?

    Published on September 13th, 2019 in Back End, Technical,

    Senior Mindset Book

    Get promoted, earn a bigger salary, work for top companies

    Learn more

    Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.

    Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.

    Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help πŸ‘‰ swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.

    Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers πŸ‘‰ ServerlessHandbook.dev

    Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization

    Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections

    Want to brush up on modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog

    Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❀️

    Created by Swizec with ❀️