Swizec Teller - a geek with a hatswizec.com

    Serverless Chrome on AWS Lambda, the guide that works in 2019 and beyond

    Getting Chrome and Puppeteer to work on AWS Lambda is ... tricky. There's many guides, many blogs out there and they're all broken in subtle little ways you won't realize until it's too late.

    Outdated versions of Chrome, build packages too big for an AWS Lambda, or silently breaking with an infinite timeout. When you end up trawling through old Github issues looking for clues you know something's wrong.

    That was me.

    Click through for source
    Click through for source

    I'm writing this guide as a love letter to my future self. Never again will I fall into this trap. Maybe it helps you too :)

    Serverless Handbook for Frontend Engineers – free chapter

    Dive modern backend. Understand any backend.

    Serverless Handbook taught me high-leveled topics. I don't like recipe courses and these chapters helped me to feel like I'm not a total noob anymore.

    The hand-drawn diagrams and high-leveled descriptions gave me the feeling that I don't have any critical "knowledge gaps" anymore.

    ~ Marek C, engineer

    Start with a free chapter and email crash course ❤️

    Join over 10,000 engineers just like you already improving their careers with my letters, workshops, courses, and talks. ✌️

    The objective

    We're building a serverless screenshot service. It's the TODOapp of the headless serverless Chrome world. Every other guide uses this example :D

    You can see my final repo on Github. Disregard the README. A long time ago I used a starter kit that no longer works. Chrome version too old, hangs on newPage.

    We'll take screenshots of Instagram posts. Convenient for embedding in emails. You could use a similar setup for testing or to make social sharing thumbnails. Whatever your heart desires.

    FBZeWxj

    I built my screenshot Lambda for TechLetterApp. A tool that converts markdown into newsletters.

    You will learn about:

    • serverless
    • AWS Lambdas
    • headless Chrome
    • Chrome Puppeteer
    • uploading to S3

    Our Lambda is going to take an Instagram URL, take a screenshot, upload the file to S3, and return the URL.

    Init new project

    Start a new project. We're going raw, no init scripts. Don't need 😇

    Click through for source
    Click through for source

    Yarn or npm will ask you a bunch of questions about your new project. Answer them and you'll end up with a package.json file.

    This is also a great opportunity to run git init.

    S3 bucket

    Before we begin, you'll need an S3 bucket as well. There is surely a way to create these via Serverless, but I don't know it.

    Go into your AWS console, search for S3, create a new bucket and give it a name.

    zvF6SZ3

    Serverless.yml

    We start our project with serverless.yml – the Serverless configuration file. Handles all our AWS setup so we can avoid Amazon's confusing dashboard and avoid clicking around. Also means your configuration and server setup becomes part of your code so it's repeatable and always the same.

    Version controlled too 👌

    I'm assuming you already have Serverless and have set up your AWS credentials.

    Create a serverless.yml file and add this stuff 👇

    Click through for source
    Click through for source

    Here's what it means

    • service gives your service a name. Used when constructing endpoints for your Lambdas and such
    • provider tells Serverless that we're using AWS and sets some basic config. Which node version we're using, how much memory we want for Lambdas, the default timeout.
    • provider/iamRoleStatements configures permissions for our service. Our service needs upload access to S3. Permissions are the most common source of weird bugs and failures in my experience. Best copy from somewhere else :)
    • functions specifies our Lambda functions. We have a screenshot-function handled by the handler method in src/index.js
    • package specifies service packaging instructions. We're telling Serverless to exclude the local Chromium build just in case it's there.
    • plugins Serverless comes with a rich ecosystem of plugins. This is where we specify them. We need serverless-plugin-chrome
    • custom lets us add a bunch of configuration. Usually for our plugins. Here we specify some runtime flags for Chrome and tell serverless-plugin-chrome to make Chrome available to the screenshot-function Lambda

    Other guides will show you different ways of running Chrome. Those are outdated and bad. This is the best approach.

    Using serverless-plugin-chrome means you're running a remote cloud version of Chrome instead of packaging the entire binary into your Lambda. Makes deploys faster, the Lambda smaller, and your costs lower. I think.

    Don't quote me on the costs 😅

    Install dependencies

    Next we need to install some libraries.

    Click through for source
    Click through for source

    @serverless-chrome/lambda will let us talk to Chrome gm is ImageMagick, we'll use it to resize and optimize images for size puppeteer-core is a minimal version of Puppeteer so we can drive Chrome with JavaScript superagent is a JavaScript library for making requests. We're using it because I copied that part of the code from another guide aws-sdk has a bunch of utilities for talking to AWS. In our case for S3 uploads. url will make it easier for us to parse URLs serverless-plugin-chrome handles all the Chrome stuff for us. We don't have to care 👌

    handler.js

    Our handler.js file takes care of one thing: Accepting an HTTP request, setting CORS headers, initiating Chrome stuff, calling our code, and responding with an HTTP response.

    Looks like this

    Click through for source
    Click through for source

    Check the comments above to understand the code. Easier that way :)

    So we've got our handler. Let's create the other files.

    getChrome.js

    I borrowed getChrome from another guide. Good guide but some details didn't work out for me and lead me straight to a sea of GitHub issues.

    getChrome makes sure to launch Chrome and return an instance of the browser we can use.

    Click through for source
    Click through for source

    Ask @serverless-chrome/lambda to run the browser and use superagent to get its webSocket URL. We use it in the next step above to connect with Puppeteer.

    Great, that's our browser. Nothing packaged into our Lambda making it fat. Wonderful.

    takeScreenshot.js

    Now the fun stuff, taking a screenshot. I'm gonna explain the code in comments so it's easier to follow 🤓

    Click through for source
    Click through for source

    In a nutshell, our takeScreenshot function opens a new browser page, navigates to a specific URL, looks for the exact DOM node we want to screenshot, uploads to S3, and returns the result.

    I used to try closing the page after screenshotting but that seems to hang sometimes for some websites. Causes problems so it's best to avoid.

    Lambdas kill their execution environment, and the browser, after running anyway. Shouldn't be any harm in keeping pages open.

    As you can see I wrote this code so it's easy to add other websites. ✌️

    uploadScreenshot.js

    Last step in the puzzle is uploading our screenshot to S3. You can't host from temporary Lambda files. Gotta put it somewhere permanent.

    Again, I'll explain the code inline. Think that's a good approach.

    Click through for source
    Click through for source

    We should now have everything in place to use our Instagram screenshot service.

    The very important hack

    This is a hack. It's ugly. I found it in GitHub issues. Fixes the "socket hang up" error with Chrome.

    Go into your node_modules/@serverless-chrome/lambda/dist/bundle.* files. There's two of them.

    In each of those files look for a line like this

    Click through for source
    Click through for source

    Remove the --v=99 part

    Click through for source
    Click through for source

    You have to do this every time you run Yarn or reinstall anything for any reason

    Yes it sucks. Yes I should make an upstream PR. But doing it yourself unblocks you and makes your Lambda work right now.

    I have no idea why that breaks it.

    Another option might be to remove the DEBUG flag in your serverless.yml ... but then how will you debug?

    And, deploy :)

    Click through for source
    Click through for source

    Serverless packages your lambda, deploys to AWS, sets everything up for you, and returns a URL.

    That URL is your API endpoint. Call it with an Instagram URL and behold your wonderful screenshots.

    I wrapped mine in a nice UI. You can try it at TechLetter.App

    PS: I'm working on new education materials, workshops, course and such for serverless, lambdas, full stack React, GraphQL, and Gatsby. Hit reply if you're interested or ping me on Twitter.

    Did you enjoy this article?

    Published on April 11th, 2019 in Back End, Technical

    Want to dive into serverless? Not sure where to begin?

    Serverless Handbook was designed for people like you getting into backend programming.

    360 pages, 19 chapters, 6 full projects, hand-drawn diagrams, beautiful chapter art, best-looking cover in tech. ✌️

    Serverless Handbook on your bookshelf
    Serverless Handbook on your bookshelf

    Learn how to choose the right database, write cloud functions, think about scalability, gain the architecture mindsets for robust systems, and more.

    Leave your email to start with a free chapter and email crash course 👇

    Serverless Handbook for Frontend Engineers – free chapter

    Dive modern backend. Understand any backend.

    Serverless Handbook taught me high-leveled topics. I don't like recipe courses and these chapters helped me to feel like I'm not a total noob anymore.

    The hand-drawn diagrams and high-leveled descriptions gave me the feeling that I don't have any critical "knowledge gaps" anymore.

    ~ Marek C, engineer

    Start with a free chapter and email crash course ❤️

    Join over 10,000 engineers just like you already improving their careers with my letters, workshops, courses, and talks. ✌️

    Have a burning question that you think I can answer? I don't have all of the answers, but I have some! Hit me up on twitter or book a 30min ama for in-depth help.

    Ready to Stop copy pasting D3 examples and create data visualizations of your own?  Learn how to build scalable dataviz components your whole team can understand with React for Data Visualization

    Curious about Serverless and the modern backend? Check out Serverless Handbook, modern backend for the frontend engineer.

    Ready to learn how it all fits together and build a modern webapp from scratch? Learn how to launch a webapp and make your first 💰 on the side with ServerlessReact.Dev

    Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    By the way, just in case no one has told you it yet today: I love and appreciate you for who you are ❤️

    Created bySwizecwith ❤️