Swizec Teller - a geek with a hatswizec.com

    How to build a Remark plugin to supercharge your static site

    Remark plugins are a little esoteric, and with the rise of Gatsby, theyโ€™re becoming more and more important.

    Remark is a markdown parser built in JavaScript. "Parser" doesn't do Remark justice. Remark is a whole compiler ecosystem built on plugins.

    It's the backbone of all Gatsby magic that involves converting markdown into HTML.

    There are many Remark plugins, and the list keeps growing. This article is about how to build your own. โœŒ๏ธ

    Why you should build a Remark plugin

    Because you want a feature you can't find yet. Simple as that.

    For me, it was two features, so I built 2 plugins. But I packaged them both in the markdown-to-tweet package.

    Here's an early preview ๐Ÿ‘‡

    Swizec Teller back on his Be An Expert bullshit avatarSwizec Teller back on his Be An Expert bullshit@Swizec
    Look at me, I can ๐˜๐˜„๐—ฒ๐—ฒ๐˜ ๐—ถ๐—ป ๐—ฏ๐—ผ๐—น๐—ฑ and even use ๐˜ฆ๐˜ฎ๐˜ฑ๐˜ฉ๐˜ข๐˜ด๐˜ช๐˜ด wherever I want. Even ๐˜ญ๐˜ช๐˜ฌ๐˜ฆ ๐˜ต๐˜ฉ๐˜ช๐˜ด

    #ThreadCompiler step 1 ๐Ÿค˜
    Tweet media

    My markdown-to-tweet does 4 things:

    1. It converts **bold** text into utf8 bolds: ๐—ฏ๐—ผ๐—น๐—ฑ
    2. _italic_ text into utf8 italics: ๐˜ช๐˜ต๐˜ข๐˜ญ๐˜ช๐˜ค
    3. \code`` text into monospace: ๐šŒ๐š˜๐š๐šŽ
    4. Code blocks into carbon.now.sh screenshots

    techletter app screenshot 1547049367743

    Screenshots work through an AWS lambda I built for techletter.app a few months ago and never got around to writing a blog about. I should fix that ๐Ÿค”

    Anyway, I couldn't find a remark plugin that does that, so I built my own. That was my excuse. Kent C. Dodds built one to turn YouTube and Twitter links into embeds on his blog.

    You can try my markdown-to-tweet thingy in this rudimentary Codesandbox ๐Ÿ‘‡

    Maybe I should release the underlying Remark plugins separately ๐Ÿค”

    How to build a simple Remark plugin

    Like I said, you can think of Remark as a compiler ecosystem. If you look at the traditional compiler architecture from Wikipediaโ€ฆ

    Compiler architecture
    Compiler architecture

    โ€ฆyour plugin sits squarely in that "middle code generation" part.

    A basic Remark call goes something like this:

    import remark from "remark";
    import html from "remark-html";
    // ...
    remark()
    .use(html)
    .process(input_markdown, function (err, html_output) {
    if (err) throw err;
    console.log(html_output);
    });

    You import remark and remark-html, instantiate remark, and tell it to use the html plugin. Then you call .process on your input source, and out comes a parsed HTML for you to use.

    Remark itself doesn't do much. It uses unified under the hood to parse Markdown into a MDAST โ€“ markdown abstract syntax tree.

    Plugins take this syntax tree and perform transformations. The html plugin, for example, turns nodes into HTML.

    Transforming MDAST nodes

    Thinking in abstract syntax trees is hard. Lots of recursion and strange patterns if you want to do it yourself. Just think of the last time you tried manipulating the DOM directly. It's a mess.

    Lucky for you, the Remark/Unified ecosystem comes with helper methods. It's like having jQuery for abstract syntax trees.

    Say you wanted to build a plugin that transforms **bold** nodes into utf8 bolds. You find a function that handles the utf8 conversion and build a Remark plugin.

    Your Remark plugin is a function that returns a transform. Also a function.

    import visit from "unist-util-visit";
    import { bolden } from "./unicodeTransformer";
    function utf8() {
    return function transformer(tree, file) {
    visit(tree, "strong", bold);
    function bold(node) {
    node.type = "text";
    node.value = node.children.map((child) => bolden(child.value)).join(" ");
    }
    };
    }
    export default utf8;

    Simple as that. Let me explain.

    You import the visit helper from Unified utilities, prefixed always with unist-util for some reason. There's also a map and a reduce and all the other methods you commonly use to work with data.

    The visit method deals with the mess and recursion of navigating around an abstract syntax tree.

    Your transformer function gets the current tree and a reference to the file that's being compiled.

    You call visit() to do your manipulations.

    visit(tree, "strong", bold);

    This calls your bold function on every node of type strong in your tree. You can see a list of available MDAST node types here.

    Took me a while to find that list. It helps a lot. ๐Ÿ˜…

    The manipulation itself happens in your node visitor โ€“ the bold function.

    function bold(node) {
    node.type = "text";
    node.value = node.children.map((child) => bolden(child.value)).join(" ");
    }

    You can do whatever you want. In this case, we're changing node type to text so future plugins know not to mess with it. Then we take all its children, which are text nodes, transform their text, and set the resulting string as the value of the current node.

    Text nodes with values are the lowest level in MDAST. They're strings that (usually) don't get changed further and eventually flatten into your output.

    Changing node type is very important if you don't want other plugins to mess with your changes. Or if you do and want to tell them how.

    This is called the visitor pattern, by the way. It works because JavaScript uses pass-by-reference.

    How to build an async Remark plugin

    Asynchronous Remark plugins is where it gets tricky. Remark does support async plugins, but you might find it has a tendency to skip your plugin if you do it wrong. Had that problem myself.

    Thanks to Huy Nguyen's post for helping me figure it out.

    Async plugins work in two phases:

    1. Collect nodes to change
    2. Change those nodes

    Around that, your transformer has to return a Promise and resolve it after you're done changing nodes. Like this ๐Ÿ‘‡

    function codeScreenshot() {
    return (tree) =>
    new Promise(async (resolve, reject) => {
    const nodesToChange = [];
    visit(tree, "code", (node) => {
    nodesToChange.push({
    node,
    });
    });
    for (const { node } of nodesToChange) {
    try {
    const url = await getCodeScreenshot(node.value);
    node.value = url;
    } catch (e) {
    console.log("ERROR", e);
    return reject(e);
    }
    }
    resolve();
    });
    }

    codeScreenshot is the Remark plugin in this case. It returns a function that returns a Promise.

    Inside that Promise, you can see the two phases of an async Remark plugin.

    Collect nodes to change

    const nodesToChange = [];
    visit(tree, "code", (node) => {
    nodesToChange.push({
    node,
    });
    });

    The first step collects nodes we want to change. We're still using the visitor pattern with the visit helper.

    But instead of changing the nodes, we push them into a nodesToChange array. That's because the visit function can't deal with asynchronous code.

    This is where I got stuck for a couple hours ๐Ÿ˜…

    Change those nodes

    for (const { node } of nodesToChange) {
    try {
    const url = await getCodeScreenshot(node.value);
    node.value = url;
    } catch (e) {
    console.log("ERROR", e);
    return reject(e);
    }
    }
    resolve();

    Here we iterate over the nodes we collected earlier. getCodeScreenshot takes a node value, passes it into an API, and returns a URL. Eventually.

    We await that url, then change the node. This is where I should change the node type as well. Maybe turn it into an image? ๐Ÿค”

    Thanks to how await magic works, we know our loop finishes only once all the URls have been fetched from the internet. We call resolve() to tell Remark we're done transforming nodes.

    Remark handles the rest.

    In conclusion

    • Remark plugins are functions.
    • Use the visitor pattern to manipulate nodes
    • For async plugins collect first, change later

    That's how you build a Remark plugin. You have a new superpower ๐Ÿ’ช

    Please use it to add wonderful magnificent things to the Remark and Gatsby ecosystem and supercharge all our websites.

    Everyone's moving their blogs to Gatsby right now. It's hot. You can make them hotter!

    Did you enjoy this article?

    Published on January 9th, 2019 in Front End, Technical,

    Learned something new?
    Read more Software Engineering Lessons from Production

    I write articles with real insight into the career and skills of a modern software engineer. "Raw and honest from the heart!" as one reader described them. Fueled by lessons learned over 20 years of building production code for side-projects, small businesses, and hyper growth startups. Both successful and not.

    Subscribe below ๐Ÿ‘‡

    Software Engineering Lessons from Production

    Join Swizec's Newsletter and get insightful emails ๐Ÿ’Œ on mindsets, tactics, and technical skills for your career. Real lessons from building production software. No bullshit.

    "Man, love your simple writing! Yours is the only newsletter I open and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. ๐Ÿ‘Œ"

    ~ Ashish Kumar

    Join 15,883+ engineers learning lessons from my "raw and honest from the heart" emails.

    โญ๏ธโญ๏ธโญ๏ธโญ๏ธโœจ
    4.5 stars average rating

    Have a burning question that you think I can answer? Hit me up on twitter and I'll do my best.

    Who am I and who do I help? I'm Swizec Teller and I turn coders into engineers with "Raw and honest from the heart!" writing. No bullshit. Real insights into the career and skills of a modern software engineer.

    Want to become a true senior engineer? Take ownership, have autonomy, and be a force multiplier on your team. The Senior Engineer Mindset ebook can help ๐Ÿ‘‰ swizec.com/senior-mindset. These are the shifts in mindset that unlocked my career.

    Curious about Serverless and the modern backend? Check out Serverless Handbook, for frontend engineers ๐Ÿ‘‰ ServerlessHandbook.dev

    Want to Stop copy pasting D3 examples and create data visualizations of your own? Learn how to build scalable dataviz React components your whole team can understand with React for Data Visualization

    Want to get my best emails on JavaScript, React, Serverless, Fullstack Web, or Indie Hacking? Check out swizec.com/collections

    Want to brush up on modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    Did someone amazing share this letter with you? Wonderful! You can sign up for my weekly letters for software engineers on their path to greatness, here: swizec.com/blog

    Want to brush up on your modern JavaScript syntax? Check out my interactive cheatsheet: es6cheatsheet.com

    By the way, just in case no one has told you it yet today: I love and appreciate you for who you are โค๏ธ

    Created by Swizec with โค๏ธ