Remark plugins are a little esoteric, and with the rise of Gatsby, theyโre becoming more and more important.
Remark is a markdown parser built in JavaScript. "Parser" doesn't do Remark justice. Remark is a whole compiler ecosystem built on plugins.
It's the backbone of all Gatsby magic that involves converting markdown into HTML.
There are many Remark plugins, and the list keeps growing. This article is about how to build your own. โ๏ธ
Why you should build a Remark plugin
Because you want a feature you can't find yet. Simple as that.
For me, it was two features, so I built 2 plugins. But I packaged them both in the markdown-to-tweet package.
Here's an early preview ๐
Look at me, I can ๐๐๐ฒ๐ฒ๐ ๐ถ๐ป ๐ฏ๐ผ๐น๐ฑ and even use ๐ฆ๐ฎ๐ฑ๐ฉ๐ข๐ด๐ช๐ด wherever I want. Even ๐ญ๐ช๐ฌ๐ฆ ๐ต๐ฉ๐ช๐ด#ThreadCompiler step 1 ๐ค pic.twitter.com/Zp7AIzFBTA
โ Swizec Teller (@Swizec) January 7, 2019
My markdown-to-tweet
does 4 things:
- It converts
**bold**
text into utf8 bolds: ๐ฏ๐ผ๐น๐ฑ _italic_
text into utf8 italics: ๐ช๐ต๐ข๐ญ๐ช๐ค\
code`` text into monospace: ๐๐๐๐- Code blocks into carbon.now.sh screenshots
Screenshots work through an AWS lambda I built for techletter.app a few months ago and never got around to writing a blog about. I should fix that ๐ค
Anyway, I couldn't find a remark plugin that does that, so I built my own. That was my excuse. Kent C. Dodds built one to turn YouTube and Twitter links into embeds on his blog.
You can try my markdown-to-tweet thingy in this rudimentary Codesandbox ๐
Maybe I should release the underlying Remark plugins separately ๐ค
How to build a simple Remark plugin
Like I said, you can think of Remark as a compiler ecosystem. If you look at the traditional compiler architecture from Wikipediaโฆ
โฆyour plugin sits squarely in that "middle code generation" part.
A basic Remark call goes something like this:
import remark from "remark";import html from "remark-html";// ...remark().use(html).process(input_markdown, function (err, html_output) {if (err) throw err;console.log(html_output);});
You import remark
and remark-html
, instantiate remark, and tell it to use the html
plugin. Then you call .process
on your input source, and out comes a parsed HTML for you to use.
Remark itself doesn't do much. It uses unified under the hood to parse Markdown into a MDAST โ markdown abstract syntax tree.
Plugins take this syntax tree and perform transformations. The html plugin, for example, turns nodes into HTML.
Transforming MDAST nodes
Thinking in abstract syntax trees is hard. Lots of recursion and strange patterns if you want to do it yourself. Just think of the last time you tried manipulating the DOM directly. It's a mess.
Lucky for you, the Remark/Unified ecosystem comes with helper methods. It's like having jQuery for abstract syntax trees.
Say you wanted to build a plugin that transforms **bold**
nodes into utf8 bolds. You find a function that handles the utf8 conversion and build a Remark plugin.
Your Remark plugin is a function that returns a transform. Also a function.
import visit from "unist-util-visit";import { bolden } from "./unicodeTransformer";function utf8() {return function transformer(tree, file) {visit(tree, "strong", bold);function bold(node) {node.type = "text";node.value = node.children.map((child) => bolden(child.value)).join(" ");}};}export default utf8;
Simple as that. Let me explain.
You import the visit
helper from Unified utilities, prefixed always with unist-util
for some reason. There's also a map
and a reduce
and all the other methods you commonly use to work with data.
The visit
method deals with the mess and recursion of navigating around an abstract syntax tree.
Your transformer
function gets the current tree
and a reference to the file
that's being compiled.
You call visit()
to do your manipulations.
visit(tree, "strong", bold);
This calls your bold
function on every node of type strong
in your tree
. You can see a list of available MDAST node types here.
Took me a while to find that list. It helps a lot. ๐
The manipulation itself happens in your node visitor โ the bold
function.
function bold(node) {node.type = "text";node.value = node.children.map((child) => bolden(child.value)).join(" ");}
You can do whatever you want. In this case, we're changing node type to text
so future plugins know not to mess with it. Then we take all its children, which are text
nodes, transform their text, and set the resulting string as the value
of the current node.
Text nodes with values are the lowest level in MDAST. They're strings that (usually) don't get changed further and eventually flatten into your output.
Changing node type is very important if you don't want other plugins to mess with your changes. Or if you do and want to tell them how.
This is called the visitor pattern, by the way. It works because JavaScript uses pass-by-reference.
How to build an async Remark plugin
Asynchronous Remark plugins is where it gets tricky. Remark does support async plugins, but you might find it has a tendency to skip your plugin if you do it wrong. Had that problem myself.
Thanks to Huy Nguyen's post for helping me figure it out.
Async plugins work in two phases:
- Collect nodes to change
- Change those nodes
Around that, your transformer has to return a Promise and resolve it after you're done changing nodes. Like this ๐
function codeScreenshot() {return (tree) =>new Promise(async (resolve, reject) => {const nodesToChange = [];visit(tree, "code", (node) => {nodesToChange.push({node,});});for (const { node } of nodesToChange) {try {const url = await getCodeScreenshot(node.value);node.value = url;} catch (e) {console.log("ERROR", e);return reject(e);}}resolve();});}
codeScreenshot
is the Remark plugin in this case. It returns a function that returns a Promise.
Inside that Promise, you can see the two phases of an async Remark plugin.
Collect nodes to change
const nodesToChange = [];visit(tree, "code", (node) => {nodesToChange.push({node,});});
The first step collects nodes we want to change. We're still using the visitor pattern with the visit
helper.
But instead of changing the nodes, we push them into a nodesToChange
array. That's because the visit
function can't deal with asynchronous code.
This is where I got stuck for a couple hours ๐
Change those nodes
for (const { node } of nodesToChange) {try {const url = await getCodeScreenshot(node.value);node.value = url;} catch (e) {console.log("ERROR", e);return reject(e);}}resolve();
Here we iterate over the nodes we collected earlier. getCodeScreenshot
takes a node value, passes it into an API, and returns a URL. Eventually.
We await
that url, then change the node. This is where I should change the node type as well. Maybe turn it into an image? ๐ค
Thanks to how await
magic works, we know our loop finishes only once all the URls have been fetched from the internet. We call resolve()
to tell Remark we're done transforming nodes.
Remark handles the rest.
In conclusion
- Remark plugins are functions.
- Use the visitor pattern to manipulate nodes
- For async plugins collect first, change later
That's how you build a Remark plugin. You have a new superpower ๐ช
Please use it to add wonderful magnificent things to the Remark and Gatsby ecosystem and supercharge all our websites.
Everyone's moving their blogs to Gatsby right now. It's hot. You can make them hotter!
Learned something new?
Want to become a high value JavaScript expert?
Here's how it works ๐
Leave your email and I'll send you an Interactive Modern JavaScript Cheatsheet ๐right away. After that you'll get thoughtfully written emails every week about React, JavaScript, and your career. Lessons learned over my 20 years in the industry working with companies ranging from tiny startups to Fortune5 behemoths.
Start with an interactive cheatsheet ๐
Then get thoughtful letters ๐ on mindsets, tactics, and technical skills for your career.
"Man, love your simple writing! Yours is the only email I open from marketers and only blog that I give a fuck to read & scroll till the end. And wow always take away lessons with me. Inspiring! And very relatable. ๐"
Have a burning question that you think I can answer?ย I don't have all of the answers, but I have some! Hit me up on twitter or book a 30min ama for in-depth help.
Ready to Stop copy pasting D3 examples and create data visualizations of your own? ย Learn how to build scalable dataviz components your whole team can understand with React for Data Visualization
Curious about Serverless and the modern backend? Check out Serverless Handbook, modern backend for the frontend engineer.
Ready to learn how it all fits together and build a modern webapp from scratch? Learn how to launch a webapp and make your first ๐ฐ on the side with ServerlessReact.Dev
Want to brush up on your modern JavaScript syntax?ย Check out my interactive cheatsheet: es6cheatsheet.com
By the way, just in case no one has told you it yet today: I love and appreciate you for who you areย โค๏ธ