RFC: MDX plugins

[johno]

Summary

The plugins from remark and rehype have been great to add in features and functionality, however I think MDX-specific plugins can take advantage of features more inline with the MDX language.

Motivation

There are many common needs in the MDX ecosystem (frontmatter, table of contents, syntax highlighting, oembed, etc.) which don't make sense in core but are complex to set up or configure. Not to mention, there's some fragmentation developing around how these things are accomplished in different ecosystems (like Gatsby and Next.js).

Providing a plugin-based approach for these common pieces of functionality can help ensure that the best possible solution can be arrived upon and shared amongst a large group of folks. Not to mention, this could also provide a path for folks to experiment and innovate much more in userland.

A table of contents is the perfect example to show how a piece of functionality in an MDX-based stack needs a "vertical slice" of functionality. This allows a single plugin to set up everything it needs across all transformations MDX => MDAST => HAST => JSX => React|Vue.

The way it works today

First, let's consider the ecosystem as it currently stands. For this RFC we will use adding a table of contents as the example functionality. Without MDX plugins, the current flow looks like:

• Add remark-slug
• Install github-slugger which remark-slug uses
• Customize h1-h6 to be linkable using github-slugger
• Create a custom plugin that would assign headings for toc as an export (or use a custom loader, gatsby-plugin-mdx, etc)
• Build a component for import/render that turns the toc into a linked list
• Know the data export for the toc and pass to component

One could also use remark-toc to achieve some of this, but it doesn't give you much flexibility in how/when the toc is rendered and it doesn't allow the heading information to be imported from outside the document.

This isn't the end of the world, but it results in a lot of moving parts where a single plugin could handle a lot of this and end up being more flexible.

Basic (proposed) example

Note: This is not intended to be the implementation/usage but an abstract example to illustrate the functionality.

mdx-toc

End user functionality

Inject table of contents data

By default, mdx-toc might process the MDXAST and inject an export of the table of contents as an array.

export const tableOfContents = [
  {
    value: 'Some heading',
    depth: 2
  }
]

# My document

<!-- ... -->

Import table of contents data for usage at the layout level

The injected data could be used at the layout level to add a table of contents.

import { tableOfContents }, Document from './document.mdx'

export default () => (
  <DocumentLayout>
    <SidebarNav />
    <Main>
      <Document />
    </Main>
    <SidebarTableOfContents>
      <TableOfContents headings={tableOfContents} />
    </SidebarTableOfContents>
  </DocumentLayout>
)

Use the table of contents inside the document

import { TableOfContents } from 'mdx-toc'

## Table of contents

<TableOfContents />

This TableOfContents would receive the data it expects as part of its babel transformation.

Plugin developer perspective

A plugin author could expose a file with properties for the different stages in the pipeline (all of which are optional)

module.exports.remarkPlugins = [
  remarkSlug,     // Add slugs
  remarkMdxToc // Adds export const tableOfContents = []
]
module.exports.rehypePlugins = []
module.exports.babelPlugins = [addTableOfContentsProps] // Adds data prop for component

Detailed design

An MDX plugin would be given access to each stage of the transpilation pipeline. This means that a plugin could manipulate the MDAST and HAST in order to add rich functionality.

MDX plugins would be passed as their own option and would be appended to the existing remark/rehype plugins. We'd also need to add babel plugins to the existing babel processing.

The component rendering portion would exist in userland to avoid any magic. They'd have to be manually added to MDXProvider at the layout level.

Ultimately this wouldn't require much engineering effort since we'll be mostly forwarding plugins to remark/rehype/babel. Most of the work will be in making sure the API is solid and documenting it.

Other feature considerations

Perhaps it could make sense to allow plugins to compose in additional functionality via the MDXProvider. Considering the above example for mdx-toc, that same plugin might want to do something like:

// mdx-toc/react.js
import slug from 'github-slugger'

const slugifyComponents = ({
  h1, h2, h3, h4, h5, h6, a
  ...rest
}) => {
  const slugify = Tag => props => (
    if (!props.id) return <Tag {...props} />

    return (
      <Tag {...props}>
        <a href={'#' + props.id}>
          {props.children}
        </a>
      </Tag>
  )

  return {
    a,
    h1: slugify(h1),
    h2: slugify(h2),
    h3: slugify(h3),
    h4: slugify(h4),
    h5: slugify(h5),
    h6: slugify(h6),
    // A plugin could add itself as a shortcode
    TableOfContents,
    ...rest
  }
}

export default slugifyHeadings

Example usage:

// src/components/layout.js
import componentFactory from 'mdx-toc/react'

export default ({ children }) => (
  <MDXProvider components={componentFactory(myComponents)}>
    {children}
  </MDXProvider>
)

Using a context-based approach for plugins to change rendering would allow for more fine-grained customizability for users. Plugins typically act globally on all MDX documents, however it's probable that this isn't desired when MDX is used for a large site with complex needs.

Something worth exploring is the ability to add some type of getInitialProps or query that can be used to pull in data and statically render. This can be super useful for things like Twitter embeds, oembed, or anything else that only needs to fetch data at compile time.

Drawbacks

This will dramatically add to the complexity of the MDX codebase, but I think it will generally be worth the effort and longterm maintenance. It will allow for richer functionality without having to directly modify core and will add additional features that remark plugins can't necessarily provide as easily.

Debugging could get tricky, too. Plugins will have the ability to drastically change MDX compilation and rendering.

Adoption strategy

This would be an optional feature, but will allow us to provide plugins for common asks that don't make sense adding to core.

Related issues

None that I know of.

Acknowledgements

Thanks to @jxnblk, @wooorm, and @ChristopherBiscardi for discussions on this RFC.

[ChristopherBiscardi]

By default, mdx-toc might process the MDXAST and inject an export of the table of contents as an array.

I wonder if we should think about namespacing this early. Something like putting each plugins' const injections in their own object:

export const mdxToc = { tableOfContents: [] }

The injected data could be used at the layout level to add a table of contents.

what about at the wrapper (or other MDXProvider components replacements) level?

This TableOfContents would receive the data it expects as part of its babel transformation.

This feels like we should experiment with a few different ways to achieve it. On one hand, automatically injecting props somehow seems nice because the tableOfContents is a bit of an implementation detail, on the other I'm also curious how custom components and overriding the tableOfContents content would work.

---

Overall this seems like it'll work well with gatsby-plugin-mdx, especially once we get lazy-loading of components. We can drop mdx-toc's tableOfContents in as scope and such, providing it as shortcode through context. exports constructed by plugins will also be exposed in the graphql API for querying, which is an interesting userland solution to customization of tableOfContents content.

Another thing I'm very excited about is the ability to potentially get rid of <pre><code> in favor of a <CodeBlock> using some hypothetical mdx-code-blocks plugin. We could hash the code in a plugin, and insert the id on the root pre while sticking all of the raw strings into the export (maybe looks something like the below). Then when rendering a CodeBlock we can grab the id prop and get the code from the export object.

This could potentially allow us to swap between remark-processed output (like the prism plugins, etc) and repls, leaving the prism processing (or vscode, etc) to build time. We'd need some sort of way to "extend" the exports with remark processing passes that happen after the raw string is plucked into the exports object, which makes me feel like this plugin requires holding state and also requires this plugin having multiple layers operating pre-remark, post-remark, post-hast, etc for full functionality. Thus making this the most complicated mdx plugin I can think of... sorry lol. Given that we've already established that "This will dramatically add to the complexity of the MDX codebase" I think the fact that this sort of really advanced and cool use case is enabled by that complexity is a sign of good things.

Even further, this would expose all code blocks in an MDX content into the Gatsby graphql API 🤯 allowing you to pull them easily for opengraph image generation, reconstructing a working example, etc.

export const mdxCodeBlocks = {
  props: {
    lang: 'js'
  },
  fl23j9fyh: {
    rawCode: `import whatever from thing`,
    vsCodeHighlightedHtmlString: `<div>...</div>`
    extraProps: {}
  }
}

---

Something worth exploring is the ability to add some type of getInitialProps or query that can be used to pull in data and statically render.

If we head down this route we'd probably want to make it pluggable for gatsby and next or anyone else who wants to control how the fetching happens to batch it up, etc.

[BrendanThompson]

@johno – any progress on this actually becoming something available through MDX?

[remcohaszing]

MDX 2 supports remark, rehype, and recma plugins. I think this issue can be closed?

[wooorm]

Perhaps.

I think there’s still room for a package format that supports a) plugins (remark/recma), b) components (TableOfContents above), c) data (tableOfContents above).
That might be either left for userland as a plugin that does these things can be made currently, and solving it might also be bigger than the scope of MDX (it likely has to integrate with frameworks, too)

[wooorm]

Closing. I’d like to see something practical come up first, some TableOfContents product that provides plugins and components. If that’s indeed cumbersome to use, we could think about how to use it easier with MDX.
I do worry that such products would probably be framework-specific (react). And perhaps even “meta-framework”-specific (next). Perhaps then it is better to have a react-toc and next-toc instead of an mdx-toc.