Gatsby Cloud Docs

Learn How to Create a First Class Source Plugin for Gatsby Preview

In this guide, we provide technical details for creating a source plugin optimized for Gatsby Preview

Table of Contents

Introduction

To be considered a first-class plugin for inclusion and marketing efforts for Gatsby Preview, there are several criteria that must be met. The very first of these is that a source plugin must exist, and the second is that the source plugin coupled with the CMS must be able to drive updates in less than five seconds—ideally much faster. This design document focuses on helping guide source plugin authors to be able to craft and deliver this excellent experience that Gatsby Preview customers expect and that your customers will love.

Definitions + Terminology

First, some basic terminology and definitions. A plugin is an extension point invoked via Gatsby at some point in a Gatsby application’s lifecycle. Plugins will typically invoke at least one Gatsby API, e.g. an API available in gatsby-node.js, gatsby-browser.js, etc. A plugin is quite simply a distributable package invoking these exposed APIs to do something useful in the context of a Gatsby application.

A source plugin is a specific type of plugin that extends the Gatsby GraphQL schema with data sourced from anywhere. This data could be Markdown files from the local filesystem, it could be JSON data, or in the case of Gatsby Preview, it is most often data sourced from a headless CMS. Most typically, the sourceNodes API in gatsby-node.js is used to extend the GraphQL schema and allow a Gatsby application to query this schema at build time.

Finally, a node is a JavaScript object that is able to be queried with Gatsby’s GraphQL API, at build time. A source plugin generates one to many nodes, and these nodes are then requested, filtered, and rendered to build out a highly optimized, ludicrously fast, statically generated application. The node is at the center of Gatsby’s data model and is a core concept to understand just exactly what a source plugin is doing.

Enough enough—let’s dive into implementation details.

How to Create a First Class Source Plugin for Gatsby Preview

The necessary feature that must be available in a source plugin to be considered first-class is the ability to fetch new, changed, or deleted data programmatically and augment the GraphQL schema to reflect these changes. For the best user-experience, there may also be a concept of “draft” or “preview” content, which is a mechanism to trial out changes before publishing the changes to the content. The central process of updating data can be done minimally in two ways:

  1. A source plugin designed in such a way that it registers a listener and is able to control and detect changes on its own
  2. A webhook triggered from the CMS itself to then trigger an exposed Gatsby Preview webhook, which will invoke the sourceNodes API sourceNodes, when re-invoked, will be able to pick up where it last left off (e.g. a synchronization step) and then request changes since last invoked - gatsby-source-contentful is an example of this approach - Note: while this approach works, it can be a little bit slower than the listener approach. If possible, we strongly encourage the listener approach if your existing infrastructure and APIs can support it.

While the approaches differ slightly, the central ideas are the same. A source plugin for Gatsby Preview must necessarily:

  • Invoke createNode to create or update nodes generated since the last update
  • Invoke deleteNode to delete nodes that are no longer present in the headless CMS
  • Invoke touchNode to ensure that nodes that haven’t changed are not deleted

This process must occur in the sourceNodes API, and this process must be able to pick up where it last left off (or listen for changes) for a user to be able to update content in a headless CMS and see the changes reflected live in Gatsby Preview.

Technical Detail

Up to this point, this has been fairly high-level. Let’s get to the nitty-gritty of this document and probably why you’re here: the technical details.

Packaging is outside the scope of this document, but you’ll first and foremost need a GitHub repository and a process wherein you can publish to NPM. A source plugin will typically have a structure like:

gatsby-source-your-great-cms
 ├── .babelrc
 ├── .gitignore
 ├── .npmignore
 ├── CHANGELOG.md
 ├── README.md
 └── src
     └── gatsby-node.js

For more examples, check out any of the packages in the GatsbyJS monorepo.

Now we can get to developing. Let’s start iterating in our gatsby-node.js. The very first thing we’ll want to do is we need a way to source data, and to be able to iterate over the returned list of items (e.g. content stored in the CMS) and be able to create nodes for each of these items. A very simplified representation could look something like this:

const axios = require("axios")

const NODE_TYPE = `YourSourceItem`

exports.sourceNodes = async function sourceNodes(
  { actions, createContentDigest, createNodeId },
  pluginOptions
) {
  const { createNode } = actions

  const { data } = await axios.get(
    `https://your-cms-here.com/${pluginOptions.spaceId}`
  )

  data.items.forEach(item => {
    const nodeMetadata = {
      id: createNodeId(`your-source-${item.uuid}`),
      parent: null,
      children: [],
      internal: {
        type: NODE_TYPE,
        content: JSON.stringify(item),
        contentDigest: createContentDigest(item),
      },
    }

    const node = Object.assign({}, item, nodeMetadata)
    createNode(node)
  })
}

Let’s walk through some of that code.

First, we register the API with CommonJS syntax, specifically exports.sourceNodes. Gatsby will invoke this API with necessary helpers and pluginOptions. The first argument to this function is internal Gatsby helpers that we provide for your convenience and to enable the augmentation of Gatsby’s data layer. The second (pluginOptions) are passed when the plugin is added in gatsby-config.js, oftentimes like so:

module.exports = {
  plugins: [
    {
      resolve: "gatsby-source-your-great-cms",
      options: {
        spaceId: "1234",
      },
    },
  ],
}

In this example, pluginOptions will be an object with the shape of:

{
  "spaceId": "1234"
}

Next, we need a way to request data. In this example, we’re invoking a REST API with our given spaceId. We receive back a list of items, and we iterate over each of those items. Finally, we create the specific node structure that Gatsby requires (e.g. id, parent, children, and internal) and then assign our data so that it can be queried in GraphQL. To tie it all together, if we consider a resource returned from our REST API:

{
  "uuid": "abcd-1234-except-you-know-unique",
  "title": "Hello World",
  "date": "2019-05-24",
  "author": "John Doe-ish",
  "draft": false,
  "body": "Hello! This could be Markdown content, or HTML, or whatever your user's expect."
}

We can then query for this data with Gatsby’s GraphQL API, like so:

{
  allSourceData {
    nodes {
      title
      body
      author
      date
    }
  }
}

This matches the shape of the data we provided from the REST API, and all key/value pairs we add to the node will be able to be queried from the Gatsby GraphQL schema.

However — up to this point, this is not a useful source plugin from a Gatsby Preview perspective. We have provided no way for data to be updated, which provides no way for our content to be changed, edited, or deleted in a Gatsby Preview environment. We’re stranding our content editors and users of our CMS, and leaving them no mechanism to see what their content changes will look like in the live site! Let’s fix that.

Our approach will be relatively simple but one that illustrates the central idea of how to structure a first-class source plugin for Gatsby Preview. We will listen to a web socket exposed by our API (note: this could be any type of listener, e.g. GraphQL Subscriptions, or an NPM package that wraps your API and exposes the ability to listen for changes, etc.). The key idea here is that our API will both:

  1. Expose a WebSocket that we will listen to when in preview mode
  2. The WebSocket will return data for creates, updates, and deletes

With this approach, you can now create a simple, fast, and efficient source plugin that will handle its own updates, rather than following the webhook approach.

const axios = require("axios")
const WebSocket = require("ws")

const NODE_TYPE = `YourSourceItem`
let ws

/*
 * A shared helper to be re-used in Preview + build flow
 */
const createNodeFromData = (item, helpers) => {
  const nodeMetadata = {
    id: helpers.createNodeId(`your-source-${item.uuid}`),
    parent: null,
    children: [],
    internal: {
      type: NODE_TYPE,
      content: JSON.stringify(item),
      contentDigest: helpers.createContentDigest(item),
    },
  }

  const node = Object.assign({}, item, nodeMetadata)
  helpers.createNode(node)
  return node
}

/*
 * sourceNodes will be repeatedly re-invoked whenever content changes in the CMS
 * as such, we need a way to statefully update, delete, and create new nodes
 * that have been modified since we last synced
 */
exports.sourceNodes = async function sourceNodes(
  { actions, createContentDigest, createNodeId, getNode, getNodesByType },
  pluginOptions
) {
  const { createNode, deleteNode, touchNode } = actions

  const helpers = Object.assign({}, actions, {
    createContentDigest,
    createNodeId,
    getNode,
    getNodesByType,
  })

  if (pluginOptions.preview) {
    if (!ws) {
      ws = new WebSocket(`ws://www.your-cms-here.com/${pluginOptions.spaceId}`)
    }

    /*
     * First, we'll ensure nodes aren't garbage collected if they're not changed in this call
     */
    getNodesByType(NODE_TYPE).forEach(node => touchNode({ nodeId: node.id }))

    /*
     * Next, we'll set up the web socket listener to
     * listen for changes and update, delete, create appropriately
     * It's presumed we receive a buffer of JSON data
     */
    ws.on(`message`, buffer => {
      const data = JSON.parse(buffer.toString())
      data.items.forEach(item => {
        const nodeId = createNodeId(`your-source-${item.uuid}`)
        switch (item.status) {
          case "deleted":
            deleteNode({
              node: getNode(nodeId),
            })
            break
          case "created":
          case "updated":
          default:
            /*
             * Created and updated can be handled by the same code path
             * The item's uuid is presumed to stay constant (or can be inferred)
             */
            createNodeFromData(item, helpers)
            break
        }
      })
    })

    return
  }

  /*
   * The existing, non-preview code path!
   */
  const { data } = await axios.get(
    `https://your-cms-here.com/${pluginOptions.spaceId}`
  )

  data.items.forEach(item => createNodeFromData(item, helpers))
}

At this point, we have a source plugin that has the ability to not only create data, but also update data. When data is deleted, deleteNode will be called. When data is created or modified, we will create or re-create that node with the updated data. sourceNodes will be repeatedly called by Gatsby whenever data in your CMS is updated.

We have created a potentially first-class source plugin. Now we will need to test, validate, and eventually release our super-charged plugin to our users.

Next Steps

  • Identify alpha/beta testers of your CMS who would be interested in testing out Gatsby’s Cloud offerings and your super-charged source plugin
  • Reach out to Gatsby Cloud engineer(s) for a code review of your new and improved source plugin
  • Publish an alpha release of your plugin (e.g. with an npm tag, or a pre-release version)
  • Reach out to Gatsby Cloud to get test accounts for your organization and users
  • Solicit feedback from your users, and validate whether the plugin appropriately creates, updates, and deletes data in the Gatsby Preview environment upon change(s) in the CMS
  • Validate that the source plugin takes less than five seconds on all types of updates, deletes, etc.

Once you’ve been able to validate that the source plugin works and meets not only your user’s needs but also the needs of Gatsby Preview as a first-class integration, you can now progress through the rest of the check-list. Reach out to us when you’re ready—we can’t wait to see your company on our list of first-class integrations.