Part 7: Advanced Topics

Introduction

Congrats, you’ve successfully gone through all the step-by-step instructions to build out your source plugin. You can compare what you have with the finished example repository.

This part of the tutorial will explain individual advanced topics in a reference style, not tutorial style like the parts before. Or in other words: This living document won’t give step-by-step explanations but rather general instructions, tips, and recommendations. Integrating them all into the example repository would go beyond the scope of this guide. Pick and choose what makes sense for your own source plugin, although we highly recommend always adding test coverage to your plugin.

While this document recommends a limited set of tools, feel free to use the tools you’re familiar and productive with.

Testing

From our experience a well tested source plugin is more reliable and easier to maintain. You have greater confidence that new features and bug fixes don’t introduce new bugs. It’ll also help you understand the code better after coming back to it after a while, especially if not you but a collaborator has written it.

There is no silver bullet in testing and it’ll depend on your source plugin what form of test coverage it needs. Generally speaking, a mix of unit and end-to-end tests gives a good foundation for every source plugin.

Unit testing

Here are some tips for unit testing source plugins:

• We’d recommend using Jest or Vitest as unit testing frameworks.
  • For detailed instructions on setting up Jest with Gatsby, read the Unit Testing guide.
• The most important part of your source plugin is the sourceNodes API. You’re fetching data there and creating nodes from that data. Both parts can be unit tested.
  • Don’t run unit tests against an API that is online somewhere. Your tests will fail when the API is momentarily offline or the machine itself doesn’t have an internet connection. Use HTTP mocking packages like nock to create a placeholder API for your tests.
  • If you’re modifying the incoming data before creating nodes, be sure to test that behavior. In the previous parts of this tutorial you’ve created utility functions (nodeBuilder and createAssetNode) as one of their advantages is that you can then test their logic in isolation.
• Read the Unit testing an options schema guide for the pluginOptionsSchema API.

The finished example repository uses Jest to unit test the pluginOptionsSchema and utility functions for sourceNodes.

End-to-End testing

If you want to test your plugin inside a browser and a running Gatsby app, you should use end-to-end testing tools. You could test the Image CDN feature you’ve added in Part 6 or if certain schema customizations are working correctly. It can also give you confidence that your source plugin works during gatsby build from start to finish.

Here are some tips for end-to-end testing source plugins:

• We’d recommend using Cypress or Playwright as end-to-end testing frameworks.
  • For detailed instructions on setting up Cypress with Gatsby, read the End-to-End Testing guide.
• If you have a monorepo with your plugin and an example site (like this tutorial) you should use the example site for your end-to-end tests.
• When running your end-to-end tests in a CI environment (e.g. tests running on each PR) do test the output of gatsby build. So before running the tests, run gatsby build and then gatsby serve.
  • Locally you can test against gatsby develop to iterate more quickly on the tests.

Content Sync

Content Sync is a Gatsby Cloud feature for improving the preview experience for content authors. Be sure to read the Content Sync conceptual guide before continuing. To enable this feature in your source plugin you will need to make sure that your data source (or CMS) also works with Content Sync.

Source plugin

The source plugin needs to create node manifests using the unstable_createNodeManifest action.

1. Create the customCreateNodeManifest utility function that will be a wrapper around the unstable_createNodeManifest action. It contains the logic for when to call the action and with which arguments.

   The attributes argument should allow you to create a unique manifest ID later on. isPreview exists because the source plugin should only create node manifests for preview content. The customNodeFilteringFn function is a placeholder for your custom logic to filter out any items depending on your gatsbyNode. You may want to filter out certain types of nodes, for example if your CMS has nodes that will never be previewed like redirect nodes or other types of non-content data.

2. Identify which nodes you’ll want to create a node manifest for. These typically are nodes that you can preview, entry nodes, top level nodes, etc. An example of this could be a blog post or an article, any node that can be the “owner” of a page. A good place to call this action is whenever you call createNode.

3. unstable_createNodeManifest has two required arguments: manifestId and node. node is the Gatsby node you can reuse, manifestId needs to be created.

   Important: manifestId needs to be created with attributes coming from the CMS only as the CMS will need to create the exact same manifest, too. You must not use Gatsby information for it.

   This manifestId must be uniquely tied to a specific revision of specific content. In the example below the id of the content and a timestamp is used. This will vary from your data source/CMS.

   unstable_createNodeManifest optionally also accepts a third argument called updatedAtUTC which should be the time in which the node was last updated. If you don’t have that information feel free to leave it out.

Data source / CMS

The CMS needs to send a preview webhook to Gatsby Cloud when content is changed and open the Content Sync waiting room. The details of the instructions below will differ from CMS to CMS depending on how extensions and their settings are created/stored.

You’ll need two pieces of information from Gatsby Cloud:

• Preview webhook URL (format: https://webhook.gatsbyjs.com/hooks/data_source/<id>)
• Content Sync URL (format: https://gatsbyjs.com/content-sync/<siteId>)

You can find both URLs on the Gatsby Cloud site settings page (read Build and Preview Webhooks to learn more). Both URLs need to be user configurable in the CMS extension.

Here are the other steps to integrate Content Sync into your CMS:

1. Store both URLs inside the CMS extension so that you can reuse them later.

2. Inside the CMS extension, create a manifestId with the same attributes as shown in the source plugin section. They must match.

3. Store a contentId inside the extension. The contentId should be a unique identifier for the project and stay consistent across all previews. So e.g. storing the project ID is a good option.

   contentId will enable so called “eager redirects” which causes the user to be redirected to their site frontend as soon as possible. When a user first previews their content, they’ll stay in the Content Sync loading screen until their preview is ready. On subsequent previews of the same piece of content, they’ll be redirected as soon as the page loads.

4. Create a button in your CMS that does the following:

   1. POST to the preview webhook URL of Gatsby Cloud

   2. Open the Content Sync waiting room. You can do this by redirecting to an URL in the shape of:

      As an example:

Tips

Here are some things to keep in mind and some “gotchas” depending on how the CMS acts:

• Inside the CMS, sometimes you will need to wait to make sure you have the correct updatedAt timestamp as some CMS may take a second to update their backend and then wait for the change to propagate to the frontend. While others will immediately update the frontend and then propagate that to the backend. You will need the most up to date timestamp when opening the Content Sync UI waiting room.
• Make sure that a preview webhook is being sent to Gatsby Cloud after the content is edited, whether it’s before you press the “Open Preview” button or the “Open Preview” is the trigger that sends the webhook.
• The node manifests get written out in the public dir of your gatsby site, so you can check the manifests on your local disk at /public/__node-manifests/<sourcePluginName>/<manifestId>.json or you can navigate directly to that piece of content https://<your-domain>/__node-manifests/<sourcePluginName>/<manifestId>.

MIME types

A media type (also known as a Multipurpose Internet Mail Extensions or MIME type) indicates the nature and format of a document, file, or assortment of bytes. Learn more about MIME types on MDN.

In Part 2 you learned how to use createNode. Inside the internal object that holds the type and contentDigest you can optionally also pass a mediaType key.

The usage for a markdown file looks something like this:

Source plugins are responsible for setting the mediaType on their nodes (if necessary). By providing this key you tell transformer plugins (see Creating a Transformer Plugin guide) that the node contains raw content that they can transform. Use either official media types or a made-up one if your data doesn’t fit into any existing bucket.

This distinction between Gatsby source plugins and transformer plugins is important as it allows both to be contained and focused on their specific job. Source plugins don’t need to worry about transforming their data, transformer plugins don’t care where the data is coming from.

For example, the node above with the text/markdown MIME type will be transformed by gatsby-transformer-remark into further MarkdownRemark nodes.

Tracing

If you want to give your users the ability to see tracing of your source plugin (e.g. in OpenTelemetry format) or use the performance tracing yourself to debug bottlenecks, be sure to read the Performance Tracing guide. The guide has all the information you’d need.

In context of source plugins, the tracing and parentSpan Node API helpers are relevant.

When calling startSpan on the root tracer pass the parentSpan along, for example:

Summary

You’ve also learned about more advanced topics now, great!

Take a moment to think back on what you’ve learned so far. Challenge yourself to answer the following questions from memory:

• Which parts should you unit test?
• What are end-to-end tests good for?
• What is Content Sync?
• What happens when a source plugin defines a mediaType for its nodes?

Key takeaways

• A good mixture of unit tests (testing the heart of the source plugin) and end-to-end tests (testing the functionality from start to finish) will give you confidence in shipping changes.
• Content Sync can improve the preview experience for content editors.
• MIME types can help Gatsby transformers discover nodes from source plugins.
• You can use tracing to inspect your source plugin more deeply.

What’s coming next?

In Part 8 you’ll learn how you can publish your source plugin to npm, making it available to everyone.