Part 4: Utilities
In the previous parts you built the foundation of your source plugin, from now on it’s about adding new functionalities, improving the user experience, and making the plugin better overall.
This part of the tutorial will focus on a subset of the Node API helpers. Feel free to revisit this document as it can also work as standalone instructions for those APIs.
By the end of this part of the tutorial, you will be able to:
- Use the
reporterAPI to output structured terminal messages
- Use the
cacheAPI to save artifacts between runs
- Define custom errors for your plugin through
- Use the
pluginOptionsSchemaAPI to verify your plugin’s options
You, as a source plugin author, can interact with your users in a variety of ways. You can share information in the README, chat in issues & PRs, create guides, etc. But no matter how active of a maintainer you are, it is your source plugin that will interact the most with your users through the terminal when they run
gatsby develop or
gatsby build. Gatsby outputs what it does behind the scenes with the goal of giving the users rich information that helps them be more productive with Gatsby itself.
For this, Gatsby internally uses the
reporter API and you can use it, too! When adding logs to the terminal be mindful of keeping of them concise, informative, and to a minimum. Most Gatsby plugins don’t even need to add logs (other than errors), so in doubt leave them out at the beginning and reevaluate once you have a bigger user base.
If you need information about an API, always be sure to check Gatsby’s API docs, in this case the reporter API docs. The
reporter API is available in all Node APIs and its methods can be used like this:
Here’s a short explanation on the most important methods:
info: Print a message to the terminal.
warn: Print a warning message to the terminal.
error: Print an error message to the terminal.
panic: Print an error message to the terminal and immediately exit the process.
panicOnBuild: Print an error message to the terminal and immediately exit the process (only during
gatsby build). Most often you should use this over
panicas it’ll allow users to debug the error during
verbose: Print a message to the terminal that is only visible when the “verbose” flag is enabled (e.g.
gatsby build --verbose).
activityTimer: Print an informational message to the terminal that has a timer attached to it (e.g. how long that step took).
setErrorMap: Set a custom error map to the reporter. This allows the reporter to extend Gatsby’s internal error map.
You’ll learn how to use
setErrorMap in the next three tasks as they are the most relevant and common for plugins.
As source plugins interact with third-party APIs they are inherently prone to errors (network issues, API outages, etc.). As explained in Part 2 it’s good practice to gracefully handle all possible error states, as there’s nothing worse than no actionable feedback at all.
In Part 2 you also added the
fetchGraphQL function that accesses the GraphQL API inside the
api folder. When you send a malformed request to it, it can not only give back
data but also
errors. Use that information to output it with
Go to your
gatsbyApiat the top of
errorsto the destructuring statement for the
fetchGraphQLresponse. Also add a conditional statement when
errorsis truthy directly below
panicOnBuildto output the error message that you’re receiving from your API.
Erroror a structured error (you’ll learn about the latter in Task: Define custom errors). Also add an early return to not continue creating nodes.
Pro tip: Try to rely on TypeScript types to check if the error you’re getting back is a valid
Errortype. If not, you might need to modify the error a little bit to fit the requirements.
To see if it works, add a typo to your GraphQL request inside
develop:sitescript and you should see an error in the terminal:
Yes, it works 🎉 Don’t forget to remove the typo again. In Task: Define custom errors you’ll learn how to further customize this error.
Pro tip: This is the error that the GraphQL API directly returns. You can see the same error if you go to
http://localhost:4000/graphql(the GraphQL API from
apifolder) and try the query there. Get yourself familiar with how, when, and in which form your API throws errors.
You’re now surfacing any errors that occur during the data fetching to your users. Use the
panic methods throughout your code to display all relevant errors to your user.
Please note: If necessary, add context to your error messages instead of only passing the error along. This is especially true if the error could be thrown in multiple places or if the original error is not clear enough.
Since your source plugin is retrieving information from a third-party API, that process could take a while, especially if it’s fetching a lot of data. In these instances you can use the
activityTimer method to let the user know that something is happening in the background. You shouldn’t add separate
activityTimer instances (remember: Keep the output concise and to a minimum) but instead use the
setStatus method on
activityTimer to update its text.
Now you’ll learn exactly that, by adding an activity for the sourcing and node creation process of your plugin.
Create a new
activityTimerand store that in a variable (the name doesn’t matter). You’ll use the variable to start, update, and stop the timer. Try to use a text that clearly indicates your plugin name and what it’s doing.
sourcingTimerwas initialized before the
fetchGraphQLcall you can also
start()it before that call:
start()call the timer is running until you’ll eventually call
If the sourcing fails, the timer should be stopped. So update the
panicOnBuildcall to use the
Please note: When looking at the TypeScript types for
sourcingTimeryou’ll notice that it also implements
panic. You can pass in the same arguments as you’re used to, the
activityTimerwill handle everything for you. If you had continued to use
reporter.panicOnBuildthe terminal would output a successful message for the timer which is incorrect of course!
Right now the
sourcingTimershows “Sourcing from plugin API” together with a time in seconds. You can use the
setStatus()method to append text.
Pro tip: Adding the information of how many types were created is a really nice debugging help since it’ll allow folks to quickly compare builds and see if e.g. all the information they expect is sourced.
Stop the timer once node creation is done:
develop:sitescript and you should read a new activity in the terminal:
That’s what we’d expect, nice!
The error you’ve output to the terminal in Task: Output errors was functional but missing some crucial information. Where is the error coming from? Which part of the plugin? Any suggestions for immediate fixes?
You can use the
setErrorMap method to create an error map for your plugin. Gatsby internally uses an error map to benefit from two things (among other additional benefits):
- A user can google the error with its unique ID and hopefully only find resources that are directly related to that error.
- One can track down the location in the source code where an error is thrown quicker through the use of constants and unique IDs.
Both benefits also apply to you when you define a custom error map for your plugin. You’ll see the improvement at the end of this task. As a reminder, this is how an error (for the data fetching step) currently looks like:
Let’s get to work:
plugin/src/constants.tsfile and add an
ERROR_CODESmap with its first key
The keys of the
ERROR_CODESobject can be arbitrary, we’d recommend using names that make it easy to differentiate them for their different purposes. The value of each entry is the unique ID. So make sure that each ID only exists once inside
You don’t need to worry about other plugins or Gatsby itself as the IDs are namespaced with your plugin. If possible, use IDs that are not already used by Gatsby itself.
setErrorMap. Remove the existing
You can populate the error map like so:
- The object’s keys are the unique IDs
- The object should contain these three keys and values:
text: Your text that will be shown to the user for this specific error
level: Set this to
category: Set this to
Add the following to
texthas to be a function that returns a
stringand receives a freeform
context. You can pass through any information you’d like in
contextand then use it to construct the final message that is shown to the user. So the error above requires
Pro tip: If your
textshould be a multi-line string, consider using the
stripIndentfunction from common-tags. This way your error will be displayed correctly independent from how you write the string.
In this step you’ll invoke the error with the custom ID and pass along the required
plugin/src/source-nodes.tsand add the
ERROR_CODESimport and use it inside the existing
As in step 3 of Task: Output errors, add a typo to the GraphQL error again (e.g.
title2), restart the
develop:sitescript and you should see a new output for the error:
Your error now has a custom ID (
#plugin_10000), it prints the name of the plugin (
PLUGIN), and you’ve added additional context to the error itself (
Sourcing from the GraphQL API failed:). Whenever you need to output an error message, create a new structured error inside
setErrorMap and use it to display the best possible error to the user.
You might already know that Gatsby creates two folders during its build phase:
public. It’s recommended to keep both folders around in between builds as then Gatsby can leverage its caching functionalities the best.
With the cache API you have access to a key-value store to persist data inside the
.cache folder. You can use it to persist timestamps, tokens, etc. to use delta updates, or to store the artifacts of time/memory/cpu intensive tasks (e.g. images, big files).
If the term “delta update” doesn’t ring a bell, here’s a short explanation in context of a source plugin: A delta update requires the source plugin to only fetch data from the remote API that is new, or has been changed from a previous state, in contrast to having to fetch all the data again.
Example usage: Your API endpoint accepts an optional query parameter of
lastFetched. If used, only updates since that date are returned. When storing the timestamp of last sourcing with the
cache API you then can use that date in your request, e.g.
Important: Your remote API has to support delta updates, it’s not a universal thing that works everywhere.
The most common use case of the
cache API is to store timestamps, tokens, etc. and therefore you’ll learn exactly that. Even though the example API this tutorial uses doesn’t support delta updates, you’ll go through all the necessary steps in this task to apply it to your own source plugin.
cache API is a key-value store, so you save specific data under a key, and then can retrieve a value with a given key.
Store the name of the key in an constants to minimize the likelyhood of typos. Open
Import the new
plugin/src/source-nodes.tsand grab the
Generate a date timestamp just before the
fetchGraphQLcall and use
cache.setto save the timestamp after the successful fetch. This way you’re ensuring accurate timestamps that are only saved when a successful data call happened. The key you use for
cache.sethas to be unique.
Date.now()returns the number of milliseconds elapsed since the epoch. If you need another format, read the Date.now() documentation.
It’s also important to know that
cacheis shared by all instances of your plugin. So if your user can add your plugin multiple times to
gatsby-config(e.g. by setting a different
workspaceIdin the plugin options), you need to ensure that those
cache.setcalls use unique keys. In those cases, use a unique instance identifier, for example said plugin options.
The next step is to use
cache.getto retrieve the timestamp from the cache before the
fetchGraphQLcall. Since you won’t use it for delta fetching in this tutorial, output the information in the
verbosemode of Gatsby with
Time to test your changes! Restart the
develop:sitescript but this time with the
The output got a lot busier! You should see a new log:
You’re seeing this because on this first run the
lastFetchedDatedidn’t exist yet and thus
undefinedis returned from the
cache.getcall. Stop the development server and restart the script. You now should see a timestamp defined that looks something like this:
Great, it’s working!
Pro tip: If you save files with the
cacheAPI you can also inspect what is saved by going to the
.cache/cachesfolder and looking for a folder named like your plugin. For this tutorial the path would be
In most cases a Gatsby source plugin needs one or more options as it interacts with third-party APIs that require customization and authentication (e.g. different endpoints or API keys). You don’t want other users to use your credentials, so you’ll need a way for your users to provide theirs. That’s where plugin options come in!
The Configuring Plugin Usage with Plugin Options guide goes into depth about plugin options and the
pluginOptionsSchema API so if you want to know more beyond this guide, be sure to give it a read. But to give a short summary:
Usage of that option in a Node API:
A plugin can optionally define a schema (through the
pluginOptionsSchemaAPI) to enforce a type for each option. Gatsby will validate that the options users pass match the schema to help them correctly set up their site.
Here are some tips when working with plugin options:
- Consider the options a public API contract between your users and your plugin. Follow SemVer versioning when adding, modifying, and removing options. For example, if you want to remove an option you should only do this in a major release of your plugin since it’s a breaking change.
- Add as few plugin options as necessary. In the beginning, use safe defaults where possible (to the best of your knowledge) and when users wish to have new options available, seriously contemplate if you should add them. Once added, you’ll need to maintain the option forever (or until you remove it again).
- Use clear names. Naming things is hard, but take extra care in using clear names for your plugin’s options. Best case scenario is that an option is clear enough without any additional description.
- Use “enable” options, not negated options. An example: Instead of using
noWarnings: true, use
warnings: false. The double negation that would occur with
noWarnings: falseis really confusing.
- Pass options through. If your source plugin is using a library under the hood (e.g. an SDK from a CMS) that has options on its own, allow people to configure all those options, too. You could collect those under a new key in the
options.sdkOptions) and then refer people to the SDK’s documentation.
Instead of hardcoding the URL to the GraphQL API, you’ll use an
endpoint option and thus make the URL configurable.
The boilerplate you cloned at the beginning already has an
endpointoption defined in the site’s
gatsby-config. Go to
site/gatsby-config.tsto inspect the code:
So the plugin already receives the GraphQL endpoint through the
IPluginOptionsKeysTypeScript type inside
plugin/src/types.tscurrently has a generic
[key: string]: anycatch-all. Replace it with your
endpointoption so that both internally and externally the types are correct:
Open the file
plugin/src/source-nodes.tsand validate that you can access the option. You can do this by accessing the second parameter of the
console.logit. Use the
IPluginOptionsInternaltype for it:
develop:sitescript and you should see an output in the terminal like this:
Don’t worry about the
pluginskey there. Since Gatsby’s plugins can also have sub-plugins, the key
pluginsis added by default to the
pluginOptionsobject. What you really should care about is that the
endpointoption successfully comes through.
pluginOptionsobject and use it in the
Once again, restart the
develop:sitescript and you should see the log
Sourcing from plugin API - 0.033s - Processing 3 posts and 2 authorsto indicate that the sourcing was successful.
You’ve successfully used a plugin option in your source plugin’s source code.
Right now a user could use any serializable value for
endpoint like numbers, objects, strings, or booleans. But those are not valid options for
endpoint, it should be a string, more specifically a valid URI. Use the
pluginOptionsSchema API to validate exactly that.
Create a new file called
plugin-options-schema.tsinside the plugin’s
srcfolder with the following contents:
endpointin the schema. It should be a string (a valid URI), should be required, and have a short description. You can write something like this:
pluginOptionsSchemaAPI in the plugin’s
gatsby-nodefile so that it is run:
You can test that it works correctly by opening up the site’s
gatsby-configand changing the option to an invalid value, e.g. entering a number:
develop:sitescript. The development server should crash and show an error:
It’s working correctly, great! Go back to
site/gatsby-config.tsand use the correct
Joi as a validation library is really powerful and as shown in Configuring Plugin Usage with Plugin Options you can define a really fine-grained validation schema.
Awesome! You’ve learned a lot about plugin options.
Take a moment to think back on what you’ve learned so far. Challenge yourself to answer the following questions from memory:
- How can you use the
reporterAPI and what methods does it have?
- What is the difference between the methods
- How can you show an activity for e.g. the time it takes to source data?
- What are the advantages of custom errors defined through
- Which type of storage is the
cacheAPI and for which purposes is it useful?
- How can you access plugin options?
- How can you validate that the user input for plugin options is correct?
- Use the
reporterAPI and its methods to output information (warnings, errors, logs, timers, etc.) to the terminal during
gatsby build. You can also stop the entire Gatsby process on errors if necessary.
- By default, a generic error handler is used. You can use the
setErrorMapAPI to define your own, custom errors to show richer information to your users.
cacheAPI gives you access to a key-value store to save data between runs.
- Use the
pluginOptionsSchemaAPI to validate the input of your plugin options.
Share Your Feedback!
Our goal is for this tutorial to be helpful and easy to follow. We’d love to hear your feedback about what you liked or didn’t like about this part of the tutorial.
Use the “Was this doc helpful to you?” form at the bottom of this page to let us know what worked well and what we can improve.
In Part 5 you’ll learn all about Incremental Builds and how to best build your API and plugin for it.Continue to Part 5