Community Plugin

Gatsby cdn search plugin

This plugin is in beta and still in work

Give us any feedback, open issues for any questions or ideas

Mongo query compatible search plugin for gatsby.

It is a no cost way to add search to your site.

Key technology is http2/http3 protocols, CDN, mongo-like query and N-GRAM search.

The plugin supports mongo-like query syntax with custom n-gram search.

Idea of this plugin is simple.

calculated indices in build phase of gatsby apps. (n-gram, text-lex, simple)
split the indices by chunk
create range diapason indices as “table of contents” the chunk
save all chunk and “table of contents” on CDN as assets
in runtime plugin restore indices and efficiently on-demand loaded chunk over http2 protocol
http2 multiplexing multiple requests over a single TCP connection.

The plugin has native support React via Hook “useCdnCursorQuery”.

Also, you can trace your request with log-level.

import { useCdnCursorQuery, log } from 'gatsby-cdn-search-plugin'
log.enableAll(); // full logging

Live demo

Kaggle dataset “Used Car Auction Prices” 500 000 row

live demo

source code on github

source of data

Plugins config

    plugins = [
    'gatsby-plugin-offline',
    {
      resolve: require.resolve("./cdn-indice-plugin"),
      options: {
        id: 'cars',
        chunkSize: 6000,
        dataChunkSize: 60,
        indices: [  
          {
              id: 'model',
              column: 'model',
              type: 'simple', 
          },
          { id: 'make', column: 'make' },
          { id: 'year', column: 'year' },
          { id: 'state', column: 'state' },
          { 
              id: 'id-state',
              column: 'state',
              algoritm: 'english',
              type: 'text-lex' 
          },
          { 
              id: 'ngram',
              type: "n-gram",
              actuationLimit: 1,
              actuationLimitAuto: false, 
              gramLen: 3, 
              toLowcase: true, 
              algoritm: 'english',
              stopWords: ["and"],
              columns: ['model', 'make', 'color'] 
          }
        ],
        idAttr: 'id',
        normalizer: ({ data }) => { 
          return data.recentCars
            .map(( {id, ...node} ) => ({ id: id.replace('Car__',''), ...node }));
        },
        graphQL: `query MyQuery { 
          recentCars(cursor: 0, limit: 500000){
              id
              color
              make
              mmr
              model
              seller
              sellingprice
              state
              transmission
              trim
              vin
              year
        }
      }`
      }
    }
    ]

Usage stateful React hook

import { useCdnCursorStatelessQuery, log } from 'gatsby-cdn-search-plugin'
log.enableAll(); // full logging 


const makeQuery = (search) => { // Different query strategy. It is depends of length search word
  if (search.length >= 4) {
    return { $ngram: search }; // n-gram 
  } else if (!!search.length) {
    return {
      $or: [
        { model: { $regex: new RegExp(`^${search}`, 'i'), }, }, // regexp by two columns
        { make: { $regex: new RegExp(`^${search}`, 'i'), }, }
      ],
    };
  } else {
    return undefined;
  }
}
const [state, dispatch] = useReducer(reducer, initialState);

const query = useMemo(() => makeQuery(state.search), [state.search]);

const {hasNext, next, fetching, all, page} = useCdnCursorStatefulQuery('cars', query, {year: 1}, 0, 30); // hook return cursor of data

const load = useMemo(() => {
    if (hasNext && !fetching) {
          next(); // load next slice of data 
    }}, [hasNext, fetching, next]);

Usage stateless React hook (more complicated)

import { useCdnCursorStatelessQuery, log } from 'gatsby-cdn-search-plugin'
log.enableAll(); // full logging 

const initialState = {
    loading: false,
    search: '',
    list: [],
    page: 0,
};

function reducer(state, action) {
    switch (action.type) {
        case 'pageUp':
            return { ...state, page: state.page + 1 }
        case 'type':
            return { ...state, search: action.value }
        case 'loading':
            return { ...state, loading: true }
        case 'load':
            return { ...state, loading: false, list: action.list, page: 0 };
        case 'indice':
            return { ...state, indice: action.value }
        case 'loadMore':
            return { ...state, loading: false, list: [...state.list, ...action.list] };
        default:
            throw new Error();
    }
}

const makeQuery = (search) => { // Different query strategy. It is depends of length search word
  if (search.length >= 4) {
    return { $ngram: search, year: { $lte: 2014 } }; // n-gram and  year <= 2014
  } else if (!!search.length) {
    return {
      $or: [
        { model: { $regex: new RegExp(`^${search}`, 'i'), }, }, // regexp by two columns
        { make: { $regex: new RegExp(`^${search}`, 'i'), }, }
      ],
    };
  } else {
    return { year: { $lte: 2014 } }; // only date predicate
  }
}
const [state, dispatch] = useReducer(reducer, initialState);

const query = useMemo(() => makeQuery(state.search), [state.search]);

const cursor = useCdnCursorQuery('cars', query, {year: 1}, 0, 30); // hook return cursor of data

  useEffect(() => {
    (async () => {
        let list = await cursor.next(); // load first slice of data
        dispatch({ type: 'load', list });
    })();
  }, [state.search, cursor])

  useEffect(() => {
    (async () => {
      if (await cursor.hasNext()) {
        let list = await cursor.next(); // load next slice of data 
        dispatch({ type: 'loadMore', list })
      }
    })()
  }, [state.page]);

Usage find api exactly


      import { restoreDb } from 'gatsby-cdn-search-plugin'

      const db = await restoreDb('cars');
      let result;
      if (search.length >= 4) {
        result = await db.find({ $ngram: search, year: { $gte: 2014 } }, undefined, 0, offset);
      } else if (!!search.length) {
        result = await db.find({ model: { $regex: new RegExp(`^${search}`, 'i'), }, year: { $gte: 2014 } }, undefined, 0, offset);
      } else {
        result = await db.find({ year: { $gte: 2014 } }, undefined, 0, offset);
      }

Usage cursor api exactly


      import { restoreDb } from 'gatsby-cdn-search-plugin'

      let cursor;
      const searchFetch = async (search, skip = 0, limit = 30) => {
        const db = await restoreDb('cars');
        if(cursor){
          cursor.finish();
        }
        if (search.length >= 4) {
          cursor =  db.cursor({ $ngram: search, year: { $gte: 2014 } }, undefined, skip, limit);
        } else if (!!search.length) {
          cursor =  db.cursor({
            $or: [
              { model: { $regex: new RegExp(`^${search}`, 'i'), }, },
              { make: { $regex: new RegExp(`^${search}`, 'i'), }, }
            ],
          }, undefined, skip, limit);
        } else {
          cursor = db.cursor({ year: { $gte: 2014 } }, undefined, skip, limit);
        }
        return await cursor.next();
      }

Plugin options

Options name	Type	Required	Default value	Description
id	String	True	None	Unique id database collection. The first parameter in React Hook useCdnCursorQuery.
chunkSize	Number	False	500	Indices chunk size. This affects the number of indices files. You should select this parameters depends of size of your collection
dataChunkSize	Number	False	25	Data chunk size. This affects the number of data files.
idAttr	String	True	None	Primary id attribute name.
normalizer	Function({data: any}): Row[]	True	None	It is callback for handle data from graphQl
graphQL	graphQL string	True	None	Graphql query for fetching data
indices	Array<Union<NgramIndicesOption ,TextLexIndicesOption ,SimpleIndicesOption>>	True	None	Secondary indices. Add column available to search.

NgramIndicesOption

Options name	Type	Required	Default value	Description
type	“n-gram”	True	simple	Indices using the N-Gram algorithm for search with typos. Also, support “The Porter Stemming Algorithm” for zipping indices. You can search by this column with specific not Mongo predicate. {$ngram: “search”}
id	String	True	None	Unique id of indices. Uses as operator name in the search. Query example for id “ngram” is {$ngram: “search”}
actuationLimit	Number	True	None	Minimum match n-gram in search. [color] -> [col, olo, lor]
actuationLimitAuto	Boolean	False	False	If option equal true option actuationLimit doesn’t work. Actuation limit n-gram calculates auto by the size of the search word.
gramLen: 3	Number	False	3	Size of. Example if n-gram equal 3 “color” was split to “col”, “olo”, “lor”
toLowcase	Boolean	False	False	Case sensitive search
stem	String	False	None	Preprocess indices with “The Porter Stemming Algorithm”. Available values ‘english’, ‘russian’, …
columns	String[]	True	None	Indexing columns
stopWords	String[]	False	None	The words exclude for search

SimpleIndicesOption

Options name	Type	Required	Default value	Description
type	“text-lex”	True	simple	The Porter Stemming Algorithm. You can search by this column with specific not Mongo predicate. {$lex: “search”}
id	String	True	None	Unique id of indices. Uses as operator name in the search. Query example for id “ngram” is {$lex: “search”}
algoritm	String	False	None	Preprocess indices with “The Porter Stemming Algorithm”. Available values ‘english’, ‘russian’, …
column	String	True	None	Indexing column

TextLexIndicesOption

Options name	Type	Required	Default value	Description
type	“simple”	True	“simple”	Default value simple indices by one column only. You can search by this column with regular Mongo predicate lt, gt, eq, … etc. Also, support regexp by “start with regexp”
id	String	True	None	Unique id of indices. Uses as operator name in the search. Query example for id “ngram” is {$lex: “search”}
column	String	True	None	Indexing column