Syncing RxDB changes from a graphql server in Rescript

Introduction

This is part of an ongoing series about the Rescript programming language. I’ve been taking notes as I build a toy “Recipe Book” app to explore the Rescript ecosystem. The toy app has evolved into a fairly complete offline app at this point, and I’ve also written a graphql server in rescript and express.

In this article, I’m (finally) going to marry those two projects. If all goes well , we’ll end up with an offline-enabled frontend that can sync with the graphql powered backend in realtime. Wish me luck!

Patreon

This series takes a lot of time and dedication to write and maintain. The main thing that has kept me invested in writing this series is my supporters on Patreon.

As a bonus for patrons, articles are published there at least two weeks before they make it to my public blog. Also, if I get enough Patrons, my editor has promised to publish a book on Rescript!

Other ways to show support include sharing the articles on social media, commenting, or a quick thank you on the Rescript forum.

Other articles in series

With over a dozen articles and counting, I’ve created a table of contents listing all the articles in this series in reading order.

Let’s begin

This article references two repositories that I’ve been developing in earlier articles. The server is in the rescript-express-recipes. You can git checkout the mutations branch if you want to start from the same place I am in this article. This article makes one change (to adjust the http listen port) on the frontend-hookup branch.

Most of our edits will happen in the rescript-offline repository. You can start from the components branch.

If you want to follow along with the changes in this article, I try to make a separate commit for each section. You can find them in the syncing branch.

Install the GraphQL RxDB plugin

The RxDB docs for graphql replication tell us how to set up replication to a GraphQL endpoint. We’ve already implemented those endpoints, so the majority of our work is to model the replication-graphql library in Rescript.

The first thing we’ll need to do, then, is import that library. RxDBReplicationGraphQLPlugin is shipped with RxDB, so we don’t need to add any dependencies. However, it’s not built into the library. It needs to be installed as a library.

We’ve already modeled the addRxPlugin function in Db.res. Let’s put the RxDBReplicationGraphQLPlugin definition in a new file named Sync.res:

@module("rxdb/plugins/replication-graphql")
external graphqlReplicationPlugin: 'plugin = "RxDBReplicationGraphQLPlugin"

Then in Db.res you can call addRxPlugin(Sync.graphqlReplicationPlugin) right after the other call that adds pouchDbAdapter.

Update the schema to match the server

We added updatedAt and deleted fields to our DB model on the express server in an earlier article. Now we obviously need to make the Model on our frontend have the same format.

Or do we?

Truth is, I wrote the first version of this article all wrong because I assumed those two fields were needed. But it turned out not to work because RxDB manages the deleted field for you. So you need it on the server, but you actually only need to add updatedAt to the frontend models.

Update Model.res as follows:

type id = string
type title = string
type ingredients = string
type instructions = string
type tag = string

type recipe = {
  id: id,
  title: title,
  ingredients: ingredients,
  instructions: instructions,
  tags: array<tag>,
  updatedAt: float,
}

type taggedRecipes = {
  tag: tag,
  recipes: array<id>,
  updatedAt: float,
}

Of course, you’ll need to update all the places that field is missing in the code as well. You can just let the compiler be your guide, as we did in the earlier “Quickly refactoring” article. There are only two places that need to change:

• The addRecipe call in the button click event in AddRecipeForm.res
• The tagRecord definition in addTagCallback in App.res

Just add this line in both locations and the compiler should be happy:

        updatedAt: Js.Date.now(),

The compiler will not, however, catch the fact that you need to update the schema.json to reflect this change as well. Add the following to the properties in schema.json for both collections (recipe and taggedRecipes):

      "updatedAt": {
        "type": "float"
      },

Pull replication

The GraphQL RxDB plugin requires a function that accepts the most recently known document in that collection and constructs a GraphQL query to request the actual data. Let’s define the type for that function in Sync.res:

type queryParams = {
  query: string,
  variables: {"id": Model.id, "minUpdatedAt": float, "limit": int},
}

type queryBuilder<'document> = option<'document> => queryParams

The queryParams type is interesting because via my “pragmatism take all” philosophy, I decided the variables didn’t need to be generically typed. Any query returned by one of my queryBuilders will accept variables in this format. That will not be true for all RxDB users, but it only has to be true for this application.

The queryBuilder type is a function and has a generic type for the document that is passed into it. The document is passed in as an option because the first time the query is made, RxDB will pass undefined. The function must always return a queryParams type with query and variables field.

Now implement a specific version of that for the recipe query. Start with the function signature:

let recipeQueryBuilder: queryBuilder<Model.recipe> = recipeOption => {
}

Then you need to define the GraphQL query itself. This needs to match the schema we defined in the graphql server in the earlier articles. I basically copied the example graphql I used in that article and parameterized it, as follows:

  let query = `
    query Query($id: String!, $minUpdatedAt: Float!, $limit: Int!) {
      recipeRxDbFeed(id: $id, minUpdatedAt: $minUpdatedAt, limit: $limit) {
        id
        title
        ingredients
        instructions
        deleted
        tags
        updatedAt
      }
    }
  `

Next, define the variables for this graphql query. Depending on whether the recipeOption is valid or not, this will be either a default value or be populated from fields on the Model.recipe that was passed into the function:

  let variables = switch recipeOption {
  | Some(recipe) => {
      "id": recipe.id,
      "minUpdatedAt": recipe.updatedAt,
      "limit": 5,
    }
  | None => {
      "id": "",
      "minUpdatedAt": 0.0,
      "limit": 5,
    }
  }

Finally, we construct a queryParams instance as the last line in the function, (which implicitly returns it):

  {
    query: query,
    variables: variables,
  }

Modeling the syncGraphQL function

Installing the GraphQL replication plugin adds a new syncGraphQL method to each collection. We need to model that method in Rescript in order to be able to call it.

This method accepts an object with some configuration keys. As I usually do, I’ll only model the ones I need. This is an especially useful attitude to adopt when working with RxDB, which doesn’t have much in the way of API documentation! I’m going to model these parameters as a Rescript record. Add a new type to Sync.res as follows:

type pullOptions<'document> = {queryBuilder: queryBuilder<'document>}

type syncGraphQLOptions<'document> = {
  url: string,
  pull: pullOptions<'document>,
  deletedFlag: string,
  live: bool,
}

syncGraphQLOptions is parameterized on the type of the document. This will help us later when we pass it into the collection, as you’ll only be able to use it on collections that have a matching document type. In other words, you can’t pull taggedRecipes into the recipes collection.

Now we have everything we need to model the syncGraphQL method itself. Open up Db.res and find the RxCollection module. The new function can match the existing @send declarations as follows:

  @send
  external syncGraphQL: (t<'docType>, Sync.syncGraphQLOptions<'docType>) => unit = "syncGraphQL"

Now we can actually call this function in App.res to see if it can actually sync something. Put this before the two subscribeAll calls in the Apps make function:

      db.recipes->Db.RxCollection.syncGraphQL({
        url: "http://localhost:3001/graphql",
        pull: {queryBuilder: Sync.recipeQueryBuilder},
        deletedFlag: "deleted",
        live: true,
      })

Test it out

Warning: We’re about to discover a bug via runtime error. Rescript is 100% type safe, but only assuming you accurately model the Javascript functions you are binding to. I made a mistake above, and I decided to leave it in to illustrate that even guaranteed type safety is not… guaranteed.

Before we can test this, we’ll need to correct a minor oversight in the rescript-express-recipes server: It connects on the same port that the CreateReactApp devserver does! Open Index.res in the server code and change the port to 3001.

Also before testing, open the Application tab in your chrome dev tools and delete the three _pouch_recipes-rxdb* databases under the IndexedDB accordion. That way, you can start with a clean slate to see the syncing in action.

Now start the express server in one terminal tab and the react app in the other one. Connect to http://localhost:3000/ and wait for the error message.

The error I get is Unhandled Rejection: Cannot read property 'id' of null.

You’ve probably seen this error before in Javascript. It usually means you checked something against undefined instead of against null, and then received a null value. I heard a story once that the inventor of null (A very smart man named Tony Hoare) called it a “billion dollar mistake”. Javascript has a long history of repeating mistakes, so it includes not one, but two “null-like” values. And it has been much more than a two billion dollar mistake.

I digress. Back to the error message. The problem is that I used Rescript’s option type for the value passed into the queryBuilder. An option is compiled in Javascript to either undefined or a legitimate value. But RxDB is passing null, which Rescript is interpreting as a legitimate value.

Ugh. So much for type safety.

The solution is to replace our use of option<'document> in the queryBuilder definition (in Sync.res) with a Js.Nullable.t<'document>. Then, the switch statement needs to convert that Nullable value to an option before switching on it, as follows:

  let variables = switch recipeOption->Js.Nullable.toOption {
  ...
  }

Start the server again and…

Cursed CORS!

Have a look in the console, and you should see a long list of requests being rejected because it is a cross-origin request.

I forgot that one can’t simply connect to localhost:3001 from localhost:3000. My general approach when dealing with CORS issues is to search stack overflow a lot and try things until it works. It’s marginally easier than understanding what on earth is going on under the hood.

In this case, the solution is as easy as adding proxy: http://localhost:3001 to the package.json. This tells React to proxy unknown requests to that location. (Of course, you’d have to do something different in prod). Then you have to edit the syncGraphQL call so that its url is /graphqlinstead ofhttp://localhost:3001/graphql`.

You’ll need to kill and restart the react server before it can pickup this change.

Test again

With the react dev server and the express app both running, visit the graphiql UI in the express server at http://localhost:3001/graphql.

Run a mutation such as the following:

mutation {
  setRecipe(recip: {
    id: "abc",
    title: "Bread",
    ingredients: "flour, water",
    instructions: "mix and bake",
    tags: ["carbs"],
    deleted:false
  }) {
    id
    updatedAt
  }
}

Wait fifteen seconds, then visit http://localhost:3000/recipes/abc. The recipe you just added to the graphql database should now be rendering!

Better yet, visit http://localhost:3000/recipes/def in one browser window before adding anything to the server using graphql. It will say Recipe with id def is not in your database. Now leave that window visible and open graphiql in a new window. Run a query similar to the above except with the id set to "def".

It might take a few seconds for the change to propogate (it’s polling), but the React page should eventually automatically refresh to show the recipe you just added!

Congratulations, you are half way to syncing! We’ll do push and subscriptions in a later article. But let’s repeat the process above to sync tags as well.

Adding the Tagged Recipes feed

You might want to take a stab at doing this part yourself, since it’s very similar to the recipes version. You don’t have to model any functions since we already did that using generics.

Add a new taggedRecipesQueryBuilder to the Sync.res as follows (It’s basically a copy paste of the previous query builder, with recipe replaced with taggedRecipes, and different fields being queried):

let taggedRecipesQueryBuilder: queryBuilder<Model.taggedRecipes> = recipeOption => {
  let query = `
    query Query($id: String!, $minUpdatedAt: Float!, $limit: Int!) {
      taggedRecipesRxDbFeed(tag: $id, minUpdatedAt: $minUpdatedAt, limit: $limit) {
        tag
        recipes
        updatedAt
        deleted
      }
    }
  `

  let variables = switch recipeOption->Js.Nullable.toOption {
  | Some(taggedRecipes) => {
      "id": taggedRecipes.tag,
      "minUpdatedAt": taggedRecipes.updatedAt,
      "limit": 5,
    }
  | None => {
      "id": "",
      "minUpdatedAt": 0.0,
      "limit": 5,
    }
  }

  {
    query: query,
    variables: variables,
  }
}

Then you’ll need to add the syncGraphQL call to App.res alongside the existing one in the make constructor’s useEffect3:

      db.tags->Db.RxCollection.syncGraphQL({
        url: "/graphql",
        pull: {queryBuilder: Sync.taggedRecipesQueryBuilder},
        deletedFlag: "deleted",
        live: true,
      })

Navigate to the Tags tab. Then run a query like this in the graphiql builder:

mutation setTaggedRecipes {
  setTaggedRecipes(taggedRecipes: {
    tag: "carbs",
    recipes: ["abc", "def"],
    deleted:false
  }) {
    tag
    updatedAt
  }  
}

The two tags should show up shortly!

Conclusion

I’ll be honest: I can’t believe it worked. Granted, I was working on this all weekend, but the polling endpoint seems to be fully functional now. The react app will even update when you add values by posting to the REST endpoints!

In the next article, we’ll hook up the push side of things. I am expecting a couple problems around RxDB’s special-case handling of the deleted flag.

After that, I’m planning to write something about unit tests. Stay tuned!