Modeling RxDB in Rescript, Part 2

Introduction

This is part of an ongoing series about the Rescript programming language. I’m taking notes as I build a toy “Recipe Book” app to explore the Rescript ecosystem.

In my most recent article, I started exploring the world off offline-enabled apps using RxDB. This project does not have Rescript bindings, so my first step is to model the library in Rescript.

The previous article modelled the database connection and adding collections. This one will focus on modelling the individual collections and making queries against them.

Patreon

This series takes a lot of time to write and maintain. I was originally working solid 8 hour days on these articles for a few weeks to get the quantity and quality of content you are reading. Since my new job started, it’s been much harder to stay motivated to work on this in my much more limited free time.

I’ve set up a Patreon account so people can help provide extra incentive. As a bonus for patrons, articles will be published there at least two weeks before they make it to my public blog.

Now that I have a few patrons, I can honestly say that it has been a huge motivation boost. Thank you so much for your appreciation! I am trying to maintain my schedule of one article per week for you all, but I’ve had to dig into my backlog a bit as I ramp up in my new role. My more recent articles are necessarily a bit shorter, but I’m trying very hard to put up new content every week.

Other ways to show support include sharing the articles on social media, commenting, or a quick thank you on the Rescript forum.

Other articles in series

With over a dozen articles and counting, I’ve created a table of contents listing all the articles in this series in reading order.

Let’s begin

If you haven’t been following along with these articles, you can git checkout the db-connection branch of the accompanying github repository to start at the same point I am.

I make commits to the modelling-collections branch that roughly map to the sections in this article. You may find it instructive to see the diff of the JS that the rescript compiler outputs with each change in your rescript code.

RxCollection Boilerplate

We were able to add collections with a schema in the previous article, but we don’t yet have access to those collections in Rescript. We’ll need to address that before we can query them. There are numerous methods on the RxCollection class, but as with RxDatabase, I’m only going to bother binding to the ones we will actually use.

Create a new RxCollection module. There are two ways to do this. One would be to add a RxCollection.res file. For now, though, I’m just going to put a new module in Db.res (just after the imports, since the RxDatabase methods will need to know it exists to access it):

module RxCollection = {

}

This module requires a t type for the underlying RxCollection. Each collection stores documents with a specific schema, so we will make t generic on a specific type.

module RxCollection = {
  type t<'docType>
}

For our use, 'docType will either be a recipe or array of titles. But we’ll stick with a generic for now.

Now we can start modelling the methods on the Javascript object using Rescript functions. Let’s model insert first. It returns a promise with a dictionary of the collections. We can use the same @send external structure we used for RxDatabase methods:

  @send external insert: (t<'docType>, 'doctype) => Promise.t<RxDocument.t<'docType>> = "insert"

I expected insert to return the document that was passed it, but it actually returns a (Promise of an) RxDocument. RxDocument is a wrapper around a value that has extra logic for reactively observing changes to the document.

We’ll need to model that class before the insert function will compile. Once again, we’ll model the bare minimum to get it to compile. We can add the useful observing and querying features after we have something end to end building. Note that this new module needs to go before RxCollection in the Db.res file. Otherwise RxCollection won’t be able to find it:

module RxDocument = {
  type t<'docType>
}

Now that we have modelled the RxCollection typ, we can also update the addCollections external declaration. In the previous article, we just assigned a throwaway generic 'collections' type on it. Now we can specialize it a bit to indicate that the return value is a "promise of a dict of a collection of a *something*". The "something" will still be a generic, which I named ‘docType` in the following example:

@send
external addCollections: (
  t,
  Js.Dict.t<addCollectionsOptions<'schema>>,
) => Promise.t<Js.Dict.t<RxCollection.t<'docType>>> = "addCollections"

This isn’t as type-safe as it could be, for a couple reasons:

• It doesn’t capture that the keys of the input dictionary will be the keys of the output dictionary
• It doesn’t capture that specific keys on the output dictionary will have specific types in their values

I’m not sure if Rescript is able to model that or not, but I’m not going to worry about it. For one thing, I don’t actually intend to access this particular function from client models. More importantly, I intend to write typesafe wrappers of RxDatabase that actually will provide typesafe values.

Before we continue, we can test this new RxCollection.insert API. I’m not going to commit this change, but, for testing, I replaced the db->addCollections(options)... line in the Db.make function with the following (crazy) pipeline:

  db
  ->addCollections(options)
  ->Promise.then(collections => {
    collections
    ->Js.Dict.unsafeGet("recipes")
    ->RxCollection.insert({
      Model.id: "some-cool-unique-id",
      title: "Bread",
      ingredients: "flour, water, salt, yeast",
      instructions: "mix, rise, bake",
      tags: [],
    })
  })
  ->Promise.map(doc => {
    Js.log(doc)
    db
  })

This took quite a bit of time to get right, and I’m lamenting the current lack of async/await support in Rescript. It’s fairly easy to read as a pipeline, but it’s harder to write! In words:

• take the db and pipe it to addCollections
• pipe the resulting promise to Promise.then with a callback that:
  • extracts the recipes from the dict collection with unsafeGet. I am comfortable using unsafeGet only because this is testing code I am going to throw away shortly. However, it’s probably ok to use it here anyway, since I know that I passed recipes into addCollections.
  • pipes the resulting collection to RxCollection.insert
  • RxCollection.insert returns a promise, and that promise gets returned from the callback
• pipe the promise returned by RxCollection.insert to a Promise.map call that logs the resulting doc and returns the db
• The resulting mapped promise is returned from the make function. Since it is a promise of a Db.t, it is the correct return value.

If you run the above code twice, the second time it will show an error in the console because of an update conflict. You can`t insert a document twice (there is a separate upsert mechanism). This, combined with the logging output, indicates that we are inserting documents into the db successfully.

But I actually don’t intend to use the collections returned by addCollections like this. So just revert the above pipeline after confirming it works.

Adding accessors for individual collections

Collections in RxDb are accessible as properties on the RxDb object itself, indexed by the name used when they were created. The RxDB docs give the following example for accessing a collection from Javascript:

const collections = await db.addCollections({
  heroes: {
    schema: mySchema,
  },
});
const collection2 = db.heroes;
console.log(collections.heroes === collection2); //> true

Now to map that to Rescript. Our database has two collections, recipes and tags and we need the compiled JS to access them on the db object as the two properties db.recipes and db.tags.

We can model access to these either by treating the database as a Record, or by providing a @get accessor function. Let’s try the former. Replace the type t at the top level in Db with:

type t = {
  recipes: RxCollection.t<Model.recipe>,
  tags: RxCollection.t<array<Model.title>>,
}

This provides accessors to the two collections and guarantees they will be type-checked correctly every time they are accessed.

We can test that the accessors work from our harness code in App.res. Inside the useEffect promise where you get the result of the db, add the following:

let _ =
  db.recipes
  ->Db.RxCollection.insert({
    Model.id: "some-cool-unique-id-2",
    title: "Bread",
    ingredients: "flour, water, salt, yeast",
    instructions: "mix, rise, bake",
    tags: [],
  })
  ->Promise.map(doc => {Js.log(doc)})

Again, this should show an error in the console if you reload the page twice, because the insert was successful the first time. So revert this change, since we don’t want to see that error every time we test something!

One thing to note here is that we have tightly coupled our bindings to our code. We know exactly which collections exist in the db, and what types they return.

In contrast, if we were trying to make a generic RxDB wrapper (for example, as a third party module on npm), we’d have to find some other way to allow the API consumer to generically but strongly type the collections. I have been surprised how often I prefer to just use my own bindings instead of finding a third-party library that forces me to jump through hoops to specify generics correctly.

Note: Another way to check that the insert queries are actually working is to look at the Application tab in the Chrome devtools. Find IndexedDB in the left-hand column and poke around in it until you find the data. Here’s how mine looks after a bit of testing to add a few objects:

Binding to RxQuery to get all objects

The RxCollection.find method is extremely dynamic. It can accept a variety of different arguments with different formats and structures. Worse, it’s poorly documented, with an implicit assumption that you understand some arcane variant of the mongo query syntax. And the permitted field names all depend on what type of object is being queried in the first place.

Added together, this is a recipe for a typechecking nightmare. My solution: don’t check types overly rigorously. Pragmatism beats pedantry every time.

RxCollection.find returns a RxDocument, so let’s model that first:

module RxDocument = {
  type t<'docType>
}

by this time, adding an external to call the find method to the RxCollection module should (finally) feel straightforward:

@send external find: t<'docType> => RxQuery.t<'docType> = "find"

Now, this is the version of find that accepts no arguments (besides the collection itself). By default, RxCollection.find returns a query for all items in the collection. We can model that first before we get carried away with specialized querying.

For testing, I added the following line to my App.res:

db.recipes->Db.RxCollection.find->Js.log

It is showing an RxQuery object in my console, so everything seems in order. But I feel like it’s overly verbose with the call to RxCollection in there. For API cleanliness, we can add an accessor for find to the end of Db.res:

let find = collection => RxCollection.find(collection)

This is just a function on the Db module that quietly passes its argument to the RxCollection.find function. The interesting thing about this is that I did not have to write out any types, not even the generic types. Rescript is able to infer that the new Db.find can only accept RxCollection.t from the fact that we are calling a method that accepts one. I’m in awe that they are able to provide this level of type inferencing in such an incredibly fast compiler.

Now the earlier call in App.res can be simplified to db.recipes->Db.find->Js.log. The recipes collection gets piped to Db.find which passes it to Db.RxCollection.find under the hood. The returned RxQuery gets piped into Js.log.

RxQuery doesn’t actually contain the returned values, though; it’s just a builder for a query. To actually return the values, we need to model the RxQuery.exec method, which returns a promise. Put this external in the RxQuery module:

@send external exec: t<'docType> => Promise.t<array<RxDocument.t<'docType>>> = "exec"

Read that twice to make sure you understand what’s going on.

It’s probably not worth adding an accessor to Db.res like we did for find, but I did it anyway:

let exec = query => RxQuery.exec(query)

Now my test pipeline in App.res looks like this:

let _ = db.recipes->Db.find->Db.exec->Promise.map(results => results->Js.log)

The rules of the Promise API demand that I bind the result to a variable. I use _ because I don’t care about the resulting promise. I pipe the result of exec into Promise.map with a callback that logs the result. The return value of the Js.log call (which is unit) is wrapped in a promise and returned from map.

Now we have an array of RxDocument.ts. I want to access the underlying recipe objects inside those documents. This sounds hard, if you have a look at the RxDocument functions. There’s certainly no getRecipe field on RxDocument, or even a getItem. You can read individual fields with get, but extracting that into a recipe type feels like a lot of verbose code for no benefit.

Luckily, there’s a cheat code. RxDocument has toJSON, which gives us an object with all the fields set. We just need to figure out how to bind that to a recipe. Try adding this to the RxDocument module:

@send external recipe: t<Model.recipe> => Model.recipe = "toJSON"

Now the test code to log the recipes (I have four recipes in my DB now, from various iterations of testing the insert mechanism) looks like this:

let _ =
  db.recipes
  ->Db.find
  ->Db.exec
  ->Promise.map(results =>
    results->Belt.Array.map(recipe => recipe->Db.RxDocument.recipe)->Js.log
  )

If you run the code now, you should see whatever recipes were added to your database in earlier testing phases. They’ll be printed out in an array.

Did you notice a problem with the above? What happens if the requested document does not actually contain recipes?

The only other type of document we have so far is tagged recipes, which are returned as an array of titles (strings). Let’s see what happens if we copy the previous example and change db.recipes to db.tags like this:

let _ =
  db.tags
  ->Db.find
  ->Db.exec
  ->Promise.map(results => results->Belt.Array.map(tag => tag->Db.RxDocument.recipe)->Js.log)

This is trying to extract a recipe from a collection of tags. That’s no good! The good news is that this will not compile. It is not possible to accidentally query our tags collection and assign the result to a Model.recipe type. Type safety is amazing.

The bad news is that we don’t yet have any way to query our tags collection and assign the result to an array<Model.title> type. That is easily rectified with one more external in the RxDocument module:

@send external taggedRecipes: t<Model.taggedRecipes> => Model.taggedRecipes = "toJSON"

This may be somewhat surprising. We’ve now bound two completely different variables with completely different types to the same toJSON method. That’s totally legit Rescript. In fact, it’s recommended in the docs, along with a few other ways to model such polymorphic functions.

Make the find function able to query by selector

RxDB gives two syntaxes to query for specific instances, and both are poorly documented and rather dynamic.

The first is to use mongodb-style queries like this (in Javascript):

db.recipes
  .find({ selector: { title: "Bread" } })
  .exec()
  .then((a) => a.map((r) => r.toJSON()))
  .then(console.log);

Here’s a tip: I wanted to test the above query without modelling anything in Rescript. I just wrapped it in a %raw() tag like this (in App.res where I was testing the other queries):

%raw(`
db.recipes
  .find({ selector: { title: "Bread" } })
  .exec()
  .then((a) => a.map((r) => r.toJSON()))
  .then(console.log);
`)

This is legal Rescript code, and a perfectly acceptable way to write Rescript if you have any highly dynamic or just downright confusing JS code to model. You’re sacrificing type safety for something that may or may not be more important, depending on the project: development velocity.

The second way to query RxDb is to use chained queries like this (you can also test this in %raw()):

db.recipes
  .find()
  .where("title")
  .eq("Bread")
  .exec()
  .then((a) => a.map((r) => r.toJSON()))
  .then(console.log);

Personally, I find the chained version easier to read, especially if I imagine mapping them to pipe-first syntax in Rescript. It would require modelling quite a few functions (see mquery), but if we only choose to model the ones we need, it won’t be too bad.

On the other hand, for the selector version, we can just tell find to accept an arbitrary object as an argument, and leave it up to the API consumer to guarantee that it is a valid query. This is very easy to model, but with very little compile-time safety.

I like the sounds of “easy to model”, so let’s try it. All we have to do is replace the find external declaration with one that accepts an option<Js.t<'options>> where Js.t<'options> is the type of an object with unknown properties:

@send external find: (t<'docType>, option<Js.t<'options>>) => RxQuery.t<'docType> = "find"

Of course, this breaks the query we had earlier, which did not pass any arguments in to find (other than the piped recipes collection). However, the following query now works fine:

let _ =
  db.recipes
  ->Db.find(None)
  ->Db.exec
  ->Promise.map(results =>
    results->Belt.Array.map(recipe => recipe->Db.RxDocument.recipe)->Js.log
  )

More importantly, so does this one, which allows us to query by title:

let _ =
  db.recipes
  ->Db.find(Some({"selector": {"title": "Bread"}}))
  ->Db.exec
  ->Promise.map(results =>
    results->Belt.Array.map(recipe => recipe->Db.RxDocument.recipe)->Js.log
  )

Any other set of query options can be passed inside the option without any further modelling on our part. Yes, it isn’t type-safe, but I think it’s fine for queries. Queries are the kind of thing that either work or they don’t, so a bit of intelligent unit testing will serve the same purpose as rigorous type checking, and will take a minimal amount of time.

Two find APIs

I don’t like the look of either of the two previous queries because the option looks awkward. Instead of an option, let’s provide two different bindings to the same find method:

@send external find: (t<'docType>, Js.t<'options>) => RxQuery.t<'docType> = "find"
@send external findAll: t<'docType> => RxQuery.t<'docType> = "find"

If you add an extra accessor to Db.res for findAll like this:

let find = (collection, options) => RxCollection.find(collection, options)
let findAll = collection => RxCollection.findAll(collection)

you can now issue these two queries safely:

let _ =
  db.recipes
  ->Db.find({"selector": {"title": "Bread"}})
  ->Db.exec
  ->Promise.map(results =>
    results->Belt.Array.map(recipe => recipe->Db.RxDocument.recipe)->Js.log
  )
let _ =
  db.recipes
  ->Db.findAll
  ->Db.exec
  ->Promise.map(results =>
    results->Belt.Array.map(recipe => recipe->Db.RxDocument.recipe)->Js.log
  )

Conclusion

We are now able to insert elements into collections and retrieve them again, with an appropriate amount of type safety – neither too much nor too little. Modelling most of the other standard query methods should follow naturally.

However, the real benefit of RxDB is its ability to provide subscriptions to state in a reactive manner. In the next article, we’ll cover how to hook up RxDB observers in rescript and connect them to the react components we developed in the earlier series of articles.