Introduction

This is part of an ongoing series about the Rescript programming language. I’m taking notes as I build a toy “Recipe Book” app to explore the Rescript ecosystem.

In my most recent articles, I started investigating the world of offline-enabled apps using RxDB. This project does not have Rescript bindings, so my first step is to model the library in Rescript.

The previous articles modeled making a database connection and issuing queries. This one hooks up reactive queries and connects the database to application state.

This article will be focused a bit less on Rescript and a bit more on RxDB itself. RxDB is a very interesting database and not terribly well documented. So I am hoping it will be useful even if you are not coding in Rescript. I am brand new to RxDB myself (and haven’t been using Rescript all that long either), so this will be pretty introductory stuff.

Patreon

This series takes a lot of time to write and maintain. I’ve been working on these articles 8 hours a day six day a week for a few weeks to craft the quantity and quality of content you are reading. Now that I’m working full time, my future articles will necessarily be a bit shorter. But I am still trying to post once a week.

To be honest, now that my free time is limited again, the only thing keeping me motivated to do this work is that some people find enough value in my work to pay for it. If you’re interested in providing some of that motivation, I’ve set up a Patreon account. I thank you for your support. I am not expecting this to pay for my time investment, but it helps to know if people appreciate it enough to make a contribution.

As a bonus for patrons, articles will be published there at least two weeks before they make it to my public blog (right now there’s more of a six week backlog).

Other ways to show support include sharing the articles on social media, commenting, or a quick thank you on the Rescript forum.

With over a dozen articles and counting, I’ve created a table of contents listing all the articles in this series in reading order.

Let’s begin

If you haven’t been following along with these articles, you can git checkout the modelling-collections branch of the accompanying github repository to start at the same point I am starting.

I make commits to the subscriptions branch that roughly map to the sections in this article. You may find it instructive to see the diff of the JS that the rescript compiler outputs with each change in your rescript code. Or if something is confusing in my content (first, let me know!), the git commit may make it clearer.

Modeling illegal identifiers

We’ve modeled traditional queries in RxDB in previous articles, but have yet to discuss its most interesting queries: observerables.

RxDB follows the ReactiveX programming model, which just happens to be extremely well suited to both React and Rescript.

Pretty much everything in RxDB can be observed in some way: The database, individual collections, individual documents, even fields within the documents. “All we have to do” (I hate that phrase; it usually means something very complicated is following. And it is…) is subscribe to those events and connect them to some state in our React app.

If you are familiar with GraphQL subscriptions, it’s a similar idea. You get notified when certain events occur, and then update your local state or UI state with that provided in the event.

In RxDB, the properties and functions used for observing changes are all named $ or get$ or similar. This is awkward because $ is not a legal variable name in Rescript. Luckily, Rescript does allow us to bind to illegal variables using \"$". This can be useful for binding, but we definitely want to expose something a little more ergonomic.

Instead, we can bind a function to get the value of the $ field. This goes in the Db module somewhere after t has been defined:

module RxObservable = {
  type t
}

@get external observable: t => RxObservable.t = "$"

Now you can use code like this if you have a Db.t type instantiated somewhere (e.g: in the useEffect1 in App.res):

let observable = Db.observable(db)

Modelling subscribe on RxDatabase

On its own, the RxObservable type isn’t that useful. We need to model the subscribe method on it. Using a combination of documentation, source code, and introspection, I determined one possible way to type the function. To be honest, it may not be the best way, but this was a messy bit of research and it’s working well for me. Extend the new RxObservable module as follows:

module RxObservable = {
  type t

  @get external observable: t => RxObservable.t = "$"
  @send external subscribe: (t, 'event => unit) => unit = "subscribe"
}

I tested this by subscribing to the database and inserting a new record:

let observable = Db.observable(db)
Db.RxObservable.subscribe(observable, event => {
  Js.log(event)
  Js.log(event["operation"])
})

let _ =
  Db.RxCollection.insert(
    db.recipes,
    {Model.id: "id3", title: "eggs", ingredients: "eggs", instructions: "eggs",
    tags: []},
  )->Promise.map(_ => Js.log("Inserted"))

The 'event could be improved by further modelling RxChangeEvent. But first, I am more interested in seeing what it looks like to subscribe to individual collections.

Modelling subscriptions on RxCollection

RxCollection has the same $ function as RxDatabase but returns changes to the one collection instead of the whole db. It also has three methods that can listen for one of the three types of changes: insert$, update$, remove$.

By this time, I’m familiar enough with Rescript to model these easily. Add the following bindings to the RxCollection module:

@get external observable: t<'docType> => RxObservable.t = "$"
@get external insertObservable: t<'docType> => RxObservable.t = "insert$"
@get external updateObservable: t<'docType> => RxObservable.t = "update$"
@get external removeObservable: t<'docType> => RxObservable.t = "remove$"

I tested this using similar logic to that used in the previous section, except accessing Db.RxCollection.insertObservable.

Modelling bits of RxChangeEvent

Now that I look at it, the RxCollection observables will always return the same type for a given collection ('doctype). However, a database will have lots of different types, depending on which collection was modified. However can we model this?

There are probably several options using polymorphic variants, but I’m going to do it the easy way. When observing a collection, we supply a specific type, but when observing RxDatabase we leave the type generic. Let’s see if that works.

First, the RxChangeEvent module:

module RxChangeEvent = {
  type t<'docType> = {
    documentData: 'docType,
    id: string,
    operation: string,
    collectionName: string,
  }
}

We specify several properties that we expect the RxChangeEvent.t record to have. Most notably, the documentData has a specific docType.

Then we can update the subscribe binding in the RxObservable module to replace the 'event generic:

module RxObservable = {
  type t<'docType>

  @send external subscribe: (t<'docType>, RxChangeEvent.t<'docType> => unit) => unit = "subscribe"
}

Don’t forget that RxObservable.t also needs to have a generic type, now. Once you do that, the rescript compiler will toss some errors your way. I suggest fixing them yourself as an exercise.

If you get stuck, here are the changes, for completeness. Specify the docType on the four new externals in RxCollection:

@get external observable: t<'docType> => RxObservable.t<'docType> = "$"
@get external insertObservable: t<'docType> => RxObservable.t<'docType> = "insert$"
@get external updateObservable: t<'docType> => RxObservable.t<'docType> = "update$"
@get external removeObservable: t<'docType> => RxObservable.t<'docType> = "remove$"

One place that might trip you up is the 'docType on the rxDatabase external. We can tell Rescript to just leave it generic:

@get external observable: t => RxObservable.t<'docType> = "$"

More concrete RxChangeEvent types with polymorphic variants

We indicated that RxChangeEvent.t.operation is always a string. However, it’s actually more specialized that. In fact, I think (the docs are unclear) it can only be one of three strings: INSERT, UPDATE, and REMOVE. In addition, we know that RxChangeEvent.t.collection can only be recipes or tags, since those are the only ones we defined.

We can use polymorphic variants to model this. Polymorphic variants, unlike normal variants are not associated with a specific type. They compile down to a string, so map well to the common JS idiom of using string identifiers for specifying types.

Note: There has been some discussion around renaming polymorphic variants to something easier to type and pronounce. I’m not sure if a conclusion was reached, but the Rescript docs may one day refer to them as something else.

Here’s how to limit RxChangeEvent to one of three strongly typed strings for the operation and one of two strongly typed strings for the collection name:

module RxChangeEvent = {
  type t<'docType> = {
    documentData: 'docType,
    id: string,
    operation: [#INSERT | #UPDATE | #REMOVE],
    collectionName: [#recipes | #tags],
  }
}

Just to make sure I got the syntax right, I added a conditional like this to my test code in App.res:

if event.operation == #INSERT {
  Js.log("YES")
}

The cool thing is, now it’s not possible to accidentally type, say, if event.operation == "inert".

Observing changes to queries

The above observables and events are cool enough, I guess, but I suspect you are imagining nightmares where you have to listen to all those events and somehow sync them with a redux store or something. We could certainly hook them up to the actions and reducers we already made.

Thankfully, no. The most useful observer in RxDB is actually the one that observes individual queries. Instead of emitting events like the previous two queries, it emits the current query results every time they change.

It’s more or less the same as automatically calling find every time the database changes.

Let’s model subscribe before we explore the ramifications. We need a new observable type, because this one returns an array of RxDocuments instead of a RxChangeEvent. The docs also refer to this kind of query as a BehaviourSubject, but I’ll stick with observable because it feels more consistent.

module RxQueryObservable = {
  type t<'docType>

  @send
  external subscribe: (t<'docType>, array<RxDocument.t<'docType>> => unit) => unit = "subscribe"
}

Then we need to bind to $ on the RxQuery object in order to return an object that has this subscribe method:

module RxQuery = {
  type t<'docType>
  @send external exec: t<'docType> => Promise.t<array<RxDocument.t<'docType>>> = "exec"
  @get external observable: t<'docType> => RxQueryObservable.t<'docType> = "$"
}

Finally, I added a couple lightweight wrappers to my list at the end of Db.res:

let subscribe = (query, subscription) =>
  query->RxQuery.observable->RxQueryObservable.subscribe(subscription)
let subscribeAll = (collection, subscription) => collection->findAll->subscribe(subscription)

The former accepts a query and subscription function and sets up an observable on the query and subscribes to it. The second accepts a collection and subscribes to all changes on it.

These are just a convenience to reduce the piping when actually using this API.

I did some convoluted stuff with a Js.Global.setTimeout in my test code to make sure this was doing what it is supposed to, but I assure you, you don’t want to see it!

Mapping queries to application state

We can now make our entire collection of recipes available to the application using a single useState hook. If we make any changes to the collection (for example, using Db.RxCollection.insert), the state will automatically get updated and so will the view!

Let’s first add that hook and also update the useEffect dependency:

let (db, setDb) = React.useState(() => None)
let (recipes, setRecipes) = React.useState(_ => Belt.Map.String.empty)

React.useEffect2(() => {
  let dbPromise = Db.make()->Promise.map(db => {
    setDb(_ => Some(db))
    db
  })

  Some(
    () => {
      let _ = dbPromise->Promise.then(db => db->Db.destroy)
    },
  )
}, (setDb, setRecipes))

In addition to the added hook, this has a couple subtle changes:

  • The useEffect1 got changed to useEffect2 because the dependency array now accepts two arguments.
  • The dependency array changed from an array (which useEffect1 expects) to a tuple (which useEffect2 expects).

These are oddities at the intersection of Rescript and React that you just have to accept, at least for now.

Now we can add the pipeline that subscribes to the findAll query and sets the recipes (This goes immediately after the setDb call above):

db.recipes->Db.subscribeAll(recipeDocs => {
  let newRecipes =
    recipeDocs
    ->Belt.Array.map(Db.RxDocument.recipe)
    ->Belt.Array.reduce(Belt.Map.String.empty, (recipes, recipe) => {
      recipes->Belt.Map.String.set(recipe.id, recipe)
    })
  setRecipes(_prev => newRecipes)
})

This might seem weird if you aren’t used to using reduce in a functional programming style. I’ll go over it line by line.

We start with the db.recipes collection and pipe it into Db.subscribeAll.

Db.subscribeAll accepts two arguments: the collection being subscribed to (via pipe) and the callback that gets called whenever the underlying data changes.

The single argument to the subscription callback (recipeDocs) is an array of RxDocument.t<Model.recipe>s. The goal of the next few lines is to convert that array of documents into a Belt.Map.String.t with recipe ids as keys and Model.recipes as values. The resulting map is stored in a variable newRecipes that will get passed into the setRecipes function from the useState call.

The creation of that variable is the tricky part. The first thing we do is pass the array of RxDocument.ts into Belt.Array.map, with the RxDocument.recipes function. The result is a new Belt.Array.t where the values are Model.recipe type.

We then use Belt.Array.reduce to turn this array into a map. Conceptually, reduce is pretty simple, but a lot of people have trouble applying it to collections, so I’ll break it down.

First, Array.reduce is a lot like Array.map in that it applies a function to every element in the array. The difference is what that function accepts and returns.

In fact, the function is a reducer, exactly like the reducer you might be familiar with from Redux. The function accepts two values: the current state and the element being applied in this iteration of the loop. It returns the new state. That new state is what will get passed in as “current state” for the next element.

In our case, the “current state” is a Belt.String.map that contains the elements that were in the array before the current element is added, and the “new state” is a Belt.String.map that has all the elements in the array up to and including the current element.

Of course, we also need an initial state, before any elements in the array have been processed. Since we haven’t added any elements yet, this initial state is the Belt.Map.String.empty argument passed as the second argument to reduce, (the first argument is the array that is passed in by the pipe).

Inside the reducer function, we take the current recipes map and pass it into Belt.Map.String.Set with the “next” Model.recipe as the value, and the id of that recipe as the key.

If you’ve read my earlier article on useReducer, you’ll recall that Belt.Map.String.set is an immutable function. It does not change the map that is passed into it. It returns a brand new map that has the provided key set to the given value.

This returned map becomes the return value of the reducer function. So it becomes the “current state” for the next element in the array.

Once all elements in the array have been processed, the final Belt.Map.string.t value is put into newRecipes, which finally gets stored using the useStates “set” hook.

Mapping the taggedRecipes query

We can do a similar thing to ensure that the list of tags is also always up to date. You may want to try this yourself before proceeding as it tricky to understand and learning by doing is much better than just reading.

First, You’ll need a new tags useState hook and will have to change useEffect2 to a useEffect3 so it can access the setTags function.

The subscription itself looks very similar to that used for recipes except for the slightly different structure of the Model.taggedRecipes type:

db.tags->Db.subscribeAll(tagDocs => {
  let newTags =
    tagDocs
    ->Belt.Array.map(Db.RxDocument.taggedRecipes)
    ->Belt.Array.reduce(Belt.Map.String.empty, (tags, taggedRecipes) => {
      tags->Belt.Map.String.set(taggedRecipes.tag, taggedRecipes.recipes)
    })
  setTags(_prev => newTags)
})

Conclusion

We now have recipe state in our database! In the next article, we’ll start using this state in some React components. There are some intricacies to it, but it will come together very nicely as a fully functional offline app. Later, we’ll extend the offline app to sync with a graphql server!