Matrix vs. Async Hell vs. Flutter

“Callback Hell”

Ages ago we noticed that async responses were just a special case of what reactive systems handle in their sleep: state changes from outside the running system. Outside?

Thinking outside the DAG

Just as a user decides in their own sweet time when to click a button or press a key, somewhere out there a remote server eventually gets around to responding to an AJAX request, with the Interweb shuttling request and response at its own pace. Our app controls neither the generation of the response nor its conveyance. Control? It does not even observe them. The app is prepared for this state to arrive but, until it does, this state is “outside” the reactive DAG.

To make this distinction clearer, let us examine our two cases more closely.

To click or not to click

In the case of a user click:

• the sequence is initiated in the mind of the user, when they decide to click;
• they click and the UI framework passes that event to our UI on-click handler. So far, no Matrix;
• now our on-click logic uses the Matrix API to store the event in some predesignated reactive “input” property, and…

…we are done! The external “click” intent is now percolating through the reactive DAG.

The async case is slightly more involved.

Waiting for Godot

The decision to generate the XHR will itself be a reactive result. In our TodoMVC example, we imagine we are in fact writing a prescription drug reminder list:

• the user enters or changes one of their Rx’s;
• an ae-lookup property, “ae” for adverse event, sees the new Rx and creates an XHR against the FDA database:

:ae-lookup (cF (when-let [rx (mget me :rx-name)]
                 (http/get (.https Uri "api.fda.gov" "drug/event.json"
                 {"limit"  "1"
                  "search" (str "patient.drug.openfda.brand_name:" rx)}))))

• An observer/watch on ae-lookup dispatches the XHR via the HTTP library, attaching a response handler described below;
• the network and the FDA service do their thing, and the HTTP library receives the response;
• the HTTP library passes the response to the XHR response handler;
• our app code in the handler, like the click handler, stores the response in a bespoke reactive input ae-response property, and that information propagates through the app DAG normally.

Enter Flutter

The astute reader may well have been concerned by the XHR solution. Why? Because it uses two properties to get an XHR executed. One formulaic property watches other app state and decides when to generate an XHR. A second input property captures the response, so other state can watch it.

This excess never alarmed us when our app did only one or two XHRs. Then came Flutter. All async all the time.

Forever creating async pairs held no appeal, but we quickly thought of a solution thanks to other Matrix tricks that have worked well. Let us start with an example of that.

cFI: Formulaic-then-Input Cells

The cFI macro defines a formulaic cell that gets evaluated like any other formulaic cell when the owning model is created, but immediately morphs into an input cell. They are necessary when we want to pre-fill an “input” property with a value calculated from other app state at the time a model is created. An analogy can be made to constructor functions fleshing out an OO instance based on constructor inputs and other app state.

Anyway, this formula-to-input metamorphosis precedent gave us an idea.

One property to serve them all!

What if we could get away with using one property to both formulaically derive an XHR and serve as a reactive “input” to capture its response? It might not always be feasible — sometimes we might need the two-property solution — but in many cases we saw this sequence would be possible:

• have a formulaic cell “compute” a Future;
• have Matrix “new value” internals watch out for :async? true properties being assigned a Future and, if spotted, do two things:
• first, attach a generic .then handler that, when it fires, will in effect mset! the same property to the response; and
• second, in the meantime, store nil in the property to serve as its pending value. Corollary, the property never takes on the Future as a visible value;

The above solution was developed to support the openBox getter of Hive, a Flutter package for local storage, while working on a version of TodoMVC. openBox returns a Future which will resolve asynchronously to a writable “DB”. Note how the code to load existing to-dos starts by testing if the DB has been asynchronously obtained:

(fx/material-app
  {:db        (cF+ [:async? true]
                (-> hv/Hive (.openBox "todo")))
   :todo-list (cF (when-let [db (mget me :db)] ;; <== "waits" on async DB
                    (todo/make-ToDoList "todo"
                      (stg/collection-docs "todo"))))}
  ...)

Great! Now many straightforward async tasks can be handled transparently with a single property. Speaking of which, we also saw that, where the Future response had just one purpose, we could avoid a third property by processing the response on the fly with an :async-then handler:

:ae-events? (cF+ [:async? true
                  :async-then (fn [c lookup]
                                (= 200 (.-statusCode ^dht/Response lookup)))]
              (dht/get (.https Uri "api.fda.gov" "drug/event.json"
                         {"limit"  "1"
                          "search" (str "patient.drug.openfda.brand_name:"
                                     (mget me :title))})))

The XHR was doing a lookup in an Adverse Events registry, and the app just wanted to identify entries with or without adverse events, so we did it all in one property. Sweet.

Good news

The good news was that, with just a few lines of code added to the Matrix internals, we had been able to leverage the formulaic capabilities and mutation capabilities of Matrix largely unchanged to produce a hybrid new kind of property. Or so we thought.

The bad news

Luckily for us, our first use case happened also to break our straightforward solution. Let us revisit that code:

:db (cF+ [:async? true]
       (-> hv/Hive (.openBox "todo")))

That is perfectly fine, but it does not depend reactively on other app state. So what? Can you say “Matrix optimization”?

A very significant performance gain was found early on in the development of Matrix. If, after a formula has run, it ends up not having read any reactive property, we convert it to a constant property, discarding the Cell machinery and linkages supporting the formula.

See the problem? By luck, this db formula could be optimized away. It was indeed, and when the response handler fired, the Cell managing the property got optimized away. It no longer existed to complete the async processing.

Back to the good news: we survived the necessary refactoring. It turned out to be quite deep, but unsurprisingly made Matrix a better engine.

Summary

1. How cool is the reactive paradigm? With no design intent, it solves “callback hell”.
2. I am starting to think of Matrix as more of a programming approach than library because, every time it gets applied in a new arena, we find a clean way to extend Matrix to solve some coding problem specific to that arena. Matrix is a meta-library.

Meta-summary

Our reader may be thinking, “Wait. What is wrong with using two slots for an XHR?” Can you say “boilerplate?”

The Matrix Prime Directive is “Handle state change automatically.” The Subprime Directive is “Make the Prime Directive simple and terse to code.”