Shopify metaobjects are a great filing cabinet and a terrible index

Shopify metaobjects are a great filing cabinet and a terrible index.

I found that out building ViewForge, which answers one question for a parts merchant: does this fit my 2021 Ford F-150? Answering it means storing tens of thousands of rows about what fits what, and metaobjects were obviously the right home for them. The data belongs to the merchant. It survives an uninstall. It stays editable in the Shopify admin. Nobody has to take my word that a copy of their catalog is safe in a database they cannot see.

Then I tried to ask it a question. A filtered count. A sort on something that is not the sort key. A join. Shopify either says no, or says yes slowly enough that you wish it had said no.

So which do you pick, the store that is correct or the store that is fast? Neither. You keep both and you make the sync somebody’s problem. Mine, specifically.

MetaDB runs two tiers. Metaobjects stay the system of record, durable and merchant owned. A Postgres mirror sits beside them and answers everything Shopify will not, and the mirror is deliberately disposable. If it burns down I rebuild it from the metaobjects and lose nothing that matters. That is the whole trick, and it took me longer to arrive at than it should have: only one tier is allowed to be precious.

The part I actually enjoyed is the sync engine, because a sync engine’s real job is not writing. It is dying well. Any server can lose power halfway through a multi step write, and the question is what the next server can work out from the wreckage.

So the log of in flight writes does not live on the server doing the writing. It lives in signed Shopify metafields. The record of what was being done sits in the same place the data does, which means a machine that has never heard of the machine that died can pick up the log, see what was half finished, and finish it.

One more decision I keep having to defend. MetaDB ships to the other apps as a library, not as a service. Warranty Records, Quotes, Spec Layer and eventually ViewForge each run the engine inside their own server. There is no central MetaDB that everything phones. A service would mean one shared rate limit budget, one shared uptime, and one bad afternoon taking four products down together. A library means every app keeps its own Shopify access and its own blast radius.

Is it finished? No. The write path, the worker runtime, the drift check and the webhook consumer are in, and the apps on top are what comes next. I want to write about the drift check on its own, because “what do you do when the two tiers disagree” turned out to be the question the entire design hangs from.