The Merchant Ladder

A merchant is no longer an establishment. It is the entity that owns the transaction — the destination of the funds. On a card statement that is still a shop name. On a manual entry, the name you typed is the merchant: "churrasco de quinta com os amigos" resolves to a merchant called Churrasco, and your Thursday ritual sits at the same level as Uber.

That single grain is what makes essentiality computable: how essential a category is to your spending, and how essential a merchant is within that category. So every transaction must carry one, and the database now refuses any that doesn't.

Resolution answers two questions in order, and they are genuinely different questions. The rest of this page walks the whole path: where a transaction arrives from, what normalization strips, how identity is found or created, how the category is chosen, and what a correction teaches.

Everything below happens because a transaction needs a category, and the only thing it arrives with is a name. Where that name comes from changes how much work the pipeline has to do.

Typed by hand. You write "churrasco de quinta com os amigos". The name is clean, human, and often not a business at all. POST /transactions resolves it inline: if you didn't pick a category, the pipeline picks one, and the request fails rather than saving a transaction with no merchant.

Imported from a statement. Fifty to two hundred rows of card gibberish at once. The app makes two calls: first POST /transactions/categorize/batch with the unique names, which resolves everything and returns a suggested category per descriptor; then POST /transactions/batch carrying those categories back. Splitting it that way lets you see and correct the guesses before anything is written.

Repeated or edited. A repeat clones an existing transaction onto a new date. An edit is the interesting one — changing a category is the single most valuable signal the system gets, and it is handled separately from a normal write.

A card network packs merchant name, city and country into fixed-width fields and truncates whatever doesn't fit. IFD*INCONFIDENTES ALIMENTBELO HORIZONTBR is an iFood order from a restaurant called Inconfidentes in Belo Horizonte, with the words sawn off mid-letter and glued together. Normalization is a sequence of narrow rules, each undoing one thing the network did.

Two of these are sharper than they look. The country code is only removed when it is glued to a word, because a space-separated one is part of the name — AUTO POSTO VIA BR is a real petrol station and must survive. And the city list is matched as truncated prefixes from six characters up, because BELO HORIZONT never appears in full.

How the gateway token is identified

Nothing recognizes IFD as iFood at this stage. Finding the token is positional: it is the run of two to ten alphanumerics before the first asterisk, and that pattern is applied to every descriptor without consulting any list. CIELO*, PAG* and ZZ9* are all extracted exactly the same way.

There is a second list, and it does a different job. Normalization has to remove the prefix from the name, and for that it uses a roster of known processors (IFD, PAG, MP, CIELO…) plus a catch-all rule for any short unknown prefix. That list decides what gets stripped. It never decides what the token means.

Because the token is offered last, the same prefix can end up mattering or not. IFD*OUTBACK resolves to Outback — the tail is a merchant we know, and tier 1 answers before tier 4 is ever reached. IFD*INCONFIDENTES ALIMENTBELO HORIZONTBR resolves to iFood, because the restaurant is not in the dictionary and the gateway is the only thing left that names a destination. One rule, two outcomes, decided entirely by what else is known.

One guard: if the extracted token also appears as an ordinary word in the descriptor, it is not treated as a gateway at all — otherwise a merchant that happens to start with a short word followed by an asterisk would be misread as an aggregator.

The normalized string is expanded into lookup keys ordered by specificity, and each tier is tried in turn against the dictionary. The first hit wins and the rest are never queried.

The invariant that holds it together

Every merchant carries an alias identical to its own normalized name — a self-alias. Without it a merchant is unreachable from any longer phrase that merely contains it, because tier 2 searches aliases and not merchant names. The seed writes one for every merchant it creates, and so does the code path that mints a merchant at runtime.

This is the real resolver — the same normalization rules and the same key ordering as the backend, loaded with the actual 310-alias dictionary. Pick a descriptor or type your own, then walk it one probe at a time.

Stage A must end at a merchant — the column is NOT NULL, and essentiality has nothing to measure without one. So a miss creates identity rather than giving up. How well it does that decides whether the dictionary stays clean or fragments into near-duplicates.

With no AI key configured, the normalized string simply becomes a merchant. With one, the descriptor goes to a batched call that is deliberately not open-ended.

The alias table is the cache

This is the part that makes the cost bounded. Every AI resolution writes an alias — even a low-confidence one. The descriptor a model saw once is answered by tier 2 forever after, for every user. A wrong merchant is one run of the merge script; a skipped alias is a bill you pay on every import, permanently.

So the cost is not per import, it is per descriptor ever seen. A batch of sixty unknown descriptors is roughly 1.5K in and 1.5K out — about a cent on Haiku 4.5, the configured default. The second import of the same card is nearly all cache hits, and Brazilian descriptors overlap heavily between users, so the dictionary gets cheaper for everyone as it fills.

One guard on top: a confident AI answer is promoted into the shared prior only above 0.6 confidence, and never for transfers — a Pix or TED descriptor names a person, not a business, and has no business teaching the global dictionary anything.

Stage A is shared by everyone. Stage B is not, and the order matters more than the mechanism.

Your own history winning is the entire point. Uber is transport for almost everyone, but if you drive for Uber it is income, and after one correction it stays income for you without touching what Uber means to anybody else.

The response carries which of the four answered, so the app can show the difference between a category you taught it and one it guessed — and so an import preview can flag the rows worth checking.

Every saved transaction adds one occurrence and its spend to your row for that merchant and category. Corrections are handled differently, and the difference is load-bearing.

A correction moves the evidence. Re-categorizing decrements the old pair and increments the new one. Simply adding to the new pair would leave both tied, and the highest-occurrence lookup would keep handing back the category you just rejected. This is the bug that makes personalization feel broken, and it is one line of intent that is easy to get wrong.

Nothing you do touches the global dictionary directly. The shared prior is a rollup of everyone's rows, recomputed by a job that requires at least three distinct users to agree before it will write a prior. One enthusiastic user filing Uber under shopping cannot redefine Uber for the world.

Curated and derived priors are separate rows. They carry a source marker — seed or rollup — and the unique key includes it. The rollup only ever deletes and rewrites its own; hand-curated priors are invisible to it. Before that column existed, a single rollup run destroyed 50 curated priors by upserting onto them.

The two *_categories tables look nearly identical and cannot be merged: one is the sum of the other. Source rows and their aggregate do not belong in the same table.

Until recently there was a categorization_rules table: eleven rows of regular expressions with priority numbers, matched against the raw descriptor. It worked, and it had two problems that got worse with every row added.

The first was that priority was hand-assigned. Every new rule had to be slotted against every existing one, and nobody could say from the outside why rule 40 beat rule 30. The second was that a rule produced a category and nothing else — there was no merchant, so there was nothing to hang essentiality on, and no way for one user's correction to mean anything.

Replacing it with a dictionary made priority fall out of the data instead: specificity is the length of the key that matched, and the tiers order themselves. Regex coverage that genuinely needed a pattern became the 16 prefix stems. The rest became ordinary alias rows.

The ordering bug this exposed

Specificity ordering only worked for the full string. Below it, single words were tried left to right, so position quietly stood in for specificity: MERCADO LIVRE*COMPRA has no alias of its own, fell through to words, and matched the generic MERCADO because it comes earlier in the string than LIVRE. Every Mercado Livre purchase was filed as groceries under the wrong merchant.

The fix is a middle rung: contiguous phrases from three words down to two, tried before any single word. Nothing else about the ladder changed.

Why it stays bounded

Every contiguous phrase is quadratic in words, and an import resolves fifty to seventy unique descriptors at once, so two rules keep the key set small. Phrases cap at three words, because brands run to about that length. And no phrase may open or close on a stopword, which kills PAO DE, DE ACUCAR and LIVRE COMPRA before they ever reach the database — while PAO DE ACUCAR, stopword safely interior, survives.

The pipeline is designed so that nothing merges or renames itself. Three jobs exist for the things that genuinely need judgement.

Merging duplicates. A trigram report lists near-identical merchants with a ready-made command. It is advisory only, and deliberately so: similarity cannot tell a generic concept from a brand containing it. Cinema and Cinemark score highly and must stay separate; SAINT LAURENT BRA and Saint Laurent score lower and are obviously the same shop.

Recomputing the prior. The rollup job sums per-user rows into the shared dictionary, subject to the three-user floor, and only touches rows it owns.

Backfilling. Existing transactions can be re-resolved against an improved dictionary, with a dry-run mode that reports what would change before anything is written.