The ledger

What Cal got wrong.

Every product in this category publishes an accuracy number. None of them publish the questions they missed. This page is the second thing, kept in public and updated when it happens rather than when it is convenient.

Why this exists

An accuracy claim you cannot check is marketing.

You are going to put what Cal says into a file. You need to know how it fails, not just how often it succeeds, because those are different questions and only one of them is useful when something goes sideways at 4pm on a Friday. So here is the log. What broke, what it cost, what changed, and when.

The log

2026-07Display

Every dollar figure in the valuation comp table rendered as $NaN.

The valuation engine emitted one set of field names and the page that displayed them expected another. Sale price, adjustment, and indicated value all came back empty on any real property.

What made it worse: the page shipped with its own sample data written to match the display code, so the bug was invisible in a demo and only appeared the moment someone ran a real address. It survived several demos that way.

FIXED. The sample data was deleted outright rather than corrected. A page that can only be trusted when it is showing fake numbers is not a page worth keeping.

2026-07Data

California's largest assistance programs disappeared from any search that filtered on credit score.

A data conversion had concatenated some minimum credit score values into themselves, so a program requiring a 660 was stored as a number in the hundreds of billions. Nothing errored. The programs simply failed every score comparison and vanished from results.

If you searched assistance for a 680 borrower in California during this period, you were shown a shorter list than you should have been, with no indication anything was missing. That is the worst kind of bug in a matching product, because a silent omission looks exactly like a correct answer.

FIXED, and a test now fails the build if any minimum credit score exceeds 850.

2026-07Data

The assistance search tool was reading fields that did not exist.

The lookup asked the dataset for first-time-buyer status, income limit, and minimum credit score using names the records did not use. Every one came back empty across all 1,311 programs, so the model answered from a much thinner picture than the data actually held. The first-time-buyer filter could never match anything at all.

FIXED. The richest part of the dataset now reaches the model. Field names are asserted in a test so a rename cannot silently break it again.

2026-07Display

The multi-lender comparison timed out on every production run.

A proxy timeout was set shorter than the work it was proxying, so the feature we lead with failed on a cold run and often had to be tried twice. It was three configuration numbers, which is the frustrating part.

FIXED. Timeouts across the chain now exceed the maximum work they wrap, and a live run is part of the deploy check.

2026-07Claim

We said lenders push live pricing specials into Cal. They had not.

The feature was built and the table was empty. Nobody had pushed anything. The sentence was written for a version of the product that had not happened yet, which is the most common way honest people end up saying untrue things.

FIXED by deleting the claim. It comes back when a lender has actually done it, and not before.

2026-06Data

One ingest path stamped the import date onto documents that did not state an effective date.

A document with no stated effective date was recorded as effective the day we imported it. On screen that read as a freshly effective policy. It was not wrong by much and it was wrong in the worst possible place, because the effective date is the part of a citation people rely on most.

FIXED. Undated documents now store nothing and are flagged unconfirmed, and the interface says so rather than showing a date. This is now item 02 on what Cal will not do.

The evaluation, honestly

Here is our accuracy number and here is why it flatters us.

Cal runs a regression suite of 75 real underwriting questions with known answers, graded automatically, on a schedule. It is a genuinely useful instrument and it catches regressions before customers do. It also has four weaknesses we would rather tell you about than have you find.

underwriting questions with known, cited answers

of those 75 have supporting material that was written into the library specifically because the question was in the suite

Recall@K

the retrieval measure we consider more honest, because it grades whether the right source was found rather than whether the answer sounded right

2 pts

the regression threshold. A drop larger than this blocks a release

Roughly a third of the suite is taught to itself. When a question failed, we sometimes added the missing guideline material to the library. That is the correct response to a real gap, and it also means the score partly measures how well we studied for our own test.
The nightly run does not use the production prompt. It uses a stand-in, so it measures retrieval and grading more than it measures the exact experience you get.
Ties are broken by a language model. When the automated grader cannot decide, another model judges. That is better than a coin flip and it is not the same as a human reading it.
The score is not a generalization estimate. It tells you whether Cal got worse since yesterday, which is what a regression suite is for. It does not tell you how Cal performs on a question nobody has thought of. Treat it as a smoke alarm, not a warranty.

Found one

If Cal told you something wrong, we want it more than you want to send it.

Send the question and what you expected. If it is a real miss it goes on this page with the fix and the date, whether or not it makes us look good. That is the entire deal.

Report a miss

Chris Black reads every one of these. Twenty years originating, which is long enough to know that the vendor who admits the bug is the one still worth calling next quarter.

NMLS 275073 · Alamo, California