How RadarModel AI Was Tested Against Broken Provider Answers

A tool that watches for removed models has one way to be useless: telling you a model is gone when it isn’t. Do it twice and the alerts get muted, and then the real removal goes by unseen. So most of RadarModel AI’s tests aren’t about finding changes. They’re about not inventing them.

The fake provider

The tests run against a fake API that answers like all four providers at once: OpenAI’s list, Anthropic’s with its cursor paging, Google’s with page tokens, OpenRouter’s with prices and limits. Each test takes a baseline, then breaks the next answer from one provider in one specific way.

Fourteen ways to break a provider’s answer, from a 500 on every attempt to a server that’s too slow. Every one ends the same way: a failed scan with exit code 2, no change reported, the stored snapshot byte for byte the same, and a quiet next scan.

Fourteen faults, one outcome.

The fourteen faults: a 500 on every attempt, a 429 on every attempt, a maintenance page served with status 200, a body cut off in the middle of the list, an error object with status 200, an empty list, a null list, a failing second page, a failing last page, a “there’s more” with no cursor, a page token that never changes, a list that suddenly holds a fraction of its models, a field with the wrong type, and a server slower than the timeout.

For each one the test checks four things. The provider is reported as an error. No change is reported for it. Its stored snapshot is byte for byte what it was. And once the provider behaves again, the next scan finds nothing at all. The other three providers have to scan normally the whole time.

Each of the fourteen ran three times in the recorded run. No false removals.

The faults you can’t see

Some broken answers look perfectly fine. Google’s list arrives in pages; if a page token goes missing, page two silently isn’t there and the answer looks complete. No error, nothing to catch.

That’s why a watched model has to be missing from two scans in a row before it counts as breaking. The first miss is a warning. The tests cover this case directly: a short list warns once, and when the full list comes back the next scan is quiet and the warning is gone from the store. OpenRouter answers that look paged when they shouldn’t be are failed scans too.

Beyond the fourteen

The rest of the suite covers the parts around them. Retries honour Retry-After. A damaged snapshot file is an error, never a fresh baseline that would hide everything since. A snapshot that can’t be saved leaves the old one in place. A cancelled run writes nothing. Keys go in headers, never into URLs or errors, and a redirect to another host is refused so a key can’t follow it. Webhook errors name the host and never the URL, because Slack’s URLs carry tokens.

The whole suite is 72 passing tests, run on Linux, macOS and Windows on every push, and again with the race detector on Linux. Before release, the built binaries run an offline smoke test: a baseline, a quiet scan, then a watched model taken out, which must warn on the first scan and exit 1 on the second.

Against the real thing

Fakes only prove so much. So the recording also scans OpenRouter’s live public list twice. On October 17, 2026 it read 470 models, and the second scan found no changes. That’s the parser against the real thing, and the comparison holding still when nothing moved.

Everything here is in the repository’s test results, with the machine and Go version they ran on. An independent review went through the code before release and its fixes are in 0.1. RadarModel Schema, which came in 0.2, has its own write-up: How RadarModel Schema was tested.