How RadarModel Schema Was Tested Against Real Databases
A drift detector earns trust slowly and loses it in one night. Report a dropped table that’s still there and the alert channel gets muted. So RadarModel Schema’s tests spend as much effort on what it must not report as on what it must.
None of them use a fake database. SQLite runs on Linux, macOS and Windows. Postgres 12, 16 and 17, MySQL 8.0 and 8.4, and MariaDB 10.6 and 11.4 run in containers on every push. Each test creates its own database on the server, does real DDL to it, and drops it afterwards.
Sixteen faults, one outcome.
Real changes, caught once
Sixteen kinds of manual change are made for real: a column added and dropped, a type changed, NOT NULL added, a default changed, an index added, dropped, extended by a column and turned descending, a check constraint rewritten, a trigger redefined and disabled, a collation changed, a view redefined, and a table added and dropped. Each has to be reported at the right severity and then not again. Where a database can’t do the change, such as altering a column type in place on SQLite, that case is skipped for it. In the recorded run that came to 16 cases on Postgres, 15 on MySQL and 10 on SQLite.
Faults that must not look like drift
The failure tests take a baseline and then break something:
- the connection drops between the two reads of a scan, or the server stops accepting connections
- the password is wrong, the connection string isn’t set, or it can’t be parsed
- the SQLite file is moved, overwritten with text, or replaced by a folder
- a migration runs between the two reads, once and then continuously
- the scan takes longer than its timeout
- privileges are revoked: everything on Postgres, one table’s SELECT on MySQL, and the TRIGGER privilege on MySQL
- the Postgres connection comes with another
search_path,IntervalStyle,DateStyleand float settings - most of the tables disappear at once
Each test checks the same things. The database is reported as an error, or read correctly with no changes. Nothing is reported as drift. The stored snapshot file is byte for byte what it was. Where it applies, the next good scan is quiet, and a real drop afterwards is still reported. In the connection string tests, the password must not appear in the error. The 15 failure tests ran three times each in the recorded run, with no failures.
A few of those cases came from an independent review before release. It found that a Postgres role with its own search_path printed type names without their schema, which showed up as a breaking type change. It found that MySQL hides triggers from a user without TRIGGER, and that a password could appear in an error when the connection string was malformed. Each now has a test that fails without its fix.
Odd but legal schemas
Some schemas are unusual without being wrong. MySQL lets a unique key and a foreign key share a name. SQLite allows two foreign keys on the same column, and a virtual table whose module this SQLite doesn’t include. Postgres has partitions, row-level security, and extensions that create their own tables. Each of these has a test that scans it and checks what was stored, and for partitions and row-level security, that a change to them is reported.
Size
A large-schema test creates 2,000 tables of eight columns with an index each and scans them twice, the second time after one table is dropped. In the recorded run on a four-core GitHub runner, Postgres took 0.41 seconds for the baseline and 0.47 with the drop, SQLite 0.81 and 0.89, and MySQL 1.6 and 0.62. MySQL’s first scan is the slower one because it checks that every table can be read.
Where to look
The whole suite is 93 tests, 37 of them for RadarModel Schema, most run once per database. They pass on every push, and again with the race detector. The built binaries run an offline smoke test before each release. Everything here is in the repository’s test results, with the machine and versions they ran on.