A bug that crashes is a bug you fix that afternoon. The ones that survive are polite. They report success, they degrade instead of stopping, and they only ever appear on a machine that has been used before.
Three of those shipped in our Windows app this week. What links them is not a test somebody forgot to write. It is a machine none of our tests were ever run on.
The name disappeared and nothing said why
The first thing our app asks is what you want to call it. That name
is not a setting buried in an app-data folder somewhere. It sits in
plain text in your own folder, in
.nameos/identity.json, so you can open it, read it and
take it with you.
If that file began with a byte-order mark — three invisible
bytes that a lot of Windows tooling writes at the front of a UTF-8
file, PowerShell's own -Encoding UTF8 among them —
the app could not read it. So it fell back to the default and opened
normally.
No error. No warning. Just an assistant that had forgotten its own name, and a person with no way to work out what they had done wrong.
The failure was three bytes long, invisible in every editor, and the app's response to it was to carry on cheerfully.
That is fixed. So is the thing underneath it: a file that genuinely cannot be read now raises an amber callout that says so, rather than closing quietly and pretending nothing was there. An honest error beats a silent recovery every time, because a silent recovery recovers the program and abandons the person.
The wizard overwrote a name it had never asked about
The second one is not a crash at all. It is a decision nobody made.
Set the app up on a machine that already has one of these folders, with a name already in it, and the wizard wrote straight over it. Worse, it asks for the name at question one — five steps before it ever looks at the folder. So you were made to invent a name for something that already had one, and then watched it take yours.
Now the folder wins. If the folder you point at already carries a name, that is the name, and setup adopts it rather than replacing it. It was checked in both directions, on a first run with no folder at all, and on a deliberate rename afterwards, because "the folder always wins" would be its own bug the first time somebody genuinely wanted to change the name.
No test suite in the world was going to find that. It is not a defect in the code; it is two correct behaviours with no agreed precedence between them.
The one that proves the point
The day before those two, we found the reason to distrust all of it.
Our updater ran, exited zero, wrote the new version number into Windows' own Add/Remove Programs list, and left the old program sitting on disk. No error. Nothing queued for the next reboot to repair it. Windows' list of installed programs now said one version, the disk held another, and nothing in the machine was comparing the two. Whatever that computer is offered next, it is being decided from a record we already know is wrong.
The installer that did this had passed ten clean runs.
Ten passes in a row and the thing was broken the whole time. The runs were real. They just had nothing in them capable of failing.
The failure only appeared when somebody forced it: run the update against a copy of the app that is genuinely still running and still holding its own file open. Then, and only then, the overwrite fails.
And the cause turned out not to be the race everyone assumed. Windows raises its own error flag when that write fails; the installer script simply never read it, and then wrote a version number on top of the lie. One unchecked return value.
The fix does not try to outrun the lock, because a longer wait only fails at a different rate on a different machine. It renames the running program out of the way and writes the new one under the old name — Windows permits the rename even while the file is in use — and if the write still fails, it puts the original back and exits non-zero. Nothing after that line runs. The registry is never allowed to say more than the disk does.
What all three had in common
Every one of them only exists for somebody who has used the product before. A file that already has contents. A folder that already has a name. A program that is already running.
And a test rig is always fresh. That is the whole trap: the machine you verify on is the one machine in the world guaranteed not to have the state that breaks you.
A clean run is not evidence. It is the absence of evidence, wearing a green tick.
What we have not proven
We can show that the installer refuses to claim a version it did not write, because that failure was reproduced deliberately and the fix was tested against it. What has not been exercised end to end is the full chain from publishing a release to a machine downloading it and verifying its signature. That needs infrastructure we have only just put in place, and until it has been run, we say so rather than letting the sentence above imply it.
What to do
Stop asking whether the test passed. Ask what the test had the power to fail on.
Then go and cause the failure yourself. Hold the file open before you overwrite it. Put a byte-order mark on the config and see what your parser does with it. Install over the top of an existing setup instead of a clean machine, every time, because that is where your actual customers live.
And check the thing rather than the report of the thing. The registry said the update had happened. The disk was the only witness that mattered.