Skip to content

Newsletter · Issue #066

305 Tests Failed and None of Them Was a Bug

After this issue, the reader can tell a real regression apart from a suite that inherited the wrong NODE_ENV, and can pin their own test config so the caller's environment stops changing the result.

Published
Format
Field Note
Reader job
Reveal
Length
2 min read
Written by
Victor Solano

I ran the test suite on my site this morning before touching anything. It reported 305 failed out of 979. Yesterday the same commit was green, nothing had been pushed overnight, and git status was clean. Every failure said the same thing: React.act is not a function. That is not a message about my code. React picks which build to load at require time, from NODE_ENV, and the production build does not export act. So the question was not what broke. It was who set NODE_ENV to production.

The whole diagnosis, before reading a single stack trace.
node -e 'console.log(process.env.NODE_ENV)'

env -u NODE_ENV npx vitest run

It was my own shell. Not the repo, not a dependency bump, not a lockfile. A parent process had exported NODE_ENV=production and the suite inherited it. With the variable removed, the same commit passed 1117 of 1117.

One commit, four runs, only the caller's NODE_ENV changed.
NODE_ENV of the callerPassedFailedUncollected
production6613050
unset111700
production, after fixing React only968018
anything, after fixing the config111900
Incident
A suite that passed yesterday reported 305 failures on an unchanged commit, all of them React.act is not a function.
Decision
I fixed no test. I pinned NODE_ENV in the test config, both before the config loads and for the workers, then added two assertions that fail if either pin is removed. Row three is why both are needed: pinning it for the workers fixed React and left 18 files failing to collect, because the build tool reads the variable separately, in the config process, before any worker exists. Two consumers, one variable, and fixing the loud one hid the quiet one.
Portable lesson
A gate whose result depends on the environment of whoever ran it is not a gate. Pin the variable in the config, where every caller gets it.

Check this before you debug the failures

If a suite goes red with no diff behind it, print NODE_ENV inside the run first. A wrong value there fails hundreds of tests at once and every message points at the library instead of the cause. I also ran my new guard with the pin removed and watched both assertions fail, because a test that has never been seen red is one you are trusting on its word.

Green and red both need a reason. Find the reason.