Skip to content

Newsletter · Issue #053

The Guard Was Green Because It Could Not Fail

After this issue, the reader can tell whether a passing check has ever been capable of failing, and can get a real red out of an existing guard without building a fixture for it.

Published
Format
Teardown
Reader job
Reveal
Length
2 min read
Written by
Victor Solano

Two of my apps, Sortes and VaultSnap, each ship a regression test that exists to fail if the Google ad SDKs ever come back. Both were green. Both lockfiles still had the packages pinned in them while those tests ran.

The markers were written by a person and the file was written by SwiftPM, and the two do not agree on what the package is called. VaultSnap's test did scan Package.resolved. It looked for GoogleMobileAds, and SwiftPM had written the identity lowercase and hyphenated.

Package.resolved is gitignored in both apps, so it does not exist on a fresh clone or in CI and the guard skips it there. That is deliberate. The machine where a stray resolve reintroduces the dependency is the machine where the file exists.
# what the test looked for
"GoogleMobileAds"

# what SwiftPM had written in the file the test was reading
"identity" : "swift-package-manager-google-mobile-ads"

# the red proof, and it was free: the pins were still in the lockfile,
# so the fixed guard was run BEFORE clearing them
xcodebuild test -only-testing:SortesTests/AdRetirementRegressionTests
# failed on both markers, naming Package.resolved
# pins cleared -> 85 XCTest + 27 Swift Testing, 0 failures
Reported as one defect in two apps. It was two defects, and the patch that fixed one would have faked the other.
ItemVaultSnapSortes
What was wrongScanned the lockfile with markers that could not match itNever scanned the lockfile at all
What it neededThe lowercase markersThe path as well as the markers
Given the marker-only patchActually fixedLooks fixed in the diff, still checks nothing

The check that would have caught it

Has anyone watched this assertion go red? A guard that reads the right file and reports the opposite of its contents is worse than no guard, because it gets cited as evidence. The cheapest red is the one already sitting there: run the fixed check against the mess before you clean it up.

The thing the guard protects

Twelve apps, no ads, no ad SDK

The guards in this issue exist because the whole portfolio is ad-free and stays that way. No ad or tracking SDK is linked in any of the twelve, and no shipped binary was affected by the defect above. What was broken was the proof, not the product.

See the apps

A check nobody has seen fail is a claim, not a test.