Skip to content

Newsletter · Issue #065

Your First Duplicate-File Count Is Wrong

After this issue, the reader can measure whether their own apps genuinely share source code, and knows the two false positives that make the first run report duplicates that are not theirs.

Published
Format
Quick Teach
Reader job
Teach
Length
2 min read
Written by
Victor Solano

If anyone tells you two of your apps are the same app, that is a claim about bytes and you can settle it in a minute. Hash every source file in every project, then count how many are byte-identical across two different apps. I ran it over sixteen app directories and got 1269 files, 2 shared. Two shared files between separate products is not a number you want to be wrong about, in either direction.

The run that was wrong. The exclude list is the bug.
import hashlib, collections
from pathlib import Path

SKIP = (".build", "DerivedData", ".git")
by_hash = collections.defaultdict(set)

for app in Path("apps").iterdir():
    if not app.is_dir():
        continue
    for f in app.rglob("*.swift"):
        if any(p in f.parts for p in SKIP):
            continue
        by_hash[hashlib.sha256(f.read_bytes()).hexdigest()].add(app.name)

shared = [a for a in by_hash.values() if len(a) > 1]
print(len(shared))  # 2

Both hits were the same filename: GeneratedAssetSymbols.swift, a thirty-line file Xcode writes from your asset catalogue. Neither was mine. They matched because one compiler generated both from similar inputs, and they sat in directories named .derivedData-ui-glass, .derivedData-device, and plain build. My exclude list matched DerivedData, capital D. It never looked at any of them.

Match the prefix, not one spelling. Same corpus: 1262 files, zero shared.
SKIP = (".build", "DerivedData", ".git", "build")

def is_generated(f):
    return any(p.startswith(".derivedData") or p in SKIP for p in f.parts)
  1. Step 01

    Run it once and distrust the number

    Whatever it prints, treat it as a first draft. The interesting output is not the count.

  2. Step 02

    Print the path behind every match

    Read them. A generated file, a vendored dependency, or a test fixture is not your code and should not be counted as yours.

  3. Step 03

    Widen the exclude to a prefix, then re-run

    Match .derivedData* rather than one spelling, and add plain build. Under-excluding invents duplicates you never wrote; over-excluding hides real ones. Now the count is quotable.

Free tool

App Review Rejection Decoder

If the reason you are counting duplicate files is that App Review sent you a guideline number, paste the message in. It reads back which guidelines it cites, what each one actually asks for, and splits the fix list into what you can verify yourself and what you have to decide. No account, no charge.

Open the decoder

Print the paths before you quote the count.