Skip to content

Newsletter · Issue #051

The Archive Was 87% of the Backup

After this issue, the reader can size a backup job so the files they cannot replace never wait behind files they can.

Published
Format
Quick Teach
Reader job
Teach
Length
2 min read
Written by
Victor Solano

Run du -sh on the tree before you write the script that copies it. Mine came back 231G. One folder inside it, _old/, is 201 of that: 98G of old YouTube footage, 60G of a game archive, 43G of scraped social posts. The source I actually build with is about 30G. As one job, the files I cannot replace upload behind all of it, every run.

  1. Step 01

    Measure before you copy

    du -sh the tree, then du -sh the largest folder in it. If one directory is most of the total, you do not have a backup problem, you have an ordering problem.

  2. Step 02

    Split by what changes

    Tier 1 is the live source with the archive excluded. _old/ stays on disk and never uploads; the drive it lives on is 23% full with 1.4 TiB free. That is my call, and it is written at the top of the script so a later run does not add it back for completeness.

  3. Step 03

    Prove the copy, not the command

    Pull files back out of the destination and compare SHA-256 against the originals. I did two, both matched. Exit 0 says the transfer ended without an error. It does not say the bytes on the other side are the same bytes.

--partial so an interrupted upload resumes instead of starting over. No --delete: a safety net is not a mirror, and a local deletion must not erase the only other copy.
du -sh "$SRC"          # 231G  — whole tree
du -sh "$SRC/_old"     # 201G  — 87%, and none of it changes

rsync -rlt --partial --stats \
  --exclude='_old/' --exclude='node_modules/' \
  --exclude='build/' --exclude='dist/' --exclude='.next/' \
  "$SRC/" "$DEST/"

The run reported 132,463 files considered, 112,407 transferred, 26,424 MB, exit 0, in under five minutes.

What this still is not

Those files are on the local Google Drive mount. Google still has to upload them, and that is a separate step. I read the local upload queue three times this morning and got 0, then 16, then 0, so a zero reading is a moment and not a finish line. A journal entry I wrote twenty minutes after the run is not in the copy either; a snapshot holds what existed when it ran. Tier 2 waits until tier 1 is off the machine.

On the mount is not off the machine.