I work in managed IT, infrastructure, security, automation, backups, networks, and the untidy space where a clean diagram meets a real building full of old wiring. The posts here are field notes, not recycled product copy. Details are generalized when they could identify a client or expose a system, but the failure modes and lessons stay real.

You will also find the 90s Kid collection, because the people who survived IRQ conflicts, dial-up noise, rewinding VHS tapes, and printers that sounded like farm machinery deserve their own archive.

What you can expect

• The problem, including what made it misleading.
• The evidence that separated the real cause from the tempting guess.
• The fix, the boundary, or the lesson worth keeping.
• Enough technical detail to be useful without publishing somebody’s private infrastructure.
• A willingness to say when the machine was right and I was the problem.

ErrorZap uses privacy-first traffic and performance measurement through Cloudflare. It does not use an advertising network or build visitor profiles. See the privacy note for the short version.

For managed IT help, hire me.

There is no advertising network, cross-site visitor profile, newsletter tracking pixel, or comment-account database on ErrorZap.

The operational logs needed to securely deliver the site may still include normal request information for a limited period. If the analytics or hosting stack changes, this page will change with it.

Last updated: July 29, 2026.

Start with the expensive lessons

• How My Own Backup Deleted My Files
• The Firewall API Call That Wipes Your Router
• The Settings API That Corrupted Every Setting
• I Found My Own Server on Shodan

Then try the weird ones

• Seven Network Gotchas That Look Like Ghosts
• The Smart Blind That Took Down the Network
• The PWA That Froze When Offline
• When Postgres Won’t Start

Or go back to the 90s

The 90s Kid collection is about floppy disks, pagers, dial-up, VCR clocks, flying toasters, and learning technical patience before anybody called it troubleshooting.

Find something specific

Search every post, browse the complete archive, or subscribe using the RSS feed.

What I do remotely

• Managed IT and monitoring
• Patch management
• Ransomware-proof backups (append-only, immutable storage)
• Camera and alarm system health monitoring
• Microsoft 365 / Google Workspace management and security audits
• VoIP phone systems
• Help desk

Pricing

No games, no “call for a quote” runaround:

Contact

• Phone: 719-530-4040
• Email: service@kccsonline.com
• Form: callthenerd.com

See more of what I run at modernsystemspro.com and pueblosecure.com.

The obvious diagnosis — the one most techs would reach for, and the one the symptoms practically beg you to believe — is a disk is dying, replace it.

That diagnosis would have made everything dramatically worse. Here’s why, and what was actually going on.

The symptom vs. the disease

The logs were screaming the same error, over and over, hundreds of times:

EXT4-fs warning (device md3): I/O error writing to
  inode 339687425, block 357888066
Buffer I/O error on device md3, logical block 357888066

Same block. Every single time. The filesystem wanted to write to one specific spot on the array and the write kept failing.

When a write to a RAID array fails on the same block forever, instinct says bad sector, failing drive. So the first thing we did was pull the health stats (SMART) off all seven drives.

Every counter that would indicate a failing disk — reallocated sectors, pending sectors, offline-uncorrectable sectors, cable/CRC errors — was zero. On all seven drives. The disks were pristine.

Healthy disks. A write that won’t complete. Same block forever. That contradiction is the whole story.

What’s actually a “bad block list”?

Linux software RAID (the md subsystem, the thing mdadm drives) has a feature called the Bad Block List, or BBL. The idea sounds reasonable: if the array ever has trouble with a specific spot on a member disk, instead of kicking the whole disk out, it just writes that spot down in a little list — “don’t use this block” — and routes around it.

Helpful in theory. In practice, it has been one of the most controversial features in the Linux RAID stack for about fifteen years. Kernel developers have openly argued for removing it. The reason is exactly what we walked into:

1. Something causes a transient I/O hiccup on a member — a momentary timeout, a controller blip, a drive that stalls for a fraction of a second under heavy write load. Nothing actually wrong with the media.
2. The BBL feature misreads that hiccup as a bad block and writes it down permanently.
3. On RAID5, where every stripe is spread across multiple disks, the bad-block entry gets propagated onto other members. Over time, the same logical region ends up flagged on several disks.

We confirmed this precisely. Four of the seven disks carried byte-for-byte identical bad-block ranges — and that is the tell:

       RAID5  ·  seven disks  ·  44 TB usable
  ┌─────┬─────┬─────┬─────┬─────┬─────┬─────┐
  │ sda │ sdb │ sdc │ sdd │ sde │ sdf │ sdg │
  ├─────┼─────┼─────┼─────┼─────┼─────┼─────┤
  │  ·  │ ▓▓▓ │ ▓▓▓ │ ▓▓▓ │  ·  │  ·  │ ▓▓▓ │
  │clean│ BBL │ BBL │ BBL │clean│clean│ BBL │
  └─────┴─────┴─────┴─────┴─────┴─────┴─────┘
           ╰──── identical phantom ranges ────╯
       real bad sectors do NOT line up to the exact
       same address across four separate drives.

Why this stops recording entirely

This is the part that turns a phantom bookkeeping error into a system outage.

To write one stripe of video, RAID5 has to write to several disks at once plus update parity. If even one of those disks says “that block is on my no-go list,” the entire stripe write fails and returns an I/O error:

  ONE stripe write ──► every data member must accept it
   ┌────┐┌────┐┌────┐┌────┐┌────┐┌────┐
   │ D0 ││ D1 ││ D2 ││ D3 ││ D4 ││ P  │
   └────┘└─🚫─┘└────┘└─🚫─┘└────┘└────┘
            │         │
            ╰── "on my no-go list" ──╯
                      │
                      ▼
        ✗  THE WHOLE STRIPE FAILS  →  EIO
        →  filesystem protectively goes read-only
        →  recorder drops to "limited mode"
        →  75 cameras stop saving video

The filesystem above sees the write fail, decides its metadata is now untrustworthy, and protectively remounts read-only. The camera software sees a read-only volume and drops into limited mode. Recording stops. All of it.

So: zero failing hardware, and a total recording outage, caused entirely by a list of imaginary bad spots the software refused to write over.

The trap

Now the dangerous part. The “replace the disk” reflex.

If you pull one of those drives and let the array rebuild onto a replacement, mdadm re-creates a bad-block list on the new member and re-propagates the same poison. You’d do four sequential multi-hour degraded rebuilds, each one re-importing the exact problem you were trying to remove, while running the array in its most fragile state. Replacing disks here doesn’t just fail to fix it — it actively risks the array.

The correct fix is the opposite of intuition: don’t touch the hardware. Erase the lists.

The repair

mdadm has an escape hatch for exactly this — a flag that clears (and disables) the bad-block feature across every member as the array is assembled:

mdadm --assemble --update=force-no-bbl /dev/md3 /dev/sd[a-g]5

The careful sequence around that one command:

1. Back up everything first. Recovery codes, configuration, and a full pre-incident database snapshot, all pulled off the box. Nothing to lose before touching the array.
2. Quiet the system. Stop every service touching the volume so nothing is mid-write. (This box had seventeen interlocking services and a storage daemon that re-assembles the array on its own.)
3. Unmount and stop the array.
4. Clear the bad-block lists with the command above. Afterward, every member reported the magic words:

  sda5  →  No bad-blocks list configured
  sdb5  →  No bad-blocks list configured
  sdc5  →  No bad-blocks list configured
  sdd5  →  No bad-blocks list configured
  sde5  →  No bad-blocks list configured
  sdf5  →  No bad-blocks list configured
  sdg5  →  No bad-blocks list configured
                                 ☁  poof. gone.

5. Check the filesystem. A read-only e2fsck -fn preview came back clean — only two trivial, optional defragmentation suggestions. Zero corruption, zero data to remove. That confirmed the “filesystem damage” was never real damage; it was just the blocked writes.
6. Reboot and verify.

The result: array healthy, zero I/O errors, bad-block lists empty, and within seconds of the services coming back, the full camera fleet was writing video again. Total surgery time: about an hour. Zero permanent data loss.

The gotcha worth its own paragraph

These appliances don’t assemble their array from a config file the way a generic Linux server does. A storage daemon brings the array up at boot. That means the daemon will happily re-assemble and re-mount the array out from under you in the middle of a repair if you don’t stop it — and it has to be running again before reboot or the box comes up with no storage at all. Finding and accounting for every one of those interlocking services (seventeen, including media-server processes that respawn instantly when killed) was most of the actual work. The headline command takes one second; safely getting the system into a state where that command is safe to run takes the hour.

The kicker

Once a system surprises you, the right question is: where else do we have this and not know it yet?

So we audited every storage system under management for the same fingerprint. Most weren’t even capable of this failure (different storage technology). One was clean. And one other recorder was sitting in the earliest stage of the exact same bug — two phantom bad-block ranges just beginning to form, still recording fine, no errors yet. Caught months before it would have become an outage. We added automated monitoring across the fleet that checks these lists every 30 minutes, so the next occurrence is a 30-minute alert instead of a two-day blackout.

Takeaways

• Symptoms lie. Health data doesn’t. The single most useful five minutes of this entire incident was reading SMART and seeing all zeros. That one fact ruled out the obvious-but-wrong diagnosis and pointed at the real one.
• The bad-block list is a footgun. If you run Linux software RAID, know this feature exists, know it can manufacture failures out of transient hiccups, and know --update=force-no-bbl is how you clear it. Consider monitoring for non-empty lists proactively.
• The intuitive fix can be the destructive one. “Replace the disk” would have deepened the hole. Sometimes the move is to change nothing physical and correct the software’s bad bookkeeping.
• Always back up before array surgery, even when you’re confident. Especially when you’re confident.
• Then go look at everything else. The bug you just fixed is rarely the only instance of it you own.

The hardware was never broken. The drives were never dying. A fifteen-year-old software feature, trying to be helpful, wrote down a problem that didn’t exist and then refused to work around its own mistake. Once you can see that, the fix is almost anticlimactic. Seeing it is the job.

A customer called on a Tuesday. A critical service had been down for hours. Their first question was the one that ruins your week:

“Why didn’t we get an alert?”

I had built the alerting. I knew the rules existed. I’d watched them fire in testing months ago. So my honest, confident, completely wrong answer was: “You should have.”

That’s the trap. “Should have” is what you say right before you learn something humbling about your own stack.

The investigation

First instinct: the rule didn’t fire. Easy theory, easy to check. So I pulled the alert history.

The rule fired. On time. With the right severity, the right labels, the right everything. It did its job flawlessly and then handed the alert off to… something.

┌──────────┐    ┌────────────┐    ┌──────────┐    ┌────────┐
│  RULE    │──► │  RECEIVER  │──► │ CHANNEL  │──► │ HUMAN  │
│ ✓ fired  │    │ ✓ matched  │    │  ✗ none  │    │  ✗ —   │
└──────────┘    └────────────┘    └──────────┘    └────────┘
                                   ▲
                                   │
                          the chain dies right here

The rule routed to a receiver. The receiver had a name, a config block, a place in the tree. It looked alive. And its list of channels — the email addresses, the webhook URLs, the push targets, whatever was supposed to carry the message to a person — was empty.

Not broken. Not misconfigured. Empty. The alert was delivered, perfectly, to nobody.

The “aha”

Here’s the part that stung. Everyone — me included — had validated the wrong layer.

We tested that rules existed. We tested that rules fired. We never tested that a fired alert reached a phone, an inbox, a pager, a chat channel — a person. The chain has four links; we’d verified two and assumed the other two by vibes.

An alert is not a notification. A rule firing is not someone finding out. The gap between those two facts is exactly where this outage lived — silent, for months — manufacturing a warm, false confidence that the system “had alerting.”

It did. The alerting just talked to a wall.

The fix

Audit every receiver. Every single one needs at least one channel, and that channel has to actually reach a human or a device. No empties, no dead webhooks, no stale email lists.

Then — and this is the part people skip — add a heartbeat. A “dead-man” alert that is supposed to fire on a schedule. If it stops arriving, your delivery path is broken and you find out on purpose instead of during a customer call.

# 1. Find every receiver with an empty / missing channel list.
amtool config routes show
amtool config show | yq '.receivers[] | select(
  ([.email_configs, .webhook_configs, .pushover_configs,
    .slack_configs, .opsgenie_configs] | flatten | length) == 0
) | .name'

# 2. End-to-end test: fire a synthetic alert and confirm it lands on a device.
amtool alert add deadmans_switch \
  severity=info service=alerting \
  --annotation summary="heartbeat — if you can read this, delivery works"

# 3. The dead-man heartbeat: a rule that is ALWAYS firing on purpose.
groups:
  - name: heartbeat
    rules:
      - alert: DeadMansSwitch
        expr: vector(1)
        labels: { severity: heartbeat }
        annotations:
          summary: "Alerting pipeline is alive. Silence = it isn't."

DELIVERY CHAIN ── post-fix
  RULE ─► RECEIVER ─► CHANNEL ─► DEVICE
   ✓        ✓          ✓ (≥1)     ✓ buzzed
        + DeadMansSwitch heartbeat every 60s ►► if it goes quiet, you know

Flip the logic. A normal alert proves a problem exists. A heartbeat proves the path itself still works. You want both, because the path is the thing that fails silently.

Why it happened

Someone — possibly past me — created the receiver as a placeholder, fully intending to fill in the channels later. Later never came. The config was syntactically valid, so nothing complained. There’s no error on an empty channel list; it’s a legal, boring state.

And because the rules were visibly there, every review after that pattern-matched “alerting: configured ✓” and moved on. Nobody walked the whole chain to a buzzing phone. The placeholder calcified into production.

Takeaways

• An alert with no destination is worse than no alert. No alerting at least keeps you honest. A dead receiver hands you false confidence and bills you for it during an outage.
• Validate the entire chain: rule → receiver → channel → device. Verifying any single link tells you almost nothing about the link after it.
• Test delivery, not configuration. “The rule exists” and “a human got paged” are different claims. Only the second one matters at 2 a.m.
• Add a dead-man heartbeat. A signal that should arrive on schedule turns silent failure into a loud, on-purpose one.
• Audit receivers for empties on a cadence. Placeholders, decommissioned inboxes, and rotted webhooks accumulate. Grep them out before they grep you.

A client’s user control panel popped a browser security wall. Expired certificate. Classic.

The panel runs on a FreePBX appliance, on a high port, separate from the main admin UI. Users hit it for voicemail and call history. Now they hit a full-page TLS warning instead.

No big deal, I figured. Renew the cert, reload, done before the coffee gets cold.

The coffee got cold.

The investigation

First I confirmed it wasn’t the browser lying to me:

openssl s_client -connect 192.0.2.10:8003 </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject
# notBefore=Apr  1 00:00:00 2024 GMT
# notAfter=Apr  1 00:00:00 2025 GMT

Dead for over a year. The panel had been quietly serving a corpse the whole time, and nobody noticed until a browser update got strict about it.

So I did the obvious things. The UI has a big friendly Apply Config button. Clicked it. Green checkmark. “Reload completed successfully.”

Re-ran my openssl check. Same expired dates.

Fine. Bigger hammer. From the CLI:

fwconsole reload
fwconsole restart
fwconsole ma updateall

Three clean exits. Three “success” lines. Three pieces of nothing — the panel kept serving the exact same expired cert, byte for byte.

That’s the moment the job stops being annoying and gets interesting. When every “fix” reports success and the resource never changes, you’re not fixing the resource. You’re poking something that doesn’t own it.

The “aha”

I drew the actual ownership chain — the thing nobody draws until they’re an hour in:

┌──────────────────────────────────────────────────────┐
│  reload / apply-config / updateall                    │
│      │                                                │
│      ▼  "rebuild my running config from what exists"  │
│  ┌───────────────────┐                                │
│  │  User Panel :8003 │  ── binds ──►  cert file ✗     │
│  └───────────────────┘               (expired, stale) │
│                                            ▲           │
│                                            │ OWNS      │
│                                  ┌───────────────────┐ │
│                                  │ Certificate Mgr   │ │
│                                  │  (lifecycle here) │ │
│                                  └───────────────────┘ │
└──────────────────────────────────────────────────────┘

The reload path rebuilds config from whatever certs already exist on disk. It never regenerates one. The thing that actually mints and renews the cert is the Certificate Manager module. Reload was faithfully re-binding the panel to a file that was still expired — and honestly reporting success, because re-binding did succeed. The cert was just garbage.

Nobody was renewing it because nobody told the component that owns renewals to do its job.

The fix

Stop driving the reload path. Drive the API of the thing that owns the cert.

The Certificate Manager exposes a regeneration call. Invoke it directly, let it reissue, then point the panel at the fresh cert and reload once to bind:

# Ask the module that OWNS the cert to reissue it
fwconsole certificates --regen-self-signed

# (or, when an ACME/managed cert is in play)
fwconsole certificates --update

# Now bind the panel to the freshly minted cert, then reload to apply
fwconsole certificates --default=<new_cert_id>
fwconsole reload

Then verify against the live socket — not the UI’s word, the actual TLS handshake:

openssl s_client -connect 192.0.2.10:8003 </dev/null 2>/dev/null \
  | openssl x509 -noout -dates
# notBefore=Jun  9 00:00:00 2026 GMT
# notAfter=Jun  9 00:00:00 2028 GMT   ✓

Fresh dates. Panel loaded clean. Coffee remade.

Why it happened

“Reload” and “apply config” are rebuild operations. They take the current declared state and re-render it. If the declared state references a cert file, they’ll happily wire it up — expired or not. Validity isn’t their job.

Renewal lives in exactly one place: the module that issues certs. Every other button just consumes the output. So every reload was a no-op on the resource, dressed up as a success because the binding step really did succeed.

The trap is that “success” was true at the wrong altitude. The command did what it was scoped to do. It just wasn’t scoped to the thing I needed.

Takeaways

• When every fix reports success but the resource never changes, you’re poking the wrong subsystem. Find the component that actually owns the resource’s lifecycle and drive its API.
• “reload” / “apply config” usually means re-render, not regenerate. They consume artifacts; they rarely create them.
• Trust the wire, not the UI. openssl s_client ... | openssl x509 -noout -dates against the live port is ground truth. A green checkmark is a claim.
• Map ownership before you start hammering buttons. One quick “who actually mints this?” diagram beats an hour of confident no-ops.
• Expired certs hide for months. Monitor notAfter on every TLS port — including the weird high-port sub-services nobody remembers exist.

The pager went off at the worst possible hour, which is the only hour pagers know.

Every app talking to the database was throwing the same line:

FATAL:  the database system is starting up

Not “down.” Not “connection refused.” Starting up. As if it were thirty seconds from being fine. It said that thirty seconds ago too. And thirty seconds before that.

The investigation

First instinct: restart it harder. Bad instinct. I sat on my hands.

The host had taken an ugly storage hiccup earlier — the array blipped, the kernel got cranky, and Postgres went down without a clean shutdown. So I went to the only source of truth that matters here: the server log.

sudo tail -f /var/lib/pgsql/data/log/postgresql-*.log

And there it was, crawling:

LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  redo starts at 3F/A2000028
LOG:  redo in progress, elapsed time: 412.06 s, current LSN: 3F/A2090110

Recovery was happening. It was just happening at the speed of continental drift. In seven minutes of replay it had moved the LSN by a rounding error.

That’s not a Postgres bug. That’s a disk gasping for air.

The “aha”

Here’s the thing every junior forgets at 3 a.m.: after an unclean stop, Postgres doesn’t just “boot.” It replays its write-ahead log to drag the data files back to a consistent state. Until that replay finishes, the server refuses connections — and the polite way it refuses is the database system is starting up.

        unclean shutdown
                │
                ▼
   ┌────────────────────────┐      reads WAL
   │  PostgreSQL  (startup)  │ ───────────────┐
   └────────────────────────┘                 │
                │                              ▼
                │ replays records      ┌───────────────┐
                │ onto data files      │   WAL (pg_wal) │
                ▼                       └───────┬───────┘
   ┌────────────────────────┐                  │
   │   data files on DISK    │ ◄────────────────┘
   └───────────┬────────────┘
               │  if DISK is slow/failing,
               ▼  replay crawls → "starting up" forever
        clients see: FATAL: starting up

WAL replay is only as fast as the disk it’s writing to. The storage hiccup hadn’t killed the disk — it had wounded it. Every read was retrying, every write was waiting. Recovery was real, honest, and effectively never going to end on that hardware.

The cardinal sin would’ve been to kill -9 the startup process to “speed things up.” Kill recovery mid-replay and you risk a data directory that’s half-applied and fully useless.

The fix

Rule one of database recovery: a database cannot recover on a broken disk. Fix the floor before you ask anyone to dance on it.

So I stopped Postgres cleanly, took the storage out of the equation, and verified the hardware first.

# 1. Stop the instance cleanly — let it park, don't kill -9
sudo systemctl stop postgresql

# 2. Check the storage BEFORE touching the database
sudo smartctl -a /dev/sda | grep -iE 'reallocated|pending|health'
dmesg --ctime | grep -iE 'i/o error|ata|reset'

# 3. Move the data dir onto healthy storage (rsync, preserve everything)
sudo rsync -aHAX --info=progress2 /var/lib/pgsql/data/ /srv/pgdata-healthy/

With the data directory living on a disk that wasn’t actively dying, I pointed Postgres at it and let recovery run — and left it alone.

sudo -u postgres /usr/bin/postgres -D /srv/pgdata-healthy &
sudo tail -f /srv/pgdata-healthy/log/postgresql-*.log

This time the LSN sprinted instead of crawled:

LOG:  redo in progress, elapsed time: 9.21 s, current LSN: 3F/F1A40020
LOG:  redo done at 3F/F1A40020
LOG:  database system is ready to accept connections

Twelve seconds of replay on good hardware versus an eternity on bad. Same WAL. Same database. Different floor.

And because I never fully trust a data directory that’s been through a disk event, the first thing I did once it was up was take a logical backup — the kind that doesn’t care about block-level corruption:

pg_dump -Fc -d appdb -f /srv/backups/appdb_$(date +%F).dump

Why it happened

The unclean shutdown was never the real problem. Postgres handled that exactly as designed — replay the WAL, return to consistency, open the doors.

The problem was that recovery was asked to run on storage that couldn’t keep up. WAL replay is I/O, and I/O on a wounded disk is mostly waiting. “Starting up forever” is what a healthy recovery process looks like when it’s standing on broken ground.

Fix the ground. The database knows what to do.

Takeaways

• “Starting up” forever almost always means WAL replay — Postgres is recovering from an unclean stop, not hanging for no reason.
• Watch the server log, not the client errors. redo in progress with a moving LSN means it’s working; a stalled LSN means your storage is the suspect.
• Fix the storage first. A database physically cannot finish recovery on a slow or failing disk — verify SMART/dmesg, move to healthy media, then let it replay.
• Never kill -9 recovery to “speed it up.” A half-applied data directory can be worse than no database at all.
• Keep logical dumps (pg_dump). A physical data directory can’t always be salvaged after a disk event — a logical backup is the restore path that survives bad blocks.

The page came in at 2:14 a.m. Then 2:16. Then 2:19.

By the time I’d wiped the sleep out of my eyes there were forty-one alerts in the channel, all from the same audio detector at a customer site. “LOUD EVENT DETECTED.” Forty-one of them. My first thought was that someone was kicking the building in.

Nobody was kicking the building. Someone closed a heavy door. Then, hours later, a kitchen dropped a sheet pan on a tile floor. Each one lit up the detector like a fireworks finale.

This is the story of a sensor that screamed at everything and meant nothing.

The investigation

The setup was simple on paper. A mic feeds audio into a small model that’s supposed to flag impact-type events — glass breaks, hard slams, the kind of thing you actually want a human to look at. It fires a webhook, and the webhook pages whoever’s on call.

In practice it fired on anything loud.

I pulled the trigger log and lined it up against what people swore actually happened that night.

┌──────────────────────────────────────────────────────────┐
│  TRIGGER LOG  vs  GROUND TRUTH                            │
├──────────────┬───────────────┬───────────────┬───────────┤
│  time        │ detector says │ peak level    │ reality   │
├──────────────┼───────────────┼───────────────┼───────────┤
│  02:14:07    │ ► IMPACT      │  ▓▓▓▓▓▓▓ 0.91  │ door slam ✗│
│  02:16:55    │ ► IMPACT      │  ▓▓▓▓▓▓  0.88  │ door slam ✗│
│  02:19:31    │ ► IMPACT      │  ▓▓▓▓▓▓▓ 0.93  │ dropped pan✗│
│  03:48:12    │ ► IMPACT      │  ▓▓▓▓▓   0.81  │ HVAC bang ✗│
│  (silent)    │   ——          │  ▓▓      0.34  │ real break✗│
└──────────────┴───────────────┴───────────────┴───────────┘
        every loud thing fired ── the one real event didn't

That last row is the one that made my stomach drop. There had been a genuine event earlier in the week — a real break — and it was quieter than a slammed door. The detector had ignored it.

The “aha”

So I read the actual trigger code. And there it was, in one branch:

if peak_amplitude > THRESHOLD:
    fire_alert("IMPACT")

That’s it. That’s the whole brain.

The model attached to this thing was a perfectly good audio classifier. It returned a predicted label and a confidence score for every clip — door, speech, glass_break, hvac, the works. And the trigger logic threw all of it away and looked at one number: was it loud?

Loudness is not meaning. A slammed door is loud and boring. A real break can be quiet and important. We’d built a smoke detector that goes off when you turn the lights on.

The fix

The model already knew what it was hearing. We just had to listen to the label instead of the volume — then gate on confidence and stop it from machine-gunning the same event.

ALERT_LABELS = {"glass_break", "impact", "alarm"}
MIN_CONFIDENCE = 0.82
DEBOUNCE_SECONDS = 30

_last_fire = 0.0

def handle(clip):
    label, confidence = classifier.predict(clip)   # what was it, how sure

    # 1) classify by LABEL, not loudness
    if label not in ALERT_LABELS:
        return

    # 2) confidence gate
    if confidence < MIN_CONFIDENCE:
        return

    # 3) debounce repeats within a window
    global _last_fire
    now = time.monotonic()
    if now - _last_fire < DEBOUNCE_SECONDS:
        return
    _last_fire = now

    fire_alert(label, confidence)

Then — and this is the part people skip — I refused to trust it until I’d measured it. We had that hand-labeled log from the noisy night, so I scored the new logic against ground truth before letting it page a single human again.

# replay the labeled clips through the new gate, compare to truth
python eval_detector.py \
    --clips ./ground_truth/clips/ \
    --labels ./ground_truth/truth.csv \
    --report precision_recall

# precision  0.94   (almost no false alarms)
# recall     0.88   (catches the real ones, including the quiet break)
# false-positives/night:  41  ->  1

Forty-one false alarms a night down to one. The quiet break? Caught.

Why it happened

The amplitude check was the first thing someone wrote — a five-minute proof of concept to confirm the mic and the webhook worked end to end. It worked. It shipped. The classifier got bolted on later for “labeling,” but nobody ever rewired the trigger to actually use it.

So the smart part rode along as a passenger while the dumbest possible heuristic drove. Classic. The proof of concept becomes production because it never visibly breaks — until 2 a.m. on a night with a heavy door.

Takeaways

• Loudness is not meaning. Amplitude tells you something happened, not what. If your model predicts a label, trigger on the label.
• Gate on confidence. A bare classification with no threshold is just a louder guess. Make the model commit before it pages a human.
• Debounce. One physical event becomes many samples. Collapse repeats inside a window or you’ll get a 41-message wall.
• Validate against ground truth before you trust it. Keep a hand-labeled event log and measure precision/recall. “Seems better” is not a number.
• Audit the trigger path, not just the model. A great classifier is worthless if a five-minute if loud: is still the thing pulling the trigger.

It was a Friday night and my brand-new health dashboard had an opinion.

In a tidy little card, glowing red, it announced: “More steps strongly correlates with WORSE sleep (r = -0.61).”

I stared at it. So the cure for insomnia is… sitting still? Cool. Great. I’d just spent a weekend wiring up Postgres, a correlation engine, and a slick dark UI to discover that exercise is bad for you.

The number was real. The correlation was real. The conclusion was complete garbage. And it took me an embarrassingly long evening to figure out why.

The investigation

The setup was simple. Four metrics per day — steps, sleep hours, resting heart rate, weight — pulled from the tracker’s export into a table. The engine did the obvious thing: loop over every pair of columns, run a Pearson correlation, sort by absolute value, surface the “strongest findings.”

First I assumed a bug. Sign flipped somewhere, units crossed, a join gone sideways. I pulled the raw rows.

SELECT day, steps, sleep_hours, resting_hr
FROM health_daily
ORDER BY day DESC
LIMIT 10;

The data was fine. High-step days really did have less sleep. The math wasn’t lying. The math was just answering a dumber question than I thought I’d asked.

Then I noticed the pattern in the rows. The big-step / bad-sleep days clustered. They were the days I remembered — the chaotic ones. Travel days. Deadline days. The days I was on my feet because everything was on fire.

The “aha”

There was a third variable sitting in the middle of the whole thing, and I’d never measured it: a stressful day.

A busy, stressful day makes me walk more (running around, pacing, errands) AND sleep worse (wired, late, anxious). Steps and sleep don’t touch each other. They’re both just symptoms of the same hidden cause.

                  ┌──────────────────────┐
                  │   STRESSFUL DAY      │   <- the thing I never logged
                  │   (the confounder)   │
                  └─────────┬────────────┘
                            │
                ┌───────────┴───────────┐
                ▼                       ▼
         ┌────────────┐         ┌──────────────┐
         │  + STEPS   │  ?????  │  - SLEEP     │
         └────────────┘ <-----> └──────────────┘
            no actual arrow between these two

The engine drew the dotted line at the bottom and called it a discovery. It had no idea the box at the top even existed.

And it got worse. With only a few weeks of data, every correlation was riding on a tiny sample — a couple of weird days could swing r wildly. On top of that I was testing every pair: 4 metrics is 6 comparisons, and the more pairs you test, the more likely pure noise hands you a juicy-looking number. Test enough things and “significant” findings appear for free.

Confounders, tiny samples, and many comparisons. Three different ways to manufacture confident nonsense, all firing at once.

The fix

I stopped trusting the engine and started guarding it. Four changes.

1. Require a real sample size. No pair gets reported under a floor.

MIN_N = 21  # three weeks minimum before we say a word

if pair.n < MIN_N:
    pair.verdict = "insufficient_data"
    continue

2. Correct for multiple comparisons. If you test 6 pairs, a raw p < 0.05 means almost nothing. Hold them all to a stricter bar.

from statsmodels.stats.multitest import multipletests

pvals = [p.pvalue for p in pairs]
reject, p_adj, _, _ = multipletests(pvals, alpha=0.05, method="holm")

3. Control for the obvious third variable. Partial correlation: hold the suspected confounder fixed and see if anything survives.

import pingouin as pg

# does steps-vs-sleep survive once we account for a stress proxy?
result = pg.partial_corr(
    data=df, x="steps", y="sleep_hours", covar="stress_proxy"
)
# the -0.61 collapsed toward zero once stress was held constant

4. Relabel the output. The card no longer says “correlates with.” It says “possible link — investigate.” Every finding ships as a hypothesis, never a verdict.

The steps/sleep “finding” failed three of the four gates and got demoted to a quiet maybe. Exactly where it belonged.

Why it happened

Because a correlation engine is a confidence machine with no judgment. It will faithfully compute a number for any two columns you hand it and present that number with the same authority whether it’s bedrock truth or coincidence between two symptoms of a Tuesday from hell.

The math was never wrong. My question was. “Are these two columns correlated?” is trivial. “Does one affect the other?” is a completely different question the engine was never equipped to answer — and I let the pretty red card pretend it had.

Takeaways

• Correlation engines on personal data manufacture convincing nonsense. A real r with a fake meaning looks identical to a true insight on a dashboard.
• Hunt the confounder first. If two things correlate, ask what unmeasured third thing could be driving both before you believe either causes the other.
• Tiny samples lie loudly. A handful of weird days can swing a correlation hard. Set a minimum-N floor and enforce it in code.
• Testing many pairs guarantees false hits. Correct for multiple comparisons (Holm, Bonferroni, FDR) or you’ll “discover” links that are pure noise.
• Ship correlations as hypotheses, never conclusions. Label the output “investigate,” not “proven.” Your dashboard’s job is to point, not to swear.

A user pings me: “The app won’t open anymore.”

Classic. I ask the usual: did anything change? “It updated this morning.”

Of course it did.

I remote in and double-click the icon. The cursor does the little spinny thing. The process shows up in Task Manager — CPU ticking, memory allocated, the whole act. But no window. Ever. It just sits there, alive and useless, like a server that boots to a blinking cursor and stops caring.

Kill it. Relaunch. Same thing. Process up, window absent. Reboot the machine. Same thing. Reinstall the app clean. Same. Thing.

That’s the part that bugged me. A fresh install hanging the exact same way means the problem isn’t the binary. The binary is fine. Something outside the binary is poisoning it.

The investigation

When a process starts but never paints a window, it’s usually stuck waiting on something — a lock, a handle, a chunk of state it expects to load. So I stopped staring at the app and started watching what it touched.

  Launch ───► Process starts ───► loads cached runtime bundles
                                         │
                                   ┌─────┴──────┐
                                   ▼            ▼
                            new v2 binary   OLD v1 cache  ✗
                                   │            │
                                   └─────┬──────┘
                                         ▼
                              version mismatch ► hang
                                         ▼
                                   ✗ no window

There it was. The updater swapped the binary to v2 but left a directory of cached runtime bundles from v1 sitting right where the new build looks first. New code, stale cache, two versions trying to share one brain. The app loads the old bundles, chokes on the mismatch, and waits forever for a handshake that’s never coming.

Reinstalling didn’t fix it because reinstalling doesn’t clear that cache. The installer drops a fresh binary and walks away. The landmine stays armed.

The aha

Here’s the tell that saved me: I deleted that cache directory and the app limped to life. Window appeared. It ran. Notably, clearing the cache needed no admin rights — it lives in the user profile, not Program Files. So this isn’t a “corrupt install, nuke from orbit” problem. It’s a “stale state” problem.

But “limped” is the operative word. Deleting the cache once gets you a window, but the app’s background helper service was still holding the old runtime state in memory. It’d rebuild the cache half-correctly and get flaky again. The durable fix was to make the helper rebuild from scratch.

The fix

Order matters. Kill everything first, then clear the cache, then restart the helper so it regenerates clean state, then launch.

# 1. Kill every process the app owns — no orphans holding the cache open
Get-Process -Name "TheApp*" -ErrorAction SilentlyContinue | Stop-Process -Force

# 2. Delete the stale runtime/bundle cache (user profile — no admin needed)
Remove-Item -Recurse -Force "$env:LOCALAPPDATA\TheApp\Cache\bundles" -ErrorAction SilentlyContinue

# 3. Restart the background helper service so it rebuilds state fresh
Restart-Service -Name "TheAppHelper" -Force

# 4. Confirm exactly one helper is up before relaunching
Get-Service -Name "TheAppHelper"

Then launch the app normally. Window paints, app stays up across reboots. Done.

If you skip step 1 and a stray process still has a handle on the cache dir, the delete silently no-ops and you’re right back where you started — wondering why a “fix” did nothing. Clear the orphans first, every time.

Why it happened

Auto-updaters are optimists. They assume the new binary and the old cached state are compatible, so they swap the executable and leave the rest in place to “save time.” Most of the time that’s fine. When the runtime bundle format changes between versions, it absolutely is not — and the app has no graceful path for “my cache is from the future’s past,” so it just hangs.

Reinstalling feels like the big hammer, but it only touches the binary. The poison was always in the user-profile cache the installer never looks at.

Takeaways

• A hang right after an auto-update is stale state, not a corrupt install. Clear the cache and restart the helper before you reinstall.
• A reinstall that changes nothing is a clue, not a dead end — it means the problem lives outside the install footprint.
• Cache/runtime dirs usually sit in the user profile — clearing them needs no admin, so try the cheap fix first.
• Kill orphan processes before deleting their files. A held handle turns your fix into a no-op and wastes an hour.
• Restart the background service, don’t just delete and pray. Deleting the cache gets a window; restarting the helper makes it stay.

You probably don’t hear them. I hear nothing else.

Misophonia means certain sounds don’t annoy me — they hijack me. So I did what any sysadmin does with a problem that won’t leave: I decided to make it someone else’s job. Specifically, a script’s. The pitch was simple. Detect the trigger sounds in any audio or video, scrub them out before playback, hand me back a clean track.

The pitch was simple. The first version was a disaster.

The naive approach

My first instinct was the dumbest possible one, which is usually where I start.

Triggers feel violent, so I assumed they were loud. Threshold the loudness, mute anything that spikes, done by lunch.

# v1: the volume-gate theory of everything
import numpy as np

def scrub_naive(samples, sr, thresh_db=-18.0):
    frame = int(0.02 * sr)               # 20ms frames
    out = samples.copy()
    for i in range(0, len(samples) - frame, frame):
        chunk = samples[i:i+frame]
        rms = np.sqrt(np.mean(chunk**2)) + 1e-9
        db = 20 * np.log10(rms)
        if db > thresh_db:               # "too loud" -> kill it
            out[i:i+frame] = 0.0
    return out

I ran it on a podcast. It muted the consonants. It muted laughter. It muted entire emphasized words. And the lip-smacks? Sailed right through, untouched, smug.

Because the triggers were never loud. A lip-smack sits below normal speech in raw energy. I’d built a machine that deleted the wrong things at the wrong volume for the wrong reason.

The aha

I pulled the waveforms apart and actually looked at them. That’s when it clicked.

A vowel is periodic and narrowband — energy stacked in tidy harmonic bands, sustained over time. A mouth click is the opposite animal: short, broadband, transient. A flat smear of energy across the whole spectrum, gone in 30 milliseconds.

   AMPLITUDE-OVER-TIME  (loudness lies)        SPECTRUM  (signature tells the truth)

   speech  ┌─────────────┐   click ┌─┐         speech ▁▃█▇▅▂   (harmonic, banded)
           │ /\  /\  /\   │        │█│         click  ▅▅▆▅▆▅   (flat, broadband)
   ────────┘/  \/  \/  \  └──┐ ────┘ └──        breath ▂▂▃▂▂▃   (broadband, low, hissy)
            loud + sustained    quiet + 30ms
                  ^ v1 chased THIS, the wrong axis

The trigger wasn’t a level. It was a shape. A fingerprint smeared across frequency and time. And “find this fingerprint in a stream” is not a thresholding problem.

It’s a classification problem. I’d been trying to solve a recognition task with a ruler.

The fix

So I threw out the ruler and described the signature instead. Per frame, extract the features that actually separate a click from a vowel — spectral flatness (how broadband it is), zero-crossing rate, onset sharpness, MFCCs. Then a small classifier outputs a probability, and only frames over a confidence threshold get touched.

import librosa, numpy as np

def features(chunk, sr):
    S = np.abs(librosa.stft(chunk, n_fft=512, hop_length=128))
    return np.concatenate([
        [librosa.feature.spectral_flatness(S=S).mean()],   # broadband?
        [librosa.feature.zero_crossing_rate(chunk).mean()],# transient hiss?
        librosa.feature.mfcc(y=chunk, sr=sr, n_mfcc=13).mean(axis=1),
    ])

def scrub(samples, sr, clf, p=0.85, debounce_ms=40):
    frame, out = int(0.02*sr), samples.copy()
    cooldown, hold = 0, int(debounce_ms/20)
    for i in range(0, len(samples)-frame, frame):
        prob = clf.predict_proba(features(samples[i:i+frame], sr)[None])[0,1]
        if prob >= p: cooldown = hold          # fire + arm debounce
        if cooldown > 0:                        # short cross-fade, not a hard cut
            out[i:i+frame] *= np.linspace(1, 0, frame) if cooldown==hold else 0.0
            cooldown -= 1
    return out

Two details earned their keep. The confidence threshold stops it firing on every sibilant s. The debounce keeps one click from flickering the gate on and off mid-word. And I duck with a short cross-fade instead of a hard zero, so the cut doesn’t itself become a click.

On isolated triggers — a click in a gap, a breath between sentences — it works. Validated against a hand-labeled set, and it catches the things that used to make me leave the room.

Why it happened

I anthropomorphized the problem. The sounds feel loud and aggressive to me, so I encoded my emotional read of them as a signal property. The DSP didn’t care about my feelings. Loudness and identity are different axes, and I’d built my whole v1 on the wrong one.

And I’ll be honest about the part that isn’t solved: triggers that land on top of speech. When a lip-smack overlaps a spoken word, ducking the frame mangles the word too. Source separation in real time is a genuinely harder problem, and pretending I’d nailed it would be the same mistake in a nicer suit. That’s phase two. For now the tool is honest about what it can and can’t pull apart.

Takeaways

• “Detect this sound” is classification, not thresholding. If you reach for a volume gate, ask whether the thing you want is actually defined by loudness — it usually isn’t.
• Define the signature in feature space. Spectral flatness, ZCR, onset, MFCCs separate a broadband transient from a harmonic vowel far better than amplitude ever will.
• Validate against labeled examples. “It feels right” is not a metric. A confidence threshold is only meaningful when you’ve measured it against ground truth.
• Debounce and cross-fade your edits. A hard cut to fix a click can introduce a new click. Smooth the seams.
• Be loud about the cases you can’t solve. Isolated triggers: handled. Triggers overlapping speech: not yet, and saying so is the whole point.

It started as a slow-afternoon habit: scan yourself the way an attacker would. Pull up Shodan, type in your own public IP, and brace.

I expected a clean sheet. UFW was configured. ufw deny on everything that wasn’t 80, 443, and the VPN. I’d checked it a dozen times. Green across the board.

Shodan disagreed.

There it was, indexed and timestamped: an admin/management panel answering on a port that was supposed to be firewalled into oblivion. Banner, title, the works. Anyone with a browser could’ve found it. Some of them probably had.

The investigation

First reaction: Shodan’s cache is stale. It happens. So I scanned from the outside, from a box that had no business reaching anything internal.

nmap -Pn -p 9000 192.0.2.10 --open   # from a cloud box, not my network
# (really: nmap -Pn -p 9000 <my-public-ip>)
nmap -Pn -p 9000 192.0.2.10

PORT     STATE  SERVICE
9000/tcp open   http

Open. From the public internet. Not stale.

So I SSH’d in and asked UFW what it thought it was doing.

sudo ufw status verbose | grep 9000
# (nothing — port 9000 is not allowed)
sudo ufw status | head
# Status: active
# 22/tcp   ALLOW   Anywhere
# 443/tcp  ALLOW   Anywhere

UFW was adamant: 9000 is closed. The internet was equally adamant: 9000 is open. One of them was lying, and firewalls don’t lie. They just get bypassed.

The “aha”

Then I remembered what was actually listening on 9000.

A Docker container.

docker ps --format '{{.Names}}\t{{.Ports}}'
# admin-panel   0.0.0.0:9000->9000/tcp

0.0.0.0:9000->9000/tcp. There it is. That -p 9000:9000 in the compose file. Docker doesn’t ask UFW for permission — it writes its own iptables rules, straight into the DOCKER chain, and they get evaluated before your tidy UFW INPUT rules ever run.

Your ufw deny is a polite note on the front door. Docker built a second door around back and propped it open.

  Inbound packet → :9000
        │
        ▼
  ┌───────────────────────────┐
  │ iptables nat/PREROUTING   │
  │  DOCKER chain (DNAT)  ◄────┼── Docker put this here.
  │  → 172.17.0.2:9000        │     UFW never gets a vote.
  └───────────────────────────┘
        │            ╲
        ▼             ╲ (never reached)
   container :9000     ┌────────────────────┐
                       │ ufw-user chain     │
                       │  DENY 9000  (moot) │
                       └────────────────────┘

The fix

The publish target was the bug. 0.0.0.0 means “the whole world.” It almost never should.

I bound the panel to loopback only and put it behind the reverse proxy with auth, reachable just through the VPN.

# docker-compose.yml — before
    ports:
      - "9000:9000"        # = 0.0.0.0:9000, published to the internet

# after
    ports:
      - "127.0.0.1:9000:9000"   # localhost only; proxy/VPN reaches it

docker compose up -d
docker ps --format '{{.Names}}\t{{.Ports}}'
# admin-panel   127.0.0.1:9000->9000/tcp

For anything that genuinely must survive the firewall, lock it at the DOCKER-USER chain — the one place Docker promises not to clobber:

# allow only the tailnet, drop everyone else for published container ports
sudo iptables -I DOCKER-USER -i eth0 -s 100.64.0.0/10 -j RETURN
sudo iptables -I DOCKER-USER -i eth0 -p tcp --dport 9000 -j DROP

Re-scan from outside:

nmap -Pn -p 9000 192.0.2.10
# PORT     STATE    SERVICE
# 9000/tcp filtered http

filtered. Dark. The panel only answers behind the tunnel now.

Why it happened

Nobody screwed up the firewall. The firewall did exactly what it was told. The trap is the mental model: people assume UFW sits at the front gate and inspects everything inbound. Docker doesn’t go through that gate — it cuts its own keyhole in iptables and your host rules never see the packet.

-p 9000:9000 reads like “expose this locally.” It actually means “publish this to 0.0.0.0.” One missing 127.0.0.1: prefix and an internal admin tool was on the public internet for who-knows-how-long, indexed by Shodan, waiting.

Takeaways

• Scan yourself the way an attacker would. Shodan plus an external nmap from a box outside your network is the only ground truth. Your config file is a hypothesis, not a result.
• Docker port publishing ignores your host firewall. -p PORT:PORT writes its own iptables rules and bypasses UFW entirely. ufw deny does nothing for published container ports.
• Bind admin services to 127.0.0.1 or a VPN, never 0.0.0.0. Always prefix the publish: 127.0.0.1:9000:9000. Default-public is a footgun.
• Use Docker-aware firewalling. If a container port truly needs outside reach, gate it in the DOCKER-USER chain — the one chain Docker won’t overwrite.
• Admin panels belong on localhost or a tunnel, behind a reverse proxy and auth. Convenient-but-public is just public.

The ticket said: “The server is possessed.”

That’s a direct quote. An app had been rock-solid for two years, then one Tuesday it “just stopped” — no config change, no deploy, no nothing. The on-call engineer had already rebooted twice, blamed cosmic rays, and was halfway to blaming the building’s wiring.

It wasn’t possessed. It never is.

After enough years you learn that 95% of “haunted network” tickets are the same seven mundane bastards wearing bedsheets. None of them are mysterious. All of them are one check away from being solved. The trick is running the check before you start theorizing about ghosts.

Here’s the scene I keep coming back to.

  THE GHOST                        THE CHECK              VERDICT
  ─────────────────────────        ──────────────────     ──────
  "service won't start"     ──►    who owns the port?     ✓ orphan
  "site loads wrong server" ──►    what does dig say?     ✓ stale DNS
  "one client 500s"         ──►    force --http1.1        ✓ h2 quirk
  "big transfers hang"      ──►    ping -M do -s 1472     ✓ MTU/MSS
  "intermittent dropouts"   ──►    arp -a (two MACs?)     ✓ dup IP
  "works one direction"     ──►    firewall state table   ✓ asymmetric
  "localhost can't connect" ──►    loopback fw rule        ✓ blocked pipe

The Investigation, Compressed Into a Checklist

1. The service that “can’t start.” Symptom: it crashed, you restart it, and now it refuses to bind. The aha: the crashed process orphaned the port and the kernel still thinks it’s owned. The check — find the squatter:

# Linux
ss -ltnp 'sport = :8080'
# Windows
netstat -ano | findstr :8080

Kill the orphan PID, the port frees, the service starts. No exorcism required.

2. DNS serving a corpse. Symptom: you migrated a site, half the world sees the new box, this one client sees the old one. The check — ask what’s actually resolving, not what should:

dig +short app.example.com @1.1.1.1
nslookup app.example.com
# flush the local liar:
sudo resolvectl flush-caches      # systemd
ipconfig /flushdns                # Windows

Stale cache, gone.

3. The one client that 500s while everyone else is fine. That’s an HTTP/2 downgrade or framing incompatibility. Pin the protocol and watch it heal:

curl -v --http1.1 https://app.example.com/health

If --http1.1 works and the default doesn’t, you found your ghost.

4. Small requests fine, big ones hang. Classic MTU/MSS mismatch — pings and tiny payloads sail through, the TLS handshake or a large transfer stalls forever. Probe for the real path MTU:

# 1472 + 28 overhead = 1500. Shrink until it stops fragmenting.
ping -M do -s 1472 192.0.2.1

If 1472 fails but 1400 works, clamp your MSS and the hangs vanish.

5. The intermittent flapper. Symptom: connectivity that drops for seconds at random — a rogue DHCP server or a static IP collision. Check the ARP table for one IP claimed by two MACs:

arp -a | sort        # two different MACs on the same IP = your culprit
ip neigh show

6. Works one way, dies the other. Asymmetric routing where the return path takes a different door and a stateful firewall drops it for having no matching session. Look at the state table:

# Linux conntrack
conntrack -L | grep 192.0.2.50
# pf
pfctl -ss | grep 192.0.2.50

No state entry for the return flow = there’s your “ghost.”

7. The app that “should be local” but can’t reach itself. A host firewall rule or blocked named pipe is eating loopback. Confirm the listener and that localhost itself is allowed:

ss -ltnp | grep 127.0.0.1
sudo iptables -L INPUT -n -v | grep -E 'lo|127.0.0.1'

That Tuesday “possession”? Number one. Crashed worker, orphaned port, ss -ltnp named the PID in four seconds. Ghost dispelled.

Why It Happens

None of these are exotic. They’re the natural failure modes of stateful systems: processes die untidily, caches outlive their truth, two devices want the same address, a firewall remembers a flow that no longer exists. The supernatural feeling comes entirely from partial symptoms — it works for some and not others, or for small things and not big ones. Partial failure reads as spooky. It’s just state you haven’t looked at yet.

Takeaways

• Run the seven-item checklist before you theorize. The boring answer is almost always right.
• “Some clients work” is a clue, not a contradiction — it points straight at DNS, HTTP/2, or MTU.
• “Small works, big hangs” is MTU/MSS until proven otherwise. Don’t waste an hour on app logs.
• Check state, not config. ss, dig, arp, and the firewall state table tell you what is, not what should be.
• Reboots hide orphans; they don’t explain them. Find the squatter PID before you cycle power and lose the evidence.

The ticket landed with a number attached, and the number was wrong.

A monitoring report — the one we generate so nobody has to think too hard — claimed a site was running 99 wireless access points. Ninety-nine. For a building that, last I checked, you could walk end to end during a coffee that hadn’t gone cold yet.

I’d been in that building. I knew the ceilings. There were not 99 APs up there. There weren’t even a quarter of that.

So either the site had quietly become an enterprise campus overnight, or the report was hallucinating. Spoiler: reports don’t hallucinate. They count exactly what you tell them to count, which is a different and worse problem.

The investigation

First rule when a number looks insane: don’t argue with the number. Go find ground truth and make the number argue with that.

I pulled the controller’s live device list — the actual adopted, talking-to-me-right-now hardware:

# adopted APs the controller is actively managing
curl -sk -b /tmp/controller.cookie \
  https://controller.example.internal/proxy/network/api/s/default/stat/device \
  | jq '[.data[] | select(.type=="uap")] | length'

12

Twelve. Twelve adopted access points, online, real, with MAC addresses I could go put my hand on.

So where did the report get 99?

I went and read the thing that built the report instead of the report itself. The collector wasn’t querying live adopted devices. It was scraping a broader chunk of the controller database — and it was counting rows, not radios you could hang on a wall.

The aha

Three lies were stacked on top of each other, and the report added them all up with a straight face.

  WHAT THE REPORT COUNTED            WHAT WAS ACTUALLY THERE
  ─────────────────────────          ───────────────────────
  ┌───────────────────────┐
  │ adopted APs        12 │ ◄──────── ✓ real, on the ceiling
  ├───────────────────────┤
  │ pending / forgotten   │
  │ ghost DB rows      ~30│ ◄──────── ✗ stale, never cleaned
  ├───────────────────────┤
  │ per-radio entries     │
  │ (2.4 / 5 / 6 GHz)     │ ◄──────── ✗ one AP counted 3×
  ├───────────────────────┤
  │ per-SSID rows         │ ◄──────── ✗ counted broadcasts
  └───────────────────────┘            as devices
        Σ = 99                          Σ (real) = 12

Every tri-band AP showed up as three radio entries. Every SSID broadcast left its own footprint. And the controller DB still held a graveyard of devices that had been swapped out, RMA’d, or “forgotten” in the UI months ago — rows that never actually leave the database, they just stop mattering. To a human. Not to a COUNT(*).

Stack the real twelve on top of the radio multiplication on top of the ghost rows, and you arrive — with perfect arithmetic and zero truth — at 99.

The fix

The fix wasn’t clever. It was narrowing the question. Count adopted UAP-type devices, deduplicated by MAC, and exclude anything pending or forgotten:

curl -sk -b /tmp/controller.cookie \
  https://controller.example.internal/proxy/network/api/s/default/stat/device \
  | jq '[.data[]
         | select(.type=="uap")
         | select(.adopted==true)
         | select(.state==1)          # 1 = connected
         | .mac]
        | unique | length'

12

Then I did the thing the database can’t: I walked the building and counted ceilings. Twelve.

The report now pulls that single reconciled number, and it carries a footnote: adopted, connected, deduped by MAC. If it ever disagrees with the live device list again, it’s the report that has to explain itself.

Why it happened

Because counting feels trivial, so nobody guards it. length on a JSON array looks like truth. But the controller DB isn’t a list of access points — it’s an event log of everything that ever claimed to be one: every radio, every SSID broadcast, every device that got swapped and never purged. Counting rows in that pile and calling it “APs” is like counting fingerprints and reporting the number of burglars.

The artifact survived for months because 99 was plausible-adjacent. Nobody who’d been on-site ever read the report; nobody who read the report had ever been on-site. The number floated in the gap between them, fat and unchallenged, until somebody finally looked up.

Takeaways

• Never report an inventory number you can’t reconcile to ground truth. A count needs two witnesses: the live device list and a physical walk.
• Rows are not things. Per-radio and per-SSID entries multiply one AP into many. Filter by device type and dedupe by MAC before you trust a total.
• Controller databases keep ghosts. Forgotten, pending, and RMA’d devices linger in the DB long after they leave the ceiling. Scope your query to adopted and connected.
• Plausible-adjacent is the dangerous zone. A wrong number that’s obviously insane gets caught. One that’s merely “a bit high” rides for months.
• Read the query, not the report. When a metric lies, the bug is upstream in what you told it to count — go fix the question.

The ticket said “internet is down.” Then, four minutes later, “never mind, it’s back.” Then, twenty minutes after that, “it’s down again.”

That pattern is a special kind of hell. A clean outage you can chase. A flapping one that heals itself before you SSH in just laughs at you.

I got a client on the phone. Their whole site would blackhole — every desktop, every phone, the printer, all of it — for one to five minutes, then snap back like nothing happened. No schedule. No trigger anyone could name. The ISP swore the circuit was clean, and for once the ISP was right.

The investigation

I started where you start: at the edge. The gateway was up. WAN was up. The firewall logs were boring in the way you want them to be boring. No flapping interface, no CPU spike, no DHCP storm in the leases.

So I sat on a workstation and just hammered the gateway while I waited for the next outage.

ping -i 0.5 192.0.2.1

It ran fine for ten minutes. Then — dead air. Eight, nine, ten dropped replies. Total blackhole. Then it came back on its own.

The instant it died, I dumped the ARP table.

arp -a 192.0.2.1

And there it was. The gateway’s IP was mapped to a MAC address I did not recognize. Not the firewall’s MAC. Some OUI I had to look up by hand.

I watched it flap in real time:

┌─────────────────────────────────────────────────────────────┐
│  arp -a 192.0.2.1   (watched over ~30s)                      │
├─────────────────────────────────────────────────────────────┤
│  t+0s   192.0.2.1  ►  aa:bb:cc:11:22:33   (firewall)   ✓     │
│  t+6s   192.0.2.1  ►  de:ad:be:ef:00:99   (???)        ✗     │
│  t+9s   192.0.2.1  ►  de:ad:be:ef:00:99   (???)        ✗     │
│  t+14s  192.0.2.1  ►  aa:bb:cc:11:22:33   (firewall)   ✓     │
│  t+19s  192.0.2.1  ►  de:ad:be:ef:00:99   (???)        ✗     │
└─────────────────────────────────────────────────────────────┘
         ▼ when the IP pointed at de:ad:..:99, traffic vanished

When the gateway IP pointed at the firewall, the network worked. When it pointed at that mystery MAC, every packet bound for the internet got handed to a device that did absolutely nothing with it. Classic ARP poisoning. The only question was who.

The “aha”

I looked up the OUI. The mystery MAC belonged to a Tuya-based Wi-Fi module — the kind that ships inside cheap smart-home gadgets by the millions.

Then I cross-referenced the DHCP leases for that MAC and found a hostname that made me laugh out loud: a smart window blind controller. Somebody had put a $30 motorized blind on the corporate flat network.

That little NIC was broadcasting unsolicited ARP replies claiming to be 192.0.2.1. Every client on the segment believed it, updated its ARP cache, and started shipping its default route straight into a window covering.

The fix

First, stop the bleeding. I pinned the gateway’s real MAC on the affected machines so they’d stop trusting the lie while I worked.

# Linux
sudo ip neigh replace 192.0.2.1 lladdr aa:bb:cc:11:22:33 dev eth0 nud permanent

Then the real fix — get IoT off the flat network entirely and turn on the switch protections that should’ve been on from day one.

BEFORE                              AFTER
──────                              ─────
VLAN 1 (everything)                 VLAN 10  desktops/servers
  ├─ desktops                       VLAN 20  voice
  ├─ servers                        VLAN 90  IoT  ◄── blind lives here,
  ├─ phones                                          firewalled, no L2
  └─ smart blind  ◄── poisoner                       reach to clients

# Cisco-style switch hardening (concept)
# Trust only the uplink for DHCP; drop rogue offers elsewhere
ip dhcp snooping
ip dhcp snooping vlan 90
interface Gi1/0/24
 ip dhcp snooping trust

# Dynamic ARP Inspection — validate ARP against snooping bindings
ip arp inspection vlan 90
interface Gi1/0/24
 ip arp inspection trust

Last step: I yanked the blind, pushed its vendor firmware update, and parked it on the isolated IoT VLAN where it can lie about being the gateway all it wants — and nobody who matters will hear it.

Why it happened

Cheap Tuya firmware is a grab bag. Some builds have buggy network stacks that emit malformed or unsolicited ARP — gratuitous ARP gone feral, sometimes triggered by a Wi-Fi reconnect, which explains the random timing. Could also be straight-up malicious; with no-name IoT you genuinely can’t tell, and it doesn’t change the remediation.

Either way, a flat L2 network trusts every device equally. ARP has no authentication. Whatever shouts loudest and last, wins. Put one mouthy gadget on that wire and it can pretend to be anything — including the door to the internet.

Takeaways

• On weird, intermittent, self-healing outages, check the gateway’s ARP entry FIRST. arp -a <gateway-ip> flapping to an unknown MAC is a five-second smoking gun.
• Match a rogue MAC to its OUI and your DHCP leases. That’s how a “mystery device” became “the smart blind in the break room.”
• IoT does not belong on your flat network. Ever. Its own VLAN/SSID, firewalled off from clients, no L2 reach to anything that matters.
• Turn on DHCP Snooping and Dynamic ARP Inspection. They exist precisely to kill rogue gateways and ARP lies at the switch.
• Cheap IoT is an untrusted host by default — bug or malice, you treat it the same: isolate, update, and never let it speak for the gateway.

The chat ping came in at 4:47 PM. “Hey, is the app down? All my stuff is gone.”

I pulled up the site. It loaded fine. Login worked. Dashboard rendered. And it was empty. No records. No users. No history. A perfect, pristine, brand-new install — on a server that had been running for fourteen months.

Nothing was broken. That’s what made my stomach drop. A crash leaves a corpse. This was worse. The app was alive and humming, cheerfully serving a database that had been born thirty seconds ago.

The investigation

First instinct: bad migration. Nope — no migrations had run. Second instinct: someone fat-fingered a DELETE. But the Postgres-shaped panic faded fast, because this app didn’t use Postgres. It used a SQLite file. A single file, living in the app directory.

That detail mattered more than I realized.

I checked the logs. The last thing that happened before the data vanished was a deploy. Routine. Boring. A dev had pushed a tiny CSS fix and run the deploy script, same as always.

So I opened the deploy script. And there it was, line nine, smiling at me like a loaded gun:

scp -r ./app/* deploy@198.51.100.10:/srv/app/

./app/*. The whole directory. Including app/data/app.db.

The dev’s local app.db was empty — fresh clone, never seeded. Every deploy had been copying that empty file up and stomping the live one. The only reason nobody noticed sooner was that this dev happened to have a blank local copy. Previous deploys from other machines had local data that masked the bug.

Here’s the shape of the disaster:

   DEV MACHINE                        PRODUCTION
 ┌──────────────┐                  ┌──────────────┐
 │ app/         │                  │ app/         │
 │  ├ main.py   │ ── scp -r ──►    │  ├ main.py   │
 │  ├ static/   │   "deploy"       │  ├ static/   │
 │  └ data/     │                  │  └ data/     │
 │     app.db   │ ════ EMPTY ════► │     app.db   │  ◄ 14 months
 │   (0 rows)   │    OVERWRITES    │   (was full) │     GONE
 └──────────────┘                  └──────────────┘

The fix wasn’t on the live disk. But the box ran nightly off-site snapshots, and the most recent one was eleven hours old. We restored that, replayed the day’s activity from request logs by hand, and lost almost nothing. Lucky. Stupid lucky.

The fix

The old script blanket-copied everything. The new one does three things it refused to do before: backs up prod data first, uses rsync with explicit excludes, and never touches the data paths at all.

#!/usr/bin/env bash
set -euo pipefail

HOST="deploy@198.51.100.10"
APP_DIR="/srv/app"
STAMP="$(date +%Y%m%d-%H%M%S)"

# 1. Back up prod data BEFORE we touch anything.
ssh "$HOST" "tar czf /srv/backups/data-${STAMP}.tar.gz \
    -C ${APP_DIR} data uploads"

# 2. Push CODE only. Exclude every path that holds state.
rsync -avz --delete \
    --exclude='data/' \
    --exclude='uploads/' \
    --exclude='*.db' \
    --exclude='*.sqlite' \
    --exclude='.env' \
    ./app/ "${HOST}:${APP_DIR}/"

echo "Deployed. Pre-deploy snapshot: data-${STAMP}.tar.gz"

Note --delete is now safe because the data dirs are excluded — rsync won’t reach in and prune data/ if it never syncs data/ in the first place. And the backup runs before the push, so even a botched exclude has a parachute.

Why it happened

scp -r ./app/* isn’t a bug. It does exactly what it says. The bug was a design decision nobody made on purpose: letting state live inside the deploy artifact. The database sat in the same folder as the code, so “deploy the code” silently meant “deploy the database too.”

The destructive path wasn’t unlikely. It ran on every single deploy. It just stayed invisible until a dev with an empty local copy pulled the trigger.

Takeaways

• Deploy code, not data. Application state — databases, uploads, user files — must live outside anything a deploy can overwrite.
• Any deploy that can touch the database is a loaded gun. Add explicit --exclude rules for every data and upload path, and verify them in a dry run (rsync -n).
• Back up before you deploy, not after. A snapshot taken the instant before the push is the cheapest insurance you’ll ever buy.
• Make the destructive path impossible, not just unlikely. “Be careful” is not a control. Excludes and pre-deploy backups are.
• Boring near-misses are gifts. We got lucky with an eleven-hour-old snapshot. Don’t design around luck — design so the gun can’t fire.

It was supposed to be a quick audit.

A client had asked me to sweep a private repo before they handed access to a new contractor. Routine. I cloned it, ran a scanner over the working tree, and started reading. Five minutes in, the scanner lit up red on a config helper.

A reusable credential. Plaintext. In the repo.

I felt that familiar little spike of righteous sysadmin adrenaline — who the hell committed THIS — opened the blame, and read the author line.

It was me.

The investigation

Here’s the part everyone gets wrong, and the part I almost got wrong too.

My first instinct was to delete the line, commit “remove secret,” and move on. Crisis averted, right?

Wrong. The file in the working tree is the least important place that secret lives. Git is a time machine. Deleting it today does nothing about the dozens of commits where it sits, fat and happy, in the history. Anyone with git log and ten seconds can walk straight to it.

I checked how deep it went:

git log --all --oneline -S 'the_secret_string' -- path/to/config
git rev-list --all --count

It went deep. The credential had been in the repo for months, across dozens of commits, surviving file renames and a refactor. Deleting the file would leave it perfectly intact in every parent commit.

  the working tree (what you see)
        │
        ▼
   ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐
   │ commit  │◄──│ commit  │◄──│ commit  │◄──│ commit  │
   │  HEAD   │   │  ~12    │   │  ~31    │   │  ~40    │
   │  ✗ gone │   │ ▓ secret│   │ ▓ secret│   │ ▓ secret│
   └─────────┘   └─────────┘   └─────────┘   └─────────┘
        ▲              └──────────────┬──────────────┘
   you "fixed" it          still right here, forever

The “aha”

The realization isn’t “I need to scrub this.” It’s earlier and uglier than that:

A secret is compromised the instant it’s committed. Not when the repo goes public. Not when someone clones it. At commit time. Every backup, every fork, every laptop that ever pulled, every CI cache — assume the secret is out. You cannot un-ring that bell by editing history.

So scrubbing history is necessary, but it is not step one. Step one is to make the leaked value worthless.

The fix

1. Rotate, immediately. Before anything else, I logged into the provider, revoked the credential, and issued a fresh one. The old value is now a dead string. Whoever has it has nothing.

2. Purge it from history. With the value already dead, I scrubbed it so it stops scaring the next auditor (me, in six months). git filter-repo is the modern tool; BFG works too.

# install once: pip install git-filter-repo
echo 'the_secret_string==>REDACTED' > replacements.txt
git filter-repo --replace-text replacements.txt

# then force the rewritten history out
git push --force --all
git push --force --tags

Every collaborator re-clones after this. A force-push that rewrites history will wreck anyone’s local copy — warn them first.

3. Move secrets out of the repo for good. The config now reads from the environment, injected at runtime from a secrets manager. Nothing sensitive touches the tree.

# .env stays OUT of git
echo '.env' >> .gitignore

# app reads from the environment, value lives in the secrets manager
export API_TOKEN="$(vault kv get -field=token secret/app)"

4. Make it impossible to repeat. A pre-commit hook scans every staged change so a secret never reaches a commit in the first place.

# .pre-commit-config.yaml
#  - repo: https://github.com/gitleaks/gitleaks
#    rev: v8.18.0
#    hooks: [{ id: gitleaks }]
pre-commit install
gitleaks detect --source . --verbose

Why it happened

No villain here. Just the most ordinary mistake in the trade.

Months ago, mid-deploy, I needed the thing to work now. I hardcoded the credential “just to test,” told myself I’d pull it out before committing, got interrupted, and committed everything with a git add .. The TODO never came back. The repo was private, so it felt safe — and “private” is exactly the lie that lets these things rot in history for half a year.

Private isn’t a control. It’s a setting someone can flip.

Takeaways

• A secret is burned the moment it’s committed. Rotate first, always. Treat the value as public from commit time forward — because effectively it is.
• Deleting the file is not remediation. The credential lives in every old commit. You must rewrite history (git filter-repo / BFG) and force-push, or it’s still there.
• Secrets belong in a manager, not the tree. Env-injected values, .gitignore your .env, and never let credentials and code share a home.
• Automate the catch. A gitleaks/trufflehog pre-commit hook stops the next leak before it’s a commit — humans forget, hooks don’t.
• “Private repo” is not a security boundary. Audit your own repos like you’d audit a stranger’s. The author line might surprise you.

The Afternoon the Screaming Got Faster

You remember the day.

The beige box showed up.
Smaller than you expected.
A little plastic brick with a single blinking light, riding inside the family computer like a smuggled treasure.

56k.

You didn’t know what a “k” was.
You just knew the old one was a “33.6” and this one was a bigger number, and bigger numbers meant good.

Dad said it like a magic word.
“We got 56k now.”

The grown-ups nodded. Somebody whistled.
You had no idea what just happened, but you knew it was a Big Deal, the kind of Big Deal that came with a receipt and a warning not to touch anything.

Here’s what we thought it was:

• a machine that could go faster if you believed hard enough
• powered by a sound — that holy shriek, that robot gargling underwater
• forbidden during dinner, sacred after homework
• somehow connected to the telephone, which meant Grandma could ruin everything by picking up to call about a casserole

You’d lean in close while it connected.
Bee-doo. Bee-doo. Skreeeee. Kshhhhhhhh.
And you’d hold your breath like the noise might notice you and stop.

Here’s what it actually was.

That beautiful screaming was a handshake — two modems negotiating, out loud, in real time. Your modem and the one at the other end were literally singing tones at each other to figure out the cleanest way to talk over a copper phone line built for voices, not data.

The 56k part? A gorgeous little cheat. The phone network had quietly gone digital in the middle, so your downloads could ride that clean digital signal almost all the way to your house — that’s how you squeezed 56 kilobits a second out of a wire designed for chitchat. (Uploads were stuck slower. The magic only ran one way.)

And that’s why it tied up the phone: your modem was modulating and demodulating — turning ones and zeros into sound and back again — using the exact same line as the kitchen phone. One line. One conversation at a time. The internet literally was a phone call.

  +-----------------------------+
  |  U.S. Robotics  56K        o |   <- the one blinky light
  |  [CD][RD][SD][TR][MR][AA]    |
  +--__----------------------__--+
     ||  bee-doo  skreeee     ||
   ~~||~~~~~~~~~~~~~~~~~~~~~~~~||~~
     ||   ___        ___      ||
     \\__/   \______/   \____ //   <- the handshake, screaming

Sure, the picture loaded one agonizing horizontal strip at a time, top to bottom, and somebody always picked up the phone right before the last strip. Sure, “going online” was an event you scheduled around. But for one shining afternoon, you were the fastest house on the block.

And here’s the thing — that handshake never really died.

Your phone still negotiates with a cell tower. Your laptop still does a little dance with the Wi-Fi router. Every “connecting…” spinner you’ve ever cursed at is the same conversation, just gone silent. We made it faster, we made it invisible, we even made it wireless — but somewhere under all of it, two machines are still saying hi, how do we talk? before they trust each other with your data.

We just don’t get to hear it scream anymore.

And honestly?

We kind of miss it.

A field tech messaged me from a basement with no signal: “Your app just hangs. White screen. Forever.”

That’s a fun one to read on a Tuesday, because the entire point of that app was that it worked offline. Progressive Web App. Service worker. Cached shell. The whole pitch was “open it anywhere, signal or not.”

So I did what he did. I put my phone in airplane mode and tapped the icon.

Blank screen. Spinner that never spun. The app didn’t crash — it just waited. Patiently. Forever. Like a golden retriever staring at a door that’s never going to open.

The investigation

On a normal connection, everything was perfect. Load times snappy, cache populated, Lighthouse happy. The bug only showed up with the network truly gone — not slow, gone.

That asymmetry is the tell. If something works online and dies offline, you stop blaming the app and start blaming whatever sits between the app and the cache. For a PWA, that’s the service worker’s fetch handler.

I pulled it up. Here’s the offending strategy, paraphrased:

// service-worker.js — the original sin
self.addEventListener('fetch', (event) => {
  event.respondWith(
    fetch(event.request)                 // always try the network first
      .then((res) => {
        const copy = res.clone();
        caches.open('app-shell').then((c) => c.put(event.request, copy));
        return res;
      })
      .catch(() => caches.match(event.request))  // fall back to cache
  );
});

Network-first. Looks reasonable. Hit the network, cache the response, and if the network fails, serve from cache. The .catch() is right there. So why doesn’t it fall back?

The “aha”

Because offline doesn’t fail. Not quickly, anyway.

When you’re truly offline, that fetch() doesn’t reject in 20 milliseconds with a clean “no route to host.” Depending on the platform, it can sit there for tens of seconds — or effectively never resolve — before the OS decides it’s done. The .catch() is waiting for a rejection that takes its sweet time. Meanwhile, the app shell never paints, because the navigation request for index.html is stuck in that same limbo.

The cache was fine. The fallback logic was fine. The problem was that the fallback never got to run, because nothing told the network attempt to give up.

       ONLINE                          OFFLINE (the bug)
  ┌──────────────┐                ┌──────────────────────┐
  │  fetch()     │                │  fetch()             │
  │     │        │                │     │                │
  │     ▼ ~80ms  │                │     ▼                │
  │  ✓ response  │                │   ...waiting...      │
  │     │        │                │   ...waiting...      │
  │     ▼        │                │   ...waiting...   ▓  │
  │  cache.put   │                │   (never rejects) ▓  │
  │     │        │                │     │             ▓  │
  │     ▼        │                │     ✗ .catch()       │
  │  paint ✓     │                │       never fires    │
  └──────────────┘                └──────────────────────┘
                                   app shell never paints

The fix

Two changes. First, never let a fetch in the service worker run unbounded — race it against a timeout so it fails fast and the cache fallback actually fires:

function fetchWithTimeout(request, ms = 3000) {
  return Promise.race([
    fetch(request),
    new Promise((_, reject) =>
      setTimeout(() => reject(new Error('sw-fetch-timeout')), ms)
    ),
  ]);
}

Second, stop being network-first for the app shell. Serve from cache immediately, then refresh the cache in the background — classic stale-while-revalidate:

self.addEventListener('fetch', (event) => {
  event.respondWith(
    caches.open('app-shell').then(async (cache) => {
      const cached = await cache.match(event.request);

      const network = fetchWithTimeout(event.request)
        .then((res) => {
          cache.put(event.request, res.clone()); // refresh in background
          return res;
        })
        .catch(() => cached);   // offline or slow? cached is already in hand

      return cached || network; // paint instantly if we have it
    })
  );
});

Now the shell paints from cache on the first frame, online or off. The network update happens behind the scenes, and if it’s slow or absent, the timeout cuts it loose in three seconds instead of never.

Airplane mode, tap the icon, instant load. The basement tech got his app back.

Why it happened

Network-first feels safe — “always get the freshest thing, fall back to cache if it breaks.” But it quietly assumes the network fails promptly. Offline doesn’t fail promptly. It stalls. And a stalled promise with no timeout is just an infinite loading screen wearing a .catch() as a disguise.

The app shell — the HTML, JS, CSS that make the thing exist — should never depend on a live connection to render. That’s the entire premise of “offline-capable.” We’d built the cache and then put a blocking network call in front of it.

Takeaways

• Never let a service-worker fetch run without a timeout. Promise.race it against a setTimeout reject so offline fails in milliseconds, not minutes.
• Serve the app shell cache-first or stale-while-revalidate — never network-first. The UI should paint from cache on the first frame, every time.
• Test with the network truly gone, not just throttled. Slow networks reject; dead networks stall. They’re different failure modes and only the dead one exposes this bug.
• A .catch() is not a fallback if nothing triggers the rejection. Error handling only runs when the error actually fires — and “offline” often doesn’t fire fast.
• “Offline-capable” is a claim you have to test in airplane mode, not a checkbox you tick because you registered a service worker.

The cron job ran at 11 PM. By the time I poured coffee the next morning, it had called a paid LLM API a few thousand times.

Nobody touched it. Nobody approved it. It just sat there in the dark, retrying, retrying, retrying — like a vending machine eating a stuck dollar bill, except the dollar bill was billable tokens.

I didn’t find it because of a clever alert. I found it because the provider dashboard had a graph that looked like a cliff face.

The scene

I run a small fleet of unattended automations for a client. One of them enriches records overnight by sending them through a metered LLM endpoint. Cheap per call. Boring. The kind of job you set up once and forget — which is exactly the problem.

That night, the upstream service it depended on got flaky. A handful of calls returned errors. The loop did what badly-written loops always do.

It tried again. Immediately. Forever.

┌────────────────────────────────────────────────────┐
│  THE LOOP (as written)                             │
│                                                    │
│   ┌──────────┐   error   ┌──────────────┐          │
│   │ call API │ ────────► │ retry now    │          │
│   └────┬─────┘           └──────┬───────┘          │
│        ▲                        │                  │
│        └────────────────────────┘                  │
│        no backoff · no cap · no kill switch        │
│                                                    │
│   result: ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  thousands of hits │
└────────────────────────────────────────────────────┘

No sleep. No max-retry counter. No circuit breaker. Every failure became an instant request, and every instant request became another line on the invoice.

The investigation

First thing: confirm it’s us and not a stolen key. I pulled the provider’s usage view and bucketed by hour.

# usage exploded between 23:00 and 07:00 — exactly the cron window
# logs from the job host told the same story
journalctl -u record-enrich.service --since "yesterday 23:00" \
  | grep -c "POST /v1/"
# -> 4,300-something. Overnight. For a job that should make ~80 calls.

Same source IP. Same user-agent. Same service. Not a breach — a self-inflicted wound. Somehow that’s worse, because it means the call was coming from inside the house.

The “aha”

The smoking gun was four lines of code. Paraphrased:

while not done:
    try:
        resp = client.complete(payload)
        done = True
    except Exception:
        continue   # <-- the whole disaster, right here

continue. No delay. No ceiling. The instant the upstream hiccuped, this turned into a tight spin loop firing paid requests as fast as the network would carry them. A retry without backoff isn’t resilience — it’s a denial-of-wallet attack you launch against yourself.

The fix

I treated the key as compromised even though it wasn’t, because the behavior was indistinguishable from a leak. Containment first, blame later.

# 1. Rotate the key immediately — old one dies on the spot
provider keys rotate --name record-enrich --revoke-old

# 2. Disable the API entirely at the provider while I fix the code.
#    Kill the bleeding before patching the artery.
provider api disable --service record-enrich

# 3. Re-enable WITH a hard cap + budget alert. The cap is the real fix.
provider budget set --service record-enrich \
  --hard-limit-usd 25 --period monthly
provider alerts set --service record-enrich \
  --notify-at 50% --notify-at 90% --channel telegram

Then the code got the guardrails it should have shipped with:

import time

MAX_RETRIES = 5

for attempt in range(MAX_RETRIES):
    try:
        resp = client.complete(payload)
        break
    except TransientError:
        time.sleep(min(2 ** attempt, 30))   # exponential backoff, capped
else:
    raise RuntimeError("gave up after retries — failing loud, not looping")

Backoff. A retry ceiling. And a loud failure instead of a silent infinite spin. The provider cap is the seatbelt; this is actually steering the car.

Why it happened

It happened because “it’s just a small overnight job” is the exact mindset that ships uncapped loops at paid endpoints. The cost-per-call was trivial, so nobody did the multiplication. Trivial times infinity is still a number you have to pay.

The code had no concept of “too many.” The provider had no concept of “enough.” With neither a ceiling in the app nor a ceiling at the wallet, the only limiter left was how fast the network could move — and that’s a throttle on the damage, not a budget.

Takeaways

• Put a hard spend cap at the provider. It’s the only limit a runaway loop can’t out-code. Set it before you write the first request.
• Backoff and a max-retry count are not optional. A retry without delay or ceiling is a self-DoS. continue on an exception is a loaded gun.
• Wire usage alerts at 50% and 90%. You want a phone buzz at midnight, not a graph shaped like a cliff at breakfast.
• Treat weird usage as a leak until proven otherwise. Rotate the key and disable the endpoint first; debug the code second. Containment beats curiosity.
• “Small overnight job” is a smell. Anything unattended that touches a metered API gets a kill switch, a cap, and an alert — or it doesn’t get deployed.

It started, like most of my regrets, with a bookmark folder.

app-one:3000. dashboard:8501. that-thing-i-forgot:8080. A dozen self-hosted apps, each squatting on its own port, each throwing a browser security warning because I’d long since stopped pretending to manage individual certs. Half were self-signed. Half were plain HTTP. All of them were ugly.

Every new app was the same ritual: pick a port, pray it’s free, wire up TLS by hand, give up on TLS, add another :PORT bookmark, lose it. The homelab worked. It just looked like a ransom note.

So I sat down to do it properly. Once.

The investigation

The core problem wasn’t the apps. It was that I had N apps and N TLS problems. Every service believed it was personally responsible for the open internet. None of them were good at it.

Here’s the mess, drawn honestly:

        BEFORE                              AFTER
  ┌──────────────────┐            ┌──────────────────────────┐
  │  browser         │            │  browser                 │
  └───┬───┬───┬──────┘            └──────────┬───────────────┘
      │   │   │   (which port?)              │  one name, valid TLS
      ▼   ▼   ▼                              ▼
   :3000 :8501 :8080                    ┌─────────┐
   self-signed / http / ???            │  Caddy   │  *.lab.example.tld
   each app owns its own cert          └──┬──┬──┬─┘  wildcard cert
                                          │  │  │
                                          ▼  ▼  ▼
                                127.0.0.1:3000/8501/8080
                                  (apps bound to localhost)

The fix wasn’t to make every app better at certificates. It was to make none of them do certificates. Put one thing in front. Let it own TLS, routing, and renewal. Bind everything else to 127.0.0.1 so the only thing facing the world is the proxy.

That “one thing” is a reverse proxy. Mine’s Caddy.

The “aha”

The trick that makes it sing is a wildcard certificate — *.lab.example.tld, one cert that’s valid for every subdomain I’ll ever invent — obtained over a DNS-01 challenge instead of the usual HTTP-01.

DNS-01 matters here. HTTP-01 proves you own a name by serving a file on it, which means each name needs to be reachable. DNS-01 proves it by dropping a TXT record at your DNS provider, which means you can mint a wildcard without exposing a single app first. One challenge, infinite names.

Once that clicked, the whole bookmark folder collapsed into one file.

The fix

Caddy needs a DNS-provider plugin to answer the DNS-01 challenge, so build it with the module for your provider:

xcaddy build \
  --with github.com/caddy-dns/cloudflare

# verify the module is in there
./caddy list-modules | grep dns.providers

Then the global block tells Caddy how to get the wildcard, and each app is — genuinely — three lines:

# Caddyfile
{
    email admin@example.tld
    # credentials for the DNS-01 challenge
    acme_dns cloudflare {env.CF_API_TOKEN}
}

# the wildcard, requested once, renewed forever
*.lab.example.tld {
    tls {
        dns cloudflare {env.CF_API_TOKEN}
    }
}

app-one.lab.example.tld {
    reverse_proxy 127.0.0.1:3000
}

dashboard.lab.example.tld {
    reverse_proxy 127.0.0.1:8501
}

Reload without dropping connections:

caddy validate --config /etc/caddy/Caddyfile
caddy reload  --config /etc/caddy/Caddyfile

Confirm a fresh subdomain actually got real HTTPS:

curl -sI https://dashboard.lab.example.tld | head -n1
# HTTP/2 200
echo | openssl s_client -connect dashboard.lab.example.tld:443 2>/dev/null \
  | openssl x509 -noout -subject
# subject=CN = *.lab.example.tld

Adding the next app is now: pick a port, add a three-line block, caddy reload. Valid cert, no thought required.

Why it happened

The old setup wasn’t dumb, it was just additive. Each app got stood up in isolation, made its own peace with HTTPS, and got a port-shaped bookmark. Nobody ever sat down to factor out the part every app shares: a public face and a certificate.

A reverse proxy is exactly that factoring. TLS, hostname routing, and renewal are cross-cutting concerns — solve them once, in front, and every app inherits the solution for free. The wildcard cert is what makes “inherits for free” literally true: no per-app issuance, no per-app renewal, no per-app anything.

Takeaways

• One TLS problem beats N TLS problems. Put a reverse proxy in front and let it own certs, routing, and renewal so your apps never have to.
• Wildcard + DNS-01 is the unlock. *.domain minted via a DNS TXT challenge covers every subdomain you’ll ever invent — and you don’t have to expose an app to prove ownership.
• Bind apps to 127.0.0.1. If only the proxy faces the world, a misconfigured app can’t accidentally serve itself raw to the internet.
• New app = three lines. A subdomain block that reverse_proxys to a localhost port, a reload, done. Exposing a service stops being a project.
• Build the right Caddy. DNS-01 needs the provider plugin compiled in — xcaddy build --with, then list-modules to confirm before you wonder why the challenge hangs.

The machine that screamed your homework into existence

You remember the dot-matrix printer.

The big beige one.
On the floor, usually. Too heavy for the desk.
With the green-and-white striped paper that fed up out of the back in one endless accordion ribbon.

And the noise.
Dear god, the noise.

It didn’t print so much as it announced. A grinding, buzzing, sawing shriek that meant somebody, somewhere in the house, was getting their book report out the door at 9:48 PM the night before it was due.

You’d stand there next to it, hypnotized.
Watching the little head zip left.
Then right.
Then nudge the paper up a hair.
Then left again.

The vibes were:

• Forbidden. You were NOT allowed to tear the paper until it finished. Do not touch it.
• Sacred ritual. The little holes down the sides. The perforated edges you peeled off after, like the wrapper on a candy.
• Tension. Would it jam? It always might jam.
• Loud = important. If it was screaming, real work was happening.
• A faint smell of warm plastic and ink ribbon.

And those side strips. The pin-feed holes. You’d peel both ribbons off in one clean pull if you were lucky, and it felt like completing a quest.

What it actually was

Here’s the nerdy part, and it’s genuinely cool.

That print head wasn’t spraying ink. It had a tiny grid of stiff little pins behind it. Tiny metal needles. As the head slid across, an electromagnet would fire individual pins forward, hammering them into an inked ribbon, which slapped that ink onto the paper.

A character was just a pattern of dots punched out fast. Nine pins. Sometimes twenty-four if you were fancy. Impact printing — literally tiny hammers, hundreds of times a second. That’s where the scream came from.

And the green stripes and side holes? That was continuous-feed tractor paper. The sprocket holes hooked onto little spinning gears so the printer could pull a whole novel through without anyone reloading a single sheet. The machine fed itself.

The reason you couldn’t tear it early: the paper was under tension on those gears. Yank it and you’d skew the whole job.

   _______________________________________
  |  o  ___________________________   o  |
  | o  | E r r o r   Z a p . . . . |   o |
  | o  | ......  ...  .. .... ...  |   o |
  | o  | .. ...  .... ... .  ....  |   o |
  | o  |___________________________|   o |
  |  o ____________________________   o  |
  | o ||||||| BZZZT-GRRRT-ZZZP ||||   o |
  |__o___________________________ __ o__|
   '-.___pin-feed___.-' '-.___holes_.-'

It never really left

You think it’s gone. It isn’t.

That receipt curling out of the gas pump?
The slip the doctor’s office prints your appointment on?
The carbon-copy form at the auto shop?

Impact printing. Still alive. Because here’s the secret a laser can’t do: hammering through paper lets you print carbon copies in one pass. That’s why warehouses, garages, and pharmacies never gave it up.

So the next time something prints with a buzz instead of a hum, lean in.

That’s the old beige monster.
Still screaming.
Still working.

Don’t tear the paper yet.

It started with a boring task. Quarterly DNS audit. Coffee, a zone file, and a vague sense that everything was fine.

It was not fine.

Halfway down the export I hit a CNAME — a customer-facing subdomain — pointing at a cloud host we’d decommissioned back in the winter. The instance was gone. The DNS record had never gotten the memo.

I sat there for a second. That subdomain still resolved. It still answered. It just answered nothing — a hosting account that no longer existed, on infrastructure we’d long since stopped paying for.

That’s not a typo. That’s a loaded gun pointed at your own brand.

The investigation

A dangling DNS record is a pointer with no backing store. The record says “this name lives over there.” But “over there” got freed months ago — the VM was destroyed, the cloud bucket released, the hosting account closed.

The danger isn’t that it’s broken. The danger is that the resource is now available for anyone to claim.

I pulled the full zone and started checking targets one by one. Does the A record’s IP still belong to us? Is the CNAME target a hostname we still control? Or is it a now-orphaned cloud resource sitting in a provider’s free pool, waiting for the next person to grab it?

        Your DNS                      The Internet
   ┌──────────────────┐         ┌────────────────────────┐
   │ app.example.com  │         │  cloud host (DELETED)   │
   │  CNAME ──────────┼────────▶│  account closed, freed  │
   └──────────────────┘         │  ▒▒▒ up for grabs ▒▒▒  │
                                └───────────┬────────────┘
                                            │  attacker
                                            ▼  re-claims it
                                ┌────────────────────────┐
                                │  serves THEIR content   │
                                │  on YOUR subdomain      │
                                └────────────────────────┘

The “aha”

Here’s the part that turns a stale record into a breach.

If an attacker registers that freed cloud resource — same provider, same dangling target — the provider hands them the keys. Now your subdomain serves their content. And because it’s your subdomain, the blast radius is ugly:

• Pixel-perfect phishing on a domain your users already trust.
• Cookies scoped to *.example.com get harvested — including session cookies from your real apps.
• A free, fully valid TLS certificate, because the takeover host can pass the domain-control check for a name you literally pointed at it.

No CVE. No exploit chain. Just a DNS record that outlived its server.

The fix

First, inventory everything. Pull the whole zone, not just the records you remember.

# Dump every A / AAAA / CNAME in the zone
dig @ns1.example.com example.com AXFR \
  | awk '$4 ~ /^(A|AAAA|CNAME)$/ {print $1, $4, $5}'

Then resolve each one and check whether the target still answers — and whether it’s still yours.

while read -r name type target; do
  echo "=== $name ($type) -> $target"
  dig +short "$name"
  curl -sS -o /dev/null -w "  http=%{http_code} ip=%{remote_ip}\n" \
    --max-time 8 "https://$name" || echo "  DEAD / no response"
done < zone-records.txt

For the dangling ones, the rule is simple: if you don’t control the target, the record dies or gets repointed.

# Delete the stale record (RFC 2136 dynamic update)
nsupdate -k /etc/dns/update.key <<'EOF'
server ns1.example.com
zone example.com
update delete app.example.com. CNAME
send
EOF

Repoint anything still in use at infrastructure you own — a host behind your own reverse proxy, not a third-party resource you might let lapse again.

Finally, make it a standing job. A nightly check that flags any record whose target stops resolving or stops belonging to you.

# cron: nightly dangling-DNS sweep, alert on dead targets
0 4 * * *  /opt/dns/dangle-check.sh && \
  curl -fsS https://hc-ping.test/dns-sweep

Why it happened

DNS outlives servers. Always.

When you spin a box down, you delete the box. You raise the ticket, you confirm it’s gone, you move on. The DNS record — living in a different system, owned by a different runbook — just sits there pointing at a ghost. Nobody’s job ends with “and remove the CNAME.”

Multiply that by a few years of “temporary” subdomains and deprovisioned cloud accounts, and you’ve got a quiet pile of doors with no locks.

Takeaways

• Inventory every record. You can’t audit a zone you’ve only half-remembered. Pull the whole thing.
• Verify the target, not just the name. A record that resolves isn’t safe — confirm the thing it points at is still alive and still yours.
• Prune the moment a server dies. Decommissioning a host isn’t done until its DNS is gone too. Bake it into the runbook.
• Prefer targets you control. Point subdomains at your own infrastructure, not third-party resources you might forget to renew.
• Monitor continuously. Dangling DNS is created by routine. Catch it with routine — a nightly sweep beats a quarterly surprise.

The Clicker That Ruled the Living Room

You remember the universal remote.

Not the one that came with the TV. The other one.
The fat black slab that lived on the arm of the couch.
Heavier than it had any right to be.
Three triple-A batteries deep.

It had way too many buttons.
Half of them did nothing.
The other half did things nobody could explain.

And only Dad knew the magic sequence.

Here’s what we thought it was, as kids:

• a wizard’s wand that talked to every box under the TV
• a forbidden object — touch it wrong and the screen goes blue and Dad sighs
• powered by a tiny, invisible laser you could never quite see
• the reason the TV “wasn’t working” was always your fault
• the thing that took 17 button presses just to watch a tape

You’d point it at the ceiling, the dog, your sister.
You’d press TV / VCR / CABLE / AUX hoping something would happen.

And when the channel finally changed?
Pure, unearned power. You were a god for one second.

Here’s what it actually was.

That “invisible laser” was real — sort of.

Every press fired a little infrared LED in the nose of the remote.
Light your eyes can’t see, blinking on and off insanely fast — flickering out a coded pattern, like Morse code made of glow.

The TV had a tiny IR receiver behind the front panel, watching for that exact blink-pattern. Volume up, channel down, power — each one a different rhythm of flashes.

The “universal” part was the clever trick.
Sony, Panasonic, Zenith — every brand blinked in its own dialect.
So the remote stored a codebook of them all.

That cursed setup ritual — hold SET, punch in 0-0-4-7 from the little fold-out card — that was you teaching it which dialect to speak.

It wasn’t magic.
It was a flashlight that knew how to talk.

        _________________
       |   ___________   |
       |  |  CH ▲     |  |
       |  |  VOL +    |  |   ((·)) ← invisible
       |  |___________|  |        IR blink
       |  (1) (2) (3)    |
       |  (4) (5) (6)    |
       |  (7) (8) (9)    |
       |  (TV)(VCR)(AUX) |
       |_________________|
        ‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾
         the sacred clicker

Here’s the thing.

It never left.

That blinking infrared light is still in your living room right now.
Your soundbar remote? IR.
The cable box clicker buried in the couch? IR.
Same flashlight. Same secret language. Same dead battery.

We just stopped looking at it.

Then your phone became the remote, and the remote became an app, and the app needed an update, and the update needed a login…

…and somewhere, a kid is pointing a glowing wand at a TV,
pressing 17 buttons to watch one show,
feeling like a god for one whole second.

Some magic just changes batteries.

A customer wanted their shiny camera platform recording into a third-party NVR. Simple ask: pull the RTSP stream, point the recorder at it, walk away.

The recorder disagreed. Every stream it tried died on connect. No video, no error worth reading, just a connection that opened and then went cold.

I’d wired up a hundred of these. This one fought me.

The investigation

First instinct: bad URL. So I copied the stream address straight out of the platform and fed it to ffprobe, the universal “is this thing even alive” test.

ffprobe "rtsps://camera-host:7441/<stream-token>?enableSrtp"

[tls] Error: handshake failure
rtsps://camera-host:7441/...: Input/output error

There it is. rtsps://. Not rtsp://. That little s is the whole story.

The platform was advertising an encrypted stream — RTSP wrapped in TLS, with SRTP turned on for the media itself. A high port, a token in the path, the works. Very modern. Very secure.

And completely useless to a recorder that only speaks plain, unencrypted RTSP. The NVR reached out for a normal RTSP handshake and got a TLS negotiation shoved in its face. It had no idea what to do, so it hung up.

┌──────────────┐     rtsps:// + SRTP      ┌──────────────┐
│   Camera     │ ────────────────────────►│   NVR        │
│  platform    │      (TLS handshake)     │ (plain RTSP) │
└──────────────┘                          └──────┬───────┘
                                                 │
                                                 ▼
                                          ✗ handshake failure
                                          ✗ no video recorded

Two devices, both fluent in “RTSP,” neither able to hold a conversation. The walled garden, working exactly as designed.

The “aha”

Here’s the thing about these platforms: the encrypted stream is what they show you. It’s rarely the only thing they offer.

I dug into the device settings and found a toggle — a separate, unencrypted RTSP endpoint on a different port. Plain rtsp://, basic credentials, no TLS, no SRTP. The boring old protocol the recorder actually wanted, sitting right there behind a checkbox nobody enabled by default.

The fix

Enable the plain endpoint, grab the new URL, and verify with ffprobe before touching the NVR config:

ffprobe -v error -show_streams \
  "rtsp://<user>:<pass>@camera-host:7447/<stream-token>"

codec_name=h264
width=2688
height=1512
✓ Stream #0:0  Video: h264, 2688x1512, 15 fps

Green across the board. Point the recorder at that URL and it records.

If the platform won’t expose a plain endpoint, drop a restreamer in the middle — something like MediaMTX or a thin ffmpeg relay — to ingest the encrypted stream and re-publish it as plain RTSP on your own box:

ffmpeg -rtsp_transport tcp \
  -i "rtsps://camera-host:7441/<stream-token>?enableSrtp" \
  -c copy -f rtsp \
  rtsp://127.0.0.1:8554/cam1

# verify the republished stream the NVR will actually consume
ffprobe "rtsp://127.0.0.1:8554/cam1"

The NVR talks to your restreamer. Your restreamer eats the TLS for breakfast. Everybody’s happy.

Why it happened

Vendors are encrypting streams by default now, and good for them — that’s the right call for traffic crossing untrusted networks. But “secure by default” and “interoperable” are different goals, and the camera folks optimized for the first one.

The encrypted URL is front and center. The plain one is buried, off, and undocumented. So the integration looks broken when it’s really just locked. The garden has a gate; you have to go find it.

Takeaways

• Read the scheme. rtsps:// on a high port with SRTP is a different animal than rtsp://. The handshake error is your first clue, not a riddle.
• Look for the plain-RTSP port. Most platforms still expose an unencrypted endpoint behind a toggle. Check device settings before you assume it can’t be done.
• ffprobe before you wire anything. If the probe can’t pull codec and resolution, the NVR won’t either.
• Put a restreamer in the middle when the vendor refuses to deobfuscate — MediaMTX or ffmpeg -c copy ingests RTSPS and republishes clean RTSP, no re-encode.
• A walled garden isn’t a wall. It’s a default. Find the gate the vendor left in the settings, and the lock-in mostly evaporates.

The Night We Carried Our Computers Into the Basement

You remember the LAN party.

Not a download.
Not a lobby.
A pilgrimage.

You unplugged your entire desktop — the beige tower, the CRT the size of a microwave, the rat’s nest of cables — and you carried it, in your arms, like a wounded soldier, to somebody’s basement.

The monitor weighed forty pounds.
You did this on purpose.
For fun.

And when you got there, there was already a folding table, an extension cord doing something it was definitely not rated for, and a bowl of Doritos that would last exactly nine minutes.

Here’s what it felt like as a kid:

• Forbidden — six computers in one room felt like a hacker movie
• Sacred — nobody touched the host’s machine. He was the king.
• Slightly dangerous — that many power bricks in one outlet? bold
• A little magic — your screen and his screen showing the same world
• Eternal — it was 2am and nobody’s mom had come downstairs yet

You’d hear someone yell “WHO HAS THE GRAY CABLE” and the whole night hinged on it.

The reveal

That gray cable was the whole trick.

It was a Cat-5 Ethernet cable, and it ran to a little blinking box in the middle of the table — a hub (later, the fancy kids had a switch). Every computer plugged into it, and that box let your machines gossip with each other at a blistering 10 megabits a second.

No internet involved. None. The whole party was a tiny private island of computers — a Local Area Network — talking only to each other.

Before someone bought the hub, you did it the cursed way: a coaxial cable daisy-chained machine to machine, with a little metal terminator screwed onto each end. If one person tripped over the line, the entire network died — and everyone knew whose fault it was.

And the game finding everyone? That was your machine literally shouting into the wire — “anybody out there hosting?” — and a host shouting back. That’s why someone always had to “make the server.” He wasn’t being dramatic. He was being the server.

   .------.   .------.   .------.
   |  PC  |   |  PC  |   |  PC  |
   '--||--'   '--||--'   '--||--'
      ||         ||         ||
   ===++=========++=========++===
      |     [ H U B ]   .|.|.     |
      '------ blink blink --------'
        every box hears every box

Where it went

It never really died — it just stopped weighing forty pounds.

That hub in the middle of the table? It moved into the little plastic router blinking in your hallway right now. That “shout into the wire to find a host”? That’s matchmaking, except now the wire is the whole planet and the host is a warehouse in Virginia.

You still play with your friends.
You just don’t carry your computer to do it.

But some part of you misses it — the cables, the chaos, the bowl of Doritos, the gray cable that held the whole night together.

We didn’t have multiplayer.

We built it, in a basement, every single time.

The Glass Box That Bit Back

You remember the monitor.

Not a screen. A monitor.

A beige cube so deep it needed its own zip code on the desk.
It hummed before it showed you anything.
That little rising whine — the sound of the thing waking up.

And then the moment.
You walked past it, reached out a finger, and —

ZAP.

A tiny lightning bolt jumped from the glass straight into your knuckle.

You weren’t even trying to touch it. The monitor reached out and got you first.

The kid logic was airtight:

• the screen was alive
• it could see you
• the zap was a warning
• if you put your face too close you’d get radiation (Mom said so)
• the dust on it was somehow magnetic dust
• pressing your palm flat on it made your hair stand up like a wizard

You’d run a comb through your hair, hold it near the glass, and watch the strands lean in.

Forbidden science. No lab coat required.

And the colors. Oh, the colors.

That deep, glowing, slightly-curved picture that pulled into the corners like a fishbowl.
When you turned it off, the image collapsed into a single bright dot in the middle —
pew — and then a ghost stayed there for a second, like the screen was holding its breath.

Here’s what was actually going on.

That fat box was a Cathode Ray Tube, and it was basically a tiny, tamed particle accelerator pointed at your face.

At the back, an electron gun fired a beam of electrons.
Magnetic coils — the deflection yoke — whipped that beam left-to-right, top-to-bottom, fifty to seventy times a second, painting one glowing line at a time across a phosphor coating on the inside of the glass.

The whine you heard? That was the flyback transformer, singing at about 15 kilohertz — right at the edge of what young ears could catch and old ears couldn’t. A genuine 90s-kid superpower.

And the zap?

The whole front of that tube was carrying a static charge from all those electrons slamming into phosphor. You weren’t being warned. You were just the nearest path to ground.

The dot when you powered off was the beam collapsing as the magnetics died.
The ghost image was phosphor persistence — the glow literally fading.

        ____________________________
       /                            /|
      /   .--------------------.    / |
     /   |   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓   |   /  |
    /    |   ▓  > C:\>_     ▓   |  /   |
   /     |   ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓   | /   /
  /      '--------------------'  |/  /
 /___________________________ __/  /
 |   (o)   POWER   ~~hummmm~~  |  /
 |____________________________|/
        ((( zap! )))  -> your knuckle

It never really left, you know.

That curved-glass glow lives on in every retro filter, every scanline overlay on a streaming game, every “CRT mode” toggle that fakes the fishbowl on a screen thin as a credit card.

We spent twenty years chasing flat, sharp, and silent.
Now we pay good money for software that puts the fuzz and the curve back in.

But the zap?

The zap is gone.

Nothing flat ever loved you enough to bite.

The Square That Could Resurrect the Whole Machine

You remember the boot disk.

Not a regular floppy.
A special one.

It lived in a little paper sleeve, or maybe a sandwich bag, somewhere near the computer.
Sometimes it had a label written in marker.
Sometimes the label just said “BOOT — DO NOT ERASE.”

And you believed it.

You did NOT erase it.

To us, this thing had powers.

The vibes:

• It was the disk you only touched in an emergency
• Dad slid it in when the computer “wouldn’t go”
• The screen would go black and serious and full of words
• It smelled faintly like plastic and authority
• Touching the shiny brown circle inside = forbidden, basically a crime
• It made the drive go grrr-CHUNK-grrr like it was thinking really hard

It felt less like a tool and more like a magic key.
The computer was a sleeping dragon, and this little square was the only thing that knew its true name.

Here’s what it actually was.

A computer can’t pull itself up by its own bootstraps from nothing — it needs the tiniest starter program to teach it how to load everything else. That’s literally where “booting” comes from.

That special floppy held a minimal copy of DOS: just enough operating system to wake the machine, talk to the keyboard, see the drives, and give you a C:\> prompt.

When the hard drive got corrupted, or a nasty boot-sector virus moved in, or Windows face-planted on startup — the hard drive couldn’t boot itself anymore.

So you fed the computer a clean brain from the A: drive instead.

From that humble prompt, the grown-ups ran the real magic spells:
scandisk. fdisk. format. sys c:.

640 KB of pure “I will fix you whether you like it or not.”

        _________________________
       |  _____________________  |
       | |  BOOT - DO NOT ERASE| |
       | |_____________________| |
       |  ___________________    |
       | |::|             |  |   |
       | |::|   A:\>_      |  |   |
       | |::|_____________ |  |   |
       |_______________________ _|
       |  __ |##############|    |
       | |  ||##############|    |
       | |__||##############|    |
       |_______________________ _|
              [ 3.5"  1.44 MB ]

The boot disk never really died — it just stopped being a square.

That clean little rescue brain became your Windows recovery USB.
It became F8, and Safe Mode, and “Startup Repair.”
It became the install media you make when a laptop won’t wake up.

Every time you’ve held a key during startup to summon a menu out of a dead machine, you were doing the exact same ritual.

Same dragon.
Same magic key.

It just fits in your pocket now — and nobody tells you not to touch the shiny part.

The backup was “running.” It had been running for nine hours.

A client’s nightly Kopia job pointed at a user’s documents folder. Normally it finished in twenty minutes. This morning it was still chewing: no error, no progress, just a cursor blinking next to a snapshot that had hashed maybe a few hundred files and then gone silent.

No CPU. No disk I/O to speak of. No network throughput. Just a process sitting there, perfectly calm, doing absolutely nothing while claiming to be busy.

That’s the worst kind of hang. A crash leaves a body. This left a process that would happily wait until the heat death of the universe.

The investigation

First instinct: blame the repository. Maybe the backend was wedged, maybe a lock was stuck. Nope. A fresh kopia snapshot create against a different folder ran clean in seconds.

So it wasn’t Kopia. It was this folder.

I watched the process with Resource Monitor and caught it: every time the scan touched a particular file, the thread parked in a wait state. Not reading. Not erroring. Waiting on the filesystem to hand back bytes that never came.

Then I actually looked at the folder.

┌────────────────────────────────────────────────┐
│  C:\Users\user\Documents  (OneDrive-synced)     │
├──────────────┬─────────────────────────────────┤
│  Q1-report   │  ✓  green check  (on this PC)    │
│  budget.xlsx │  ✓  green check  (on this PC)    │
│  archive\    │  ☁  cloud icon   (online-only)   │ ◄── placeholder
│  photos\     │  ☁  cloud icon   (online-only)   │ ◄── placeholder
└──────────────┴─────────────────────────────────┘
        the cloud-icon files have no bytes on disk

Half the tree had little cloud icons instead of green checks. OneDrive Files On-Demand. Those files looked real — name, size, timestamp — but the actual content lived in the cloud. On disk they were placeholders.

The “aha”

OneDrive’s on-demand placeholders are implemented with the Windows cloud files filter driver, cldflt. When any process opens a placeholder and tries to read it, cldflt intercepts the read and hydrates the file — downloads it from the cloud, transparently, before returning a single byte.

A backup tool is the most naive reader on earth. It opens every file and reads it end to end. So Kopia walks into a folder full of placeholders and triggers a hydration storm: thousands of files getting pulled down on demand, throttled, retried, and in some cases just… stalling. The read call blocks. The thread parks. Forever.

The backup wasn’t broken. It was politely waiting for the entire cloud to fall back onto the disk, one file at a time, through a sync client that had no idea it was under attack.

The fix

You don’t want to read the live folder. You want to read a snapshot of it — a frozen, point-in-time view that the cloud filter doesn’t get to intercept. That’s exactly what a Volume Shadow Copy gives you. The VSS snapshot presents the volume’s state as committed bytes, with no on-demand hydration path.

Kopia has hooks for precisely this: run an action before it walks the source root, and another after. Create the shadow before, tear it down after.

# before-snapshot: create a shadow copy and expose it via symlink
$vol = (vssadmin create shadow /for=C: | Select-String 'Shadow Copy Volume Name').ToString().Split(' ')[-1]
$env:KOPIA_SNAPSHOT_PATH = $vol
cmd /c "mklink /d C:\kopia-shadow $vol\"

# after-snapshot: clean up the link and drop the shadow
cmd /c "rmdir C:\kopia-shadow"
vssadmin delete shadows /for=C: /oldest /quiet

Then point the source at the shadow instead of the live path and wire the hooks in:

kopia snapshot create C:\kopia-shadow\Users\user\Documents \
  --before-snapshot-root-action="powershell -File C:\scripts\vss-before.ps1" \
  --after-snapshot-root-action="powershell -File C:\scripts\vss-after.ps1"

The next run finished in eighteen minutes. Every placeholder read off the shadow returned instantly, because the snapshot view doesn’t route through cldflt. No hydration, no stalls, no nine-hour ghost.

Why it happened

OneDrive Files On-Demand exists so users don’t sync 400GB of vacation photos onto a 256GB laptop. Great feature for humans. Terrible surprise for any tool that assumes “the file is on this disk.”

Kopia did nothing wrong. It read files. The trap is that on a cloud-synced volume, “open and read” is a network operation in disguise, gated by a filter driver that will block your thread while it phones home. Backups that walk thousands of placeholders turn into a self-inflicted denial of service.

Snapshotting through VSS sidesteps the whole mess, and as a bonus you get a consistent point-in-time copy instead of backing up a folder that’s mutating under you.

Takeaways

• Cloud on-demand files are placeholders, not files. OneDrive, Dropbox Smart Sync, and friends all use a filter driver (cldflt on Windows) that hydrates on read.
• A naive reader hangs on hydration. Backups, indexers, and AV scanners that open every file will trigger downloads — and stall when the cloud throttles or times out.
• Back up from a VSS shadow, not the live tree. The snapshot view bypasses the cloud filter and returns committed bytes instantly.
• Use Kopia’s before/after root-action hooks to create and tear down the shadow around each run — no manual babysitting.
• A backup at “0% for hours” with no I/O isn’t slow, it’s blocked. Watch the thread state; a parked read on a cloud volume is the tell.

The Toasters That Guarded the Machine

You remember the flying toasters.

You weren’t even using the computer.
Nobody was.
It just sat there in the den, humming, and then — winged toasters.
Flapping. Drifting. With toast.

Cruising across a black void like tiny chrome angels carrying breakfast to heaven.

And you would just… watch it.
For way too long.

Here’s what we knew about it as kids:

• It only showed up when everybody left
• It meant the computer was asleep but still alive
• Touching the mouse made it vanish, which felt like waking something
• The toast was non-negotiable. The toast was sacred.
• Your friend’s house had a flying-pipes maze instead, and that felt fancier somehow

It wasn’t a program you opened.
It was a thing the machine became when nobody was looking.

That was the magic part.
The computer had a secret life and the toasters were proof.

Here’s what was actually going on.

Back then, your monitor was a CRT — a big heavy glass tube that fired electron beams at a phosphor coating to make the picture glow.

And those phosphors had a problem: if the exact same image sat on the screen too long, it could literally burn in. A ghost of that image would stain the glass forever. A spreadsheet grid. A toolbar. A menu bar, etched into the dark like a fossil.

So the screen saver was doing exactly what the name says — saving the screen.

Move the pixels around constantly, never let one spot sit still, and nothing gets burned. The toasters weren’t decoration.
They were a tiny patrol.
Keeping the picture moving so the glass stayed clean.

The famous flapping breakfast came from a software company called Berkeley Systems, in a pack named After Dark. They even got sued once over those toasters, which is the most 90s sentence ever written.

        .-----------.
       /  o   o   o  \   ~ ~ ~
      |  [=========]  |   o  o
      |  | T O A S |  |  flap
       \ |  T  ~  T | /   flap
        `-----------'
           \\   //
          --\\-//--   pop! 🍞
            `-'-'
   the patrol is now airborne

The funny part? Screen burn-in basically went away.

Flat LCDs didn’t have those needy phosphors, so the danger faded.
The toasters had won. They could rest.

Except they never actually left.

Your phone dims and floats a clock around the lock screen.
Your TV nudges its glowing logo a few pixels every minute — because OLEDs can burn in, so we quietly reinvented the whole idea.
Your work laptop snaps to a slideshow of blurry photos the second you stand up.

We don’t call it a screen saver anymore.
But every time a screen drifts gently to keep itself from getting hurt —

that’s a toaster.
Still flapping.
Still carrying the toast.

Someone pinged me with the kind of bug that makes your skin crawl: “I save a note, it saves fine, no error. But when I come back later it’s gone. And sometimes I open a file and it’s the wrong file entirely.”

No errors. No crash. Data just… wrong. The worst kind of ticket — because “wrong data, no error” means the system is lying to you with a straight face.

I assumed corruption. Maybe a bad sync, a stale cache, a half-written file on disk. I was wrong about all of it.

The investigation

First thing I did was stop trusting the app and start trusting the wire. The app talked to itself over a local REST API on a fixed port. So I asked the only question that matters when data is “wrong”:

Am I even talking to the thing I think I’m talking to?

The user ran two separate profiles of the same application — two distinct workspaces, two configs, two processes. Different data, different windows. But same binary, same defaults.

I checked what was actually listening.

        Profile A (started first)        Profile B (started second)
        ┌──────────────────────┐         ┌──────────────────────┐
        │  app instance A      │         │  app instance B      │
        │  workspace: WORK     │         │  workspace: PERSONAL │
        └──────────┬───────────┘         └──────────┬───────────┘
                   │ bind :8200 OK                  │ bind :8200 FAIL
                   ▼                                ▼
            ┌─────────────┐                  (silent: port taken)
            │  TCP :8200  │ ◄──── requests land HERE
            └─────────────┘        regardless of which UI you used
                   ▲
                   │
            edits in B's UI → API call → answered by A

Both instances were configured to expose their REST API on the same hard-coded port. Whichever process won the startup race grabbed the socket. The loser tried to bind, failed, and — because the failure was non-fatal — just shrugged and kept running its UI as if nothing happened.

So you’d type into Profile B’s window. The UI would fire an API write. That write went to port 8200, which Profile A owned. Profile A happily saved it into the wrong workspace. Reads came back from A too. Profile B’s UI was a puppet with cut strings.

The “aha”

The tell was netstat. One PID owned the port. Not two. Not a conflict logged anywhere — just one quiet winner.

# Windows: who actually holds the port?
netstat -ano | findstr :8200
#   TCP    127.0.0.1:8200    0.0.0.0:0    LISTENING    18044

tasklist /fi "PID eq 18044"
#   app.exe    18044   ...   <- this is Profile A, not B

# Linux/macOS equivalent
lsof -nP -iTCP:8200 -sTCP:LISTEN
ss -ltnp 'sport = :8200'

One listener. Two apps. The data wasn’t corrupt — it was just going to the wrong house, and the mailman never complained.

The fix

Stop them from racing for the same socket. Give each instance its own port.

# 1. Edit each profile's config — distinct ports per instance
#    Profile A
"api": { "enabled": true, "port": 8200 }
#    Profile B
"api": { "enabled": true, "port": 8201 }

# 2. Fully stop both instances (no orphan holding the old socket)
taskkill /f /im app.exe        # Windows
# pkill -f app                 # Linux/macOS

# 3. Confirm the ports are actually free before restart
netstat -ano | findstr "8200 8201"   # expect: nothing

# 4. Start both, then verify ownership — two PIDs, two ports
netstat -ano | findstr "8200 8201"
#   TCP  127.0.0.1:8200  LISTENING  20110   <- Profile A
#   TCP  127.0.0.1:8201  LISTENING  20338   <- Profile B

Two listeners, two PIDs, two ports. Writes to B stayed in B. Reads told the truth. The “disappearing edits” were gone because they’d never disappeared — they’d just been filed in the wrong cabinet.

Why it happened

Defaults. The app shipped with a single hard-coded API port and assumed exactly one instance would ever run. Reasonable for the common case, fatal for power users running multiple profiles. There was no port-conflict warning because a failed bind on a convenience API isn’t fatal — the app degrades silently instead of refusing to start. Silent degradation is how a config bug masquerades as data loss.

Takeaways

• When data looks “wrong” but nothing errors, question the endpoint first. Confirm you’re talking to the instance you think you are before you go disk-spelunking for corruption.
• netstat -ano / lsof -i is ground truth for “who owns this port.” One listener where you expected two is the whole bug, right there.
• Multiple instances of the same app will knife-fight over hard-coded ports. Give every instance a unique port in its config — don’t trust defaults to coexist.
• A non-fatal bind failure is a trap. Silent degradation turns a 5-minute config fix into a week of phantom data-loss tickets.
• Pre-flight before restart: kill all instances, verify the port is actually free, then start — and read back the listeners to prove it stuck.

The Beige Bar That Held Up The Whole House

You remember the surge protector.

The long beige slab that lived on the floor behind the desk.
The one with the orange glow at one end.
The one nobody was allowed to step on.

It hummed in a way you could feel more than hear.

And it had the switch. That big rocker with the tiny red light, the one that turned the entire family computer empire on and off in a single satisfying clunk.

You didn’t know what it did.

You just knew it was important.

The vibes were:

• Forbidden. Dad said never unplug it. So you never unplugged it.
• A little dangerous. It said WARNING on it. In yellow. With a lightning bolt.
• Overloaded. Six outlets, somehow eleven things plugged in, a tangle like a nest of beige snakes.
• The boss of everything. Modem, monitor, the printer that screamed, the tower, the speakers with the subwoofer that lived under the desk. All of it answered to the bar.
• Magic. Press the switch and the whole room woke up. CRTs ticked, fans spun, the modem blinked. You were a wizard.

It was, honestly, the closest thing your bedroom had to a circuit breaker, and you were eight, and you respected it.

Here’s what it actually was.

Inside that beige slab was a tiny, heroic component called a MOV — a metal oxide varistor.

Most of the time it does nothing. It just sits there. But the moment the voltage from the wall spikes — a lightning strike down the block, the AC compressor kicking on, the fridge cycling — the MOV’s resistance suddenly drops and it shunts that extra energy away from your computer before it can fry the motherboard.

It’s a sacrificial guard. It literally throws itself on the grenade.

That’s also why old surge protectors quietly stop protecting. Each spike wears the MOV down a little. The strip still passes power, the light still glows — but the protection is long dead. It became a fancy extension cord and nobody told you.

The glowing light? Often just “there is power here.” Not “you are safe.”

The fancier ones had a second light — “PROTECTED” / “GROUNDED” — and that one going dark was the part that actually mattered.

   ___________________________________________
  |  o   [|] [|] [|] [|] [|] [|]            (~) |   <- the glowing switch
  |  o    |   |   |   |   |   |          POWER  |
  |_______|___|___|___|___|___|________________|
          |   |   |   |   |   |
        modem mon tower spkr prnt  ...and one mystery plug
              \__ J O I N E D   F O R E V E R __/

The funny thing is, it never left.

It’s still down there behind your desk right now. Different shape, same job. Maybe it grew a coax port for the cable line, or a couple USB-A holes, or a little app that emails you when a fuse pops.

We just stopped calling it the boss.

But every time your whole battlestation winks on with one press — the monitors, the dock, the mechanical keyboard with its own light show — that’s the same beige magic.

Still sitting on the floor.
Still quietly ready to take the hit for you.

You just don’t say thank you anymore.

The bot worked. Then it didn’t. Then it worked again.

It was a small Node.js Telegram bot running on a client’s server. Its whole job was to fire off outbound API calls — sendMessage, getUpdates, the usual. Most of the time it was instant. But every so often a call would just… stop. No exception. No 500. No DNS error. The promise sat there until the request timeout finally put it out of its misery seconds later.

Intermittent. Silent. Multi-second. The worst kind of bug there is.

The investigation

First instinct was the obvious one: the upstream API is flaky. So I logged timing around every call. The hangs weren’t correlated with anything — not load, not time of day, not a specific endpoint. Same call, same payload, same host. Sometimes 80ms, sometimes 5,000ms-then-timeout.

I pulled strace on the process during a hang and watched the syscalls. The bot wasn’t busy. It was sitting in a connect() that never came back. It wasn’t waiting on the API — it was waiting on a socket that was never going to open.

That reframed everything. This wasn’t a slow server. This was a connection attempt going into the void.

   bot ──► undici (happy-eyeballs)
              │
              ├─► resolve host
              │     ├─ AAAA (IPv6)  ─► ::  ►  connect() ──► ✗ black hole (no route)
              │     └─ A    (IPv4)  ─► .   ►  connect() ──► ✓ would work fine
              │
              ▼
        races both... but stalls on the v6 attempt

The host had IPv6 addresses configured on its primary NIC. It just had no working IPv6 route to the outside world. Addresses present, connectivity dead. The packets went out and nothing ever came back.

The “aha”

The bot used undici as its HTTP client. Modern undici does Happy Eyeballs (RFC 8305) — it resolves both A and AAAA records and races IPv4 and IPv6 connection attempts, preferring the v6 path.

On a healthy host that’s great. On this host it was poison. undici would get an AAAA record, try the IPv6 path first, and that connect() would sail into a black hole. The race timer is supposed to fall back to IPv4 — but with a route that silently drops packets instead of cleanly rejecting them, the fallback timing slipped and the whole call stalled until the app-level timeout.

A quick curl proved it cold:

# hangs forever — there's an AAAA record and broken v6 routing
curl -v https://api.example.com/

# instant — forced down IPv4
curl -4 -v https://api.example.com/

-6 hung. -4 flew. That’s your smoking gun.

The fix

Two layers. First, kill IPv6 on the one NIC that had it broken — while leaving it alone on the VPN and VM interfaces that genuinely use it.

# disable IPv6 on the offending interface only (e.g. eth0), keep it elsewhere
sudo sysctl -w net.ipv6.conf.eth0.disable_ipv6=1

# make it stick across reboots
echo 'net.ipv6.conf.eth0.disable_ipv6 = 1' | sudo tee /etc/sysctl.d/99-disable-ipv6-eth0.conf
sudo sysctl --system

Second, belt and suspenders at the app layer — force the resolver IPv4-first so even a half-broken host can’t bite us again:

// top of the entrypoint, before any outbound calls
const dns = require('node:dns')
dns.setDefaultResultOrder('ipv4first')

curl -4 already proved the IPv4 path was healthy, so this was a safe, deterministic pin. Restarted the bot, hammered it with calls for ten minutes. Zero hangs. Every request back under 200ms.

Why it happened

“Broken but present” IPv6 is worse than no IPv6 at all. If the host had no AAAA-resolvable path, the client never tries v6 and you never notice. But give it IPv6 addresses with no real route, and every modern HTTP client cheerfully tries the dead path first — then waits on a connect() that silently drops instead of refusing.

No RST means no fast failure. No fast failure means a stall. And because Happy Eyeballs only sometimes loses the race badly, the stall is intermittent. That’s why it looked like a flaky upstream when it was a flaky local NIC the whole time.

Takeaways

• If something hangs only sometimes on outbound HTTP, suspect IPv6. Test it in one command: curl -4 vs curl -6. If -6 hangs, you found it.
• Addresses ≠ connectivity. A NIC can hold a perfectly valid IPv6 address and still have zero working route to the internet.
• Disable IPv6 per-interface, not globally. Kill it on the broken NIC; leave it on VPN/VM links that actually need it.
• Pin IPv4-first at the app layer too (dns.setDefaultResultOrder('ipv4first')) — defense in depth for hosts you don’t fully control.
• A connect() that never returns is a routing problem, not a server problem. strace the hang before you blame the API.

There’s a box at a remote site. It lives behind NAT, behind a router I don’t control, on a connection that hands out a new public IP whenever the ISP feels like it.

To reach it, I had a ritual. SSH into a bastion. From the bastion, SSH into the target. Type two passphrases, pray nothing changed, and hope the WAN IP I memorized last month was still the WAN IP.

ssh -J bastion target

That one flag hid a small empire of fragility.

The investigation

It broke on a Tuesday, the way these things do. ssh: connect to host ... port 22: Connection timed out.

First instinct: the target is down. It wasn’t — a colleague on-site confirmed it was humming along, recording, serving, fine.

Second instinct: the bastion is down. It wasn’t either. I could land on the bastion clean.

ssh bastion 'echo alive'
alive

So the bastion was up, the target was up, and the hop between them was dead. I jumped onto the bastion and tried the second leg by hand.

ssh me@198.51.100.x
ssh: connect to host 198.51.100.x port 22: No route to host

There it was. The remote site’s WAN IP had rotated overnight. The 198.51.100.x baked into my SSH config — and into the bastion’s known_hosts, and into a port-forward rule, and into my muscle memory — now pointed at some stranger’s DHCP lease.

The “aha”

I’d built a chain of three brittle links to solve one problem: the target has no stable address I can reach from outside.

Every fix I’d ever applied — dynamic DNS, a forwarded port, a second hop — was just another way of chasing an address that refused to hold still. The bastion wasn’t a solution. It was a workaround for a missing fact.

What if the box just had a stable address? One that didn’t care about NAT, ISP whims, or which coffee shop my laptop was sitting in?

   BEFORE                                AFTER
 ┌─────────┐                          ┌─────────┐
 │ laptop  │                          │ laptop  │
 └────┬────┘                          └────┬────┘
      │ ssh                                │ ssh
      ▼                                    │ (encrypted mesh)
 ┌─────────┐                               │
 │ bastion │  ◄── public IP that          │
 └────┬────┘      keeps rotating          │
      │ ssh                                ▼
      ▼                                ┌─────────┐
 ┌─────────┐   NAT / CGNAT             │ target  │
 │ target  │   no inbound port         │ 100.x.. │ stable tailnet IP
 └─────────┘                           └─────────┘

That address is exactly what a WireGuard-based mesh VPN hands you. Put the laptop and the target on the same tailnet, and each device gets a private IP that follows it everywhere — through NAT, through CGNAT, through IP rotations — because the mesh handles NAT traversal for you.

The fix

Install the mesh on both ends. (Generic Tailscale-style flow below.)

# On the target — behind NAT, no inbound ports needed
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --ssh --hostname remote-target

# On my laptop
tailscale up

# Confirm the target shows a stable tailnet IP
tailscale status
# 100.x.y.z   remote-target   linux   active; direct

Now SSH goes straight there. No bastion, no -J, no rotating address.

ssh me@100.x.y.z
# or, by MagicDNS name
ssh me@remote-target

Lock it down with an ACL so only my devices can reach port 22 on that box:

{
  "acls": [
    { "action": "accept", "src": ["tag:admin"], "dst": ["tag:remote:22"] }
  ]
}

Then I deleted the bastion’s forward rule, pulled the stale host key, and retired the jump chain entirely.

ssh-keygen -R 198.51.100.x

Why it happened

The bastion was never the point. It existed because the target had no reachable, durable address — so I borrowed one from a machine that did, then chained a second SSH session on top.

Every link in that chain depended on a public IP staying put. One of them rotated, and the whole thing collapsed. That’s not a bug in SSH. That’s the architecture telling you it was held together with assumptions.

A mesh VPN moves the addressing problem off the public internet entirely. The tailnet IP is yours, it’s stable, and it doesn’t care what the ISP does at 3 a.m.

Takeaways

• If you’re chaining ssh -J to reach a NAT’d box, that’s a missing stable address, not a routing puzzle. Fix the address, delete the chain.
• Mesh VPNs (WireGuard-based) give every device a durable private IP that survives NAT, CGNAT, and ISP rotations — no inbound ports to forward.
• Stable addressing kills a whole class of failures: stale known_hosts, dead port-forwards, dynamic-DNS lag, memorized WAN IPs.
• Gate it with ACLs. A flat tailnet where everything reaches everything is just a bigger blast radius. Tag, scope, restrict to port 22.
• The most reliable hop is the one you deleted. Fewer moving parts, fewer 3 a.m. timeouts.

The Little Black Box That Owned The Grown-Ups

You remember the pager.

Clipped to a belt.
Hooked to a waistband.
Riding shotgun in a fanny pack like it paid rent.

It was small. It was plastic. It was, somehow, the most important object in the house.

When it buzzed, the room changed.

Dad would tilt it toward the light, squint at a row of numbers, and go “hm.”

Just “hm.”

And then he’d disappear to the kitchen phone.

You didn’t know what it did. Not really.

Here’s what you knew:

• It only spoke in numbers, and the numbers were a secret
• You were not allowed to touch it
• It made a noise that meant the day was about to change
• Important people had one. Doctors. Drug dealers on TV. Your uncle, for some reason.
• Sometimes it said 07734 and that was the funniest thing humans had ever done

It felt like a tiny walkie-talkie that only listened.
A beeper that knew things you didn’t.
A pocket oracle that whispered to adults in a code you’d crack someday.

You’d press the little button just to watch the screen light up green.
Then put it down fast, like it was hot.

Here’s what it actually was.

A pager was a one-way radio receiver. That’s it. No screen full of apps. No internet. Just a dumb, brilliant little antenna listening for its own number on a wide radio frequency, broadcast from tall towers across the whole region.

Somebody dialed the pager’s phone number, typed digits, and a paging network flung that message out over the air to every tower at once. Your dad’s beeper heard its unique address in the stream, grabbed the digits meant for it, and buzzed.

The “code”? It was just a callback phone number. Sometimes with a 911 tacked on for “call me NOW.”

And it worked when nothing else did. Pagers ran on a frequency that soaks into buildings where cell signal still face-plants to this day. One AA battery. Weeks of standby. Basically unkillable.

It wasn’t magic.

It was just really, really good engineering wearing a tiny green screen.

        ________________________
       |  .------------------.   |
       | |                    |  |
       | |   1-555-04 07734   |  |
       | |       *911*        |  |
       | |____________________|  |
       |     [MENU]  [READ]      |
       |   .__.   o o o   .__.   |
       |__/====\___________/===\_|
          BEEP · BEEP · BEEP

The pager never really died. It just took off the belt and moved into your pocket.

That buzz against your leg? Still there. Every text, every DM, every push notification is a pager alert with a marketing budget. Hospitals still run pagers, because when the network’s on fire, the dumb little radio is the one thing that still gets through.

So next time your phone buzzes and your whole face changes —

just know your dad did that first.

With a beeper.

And a callback number that probably said 07734.

The call came in the way these always do: “the whole network is slow.”

Not down. Slow. Which is worse, because slow means it’s still trying.

I pulled up the switch stats from the site and the picture was ugly. Broadcast counters spinning like a gas pump. Port LEDs across the access switches strobing in unison — that synchronized, hypnotic blink that should make the hair on your neck stand up. CPU on the core switch pinned. Every wired device crawling, half the VLANs timing out.

That’s not congestion. That’s a storm.

The investigation

A broadcast storm has a signature. One frame gets flooded out every port, comes back in another port, gets flooded again, forever, at line rate. The switches aren’t broken — they’re doing exactly what they’re told, infinitely.

So the question is never “what’s slow.” It’s “where’s the loop.”

I started where loops live: the edge. I pulled the MAC address table and watched a handful of addresses flapping between ports — the same MAC showing up on port 14, then port 9, then port 14, dozens of times a second. A stable MAC doesn’t teleport. A looped one does.

show mac address-table | the same MAC, two ports, flapping
─────────────────────────────────────────────────────────
  02:00:00:aa:bb:cc   Gi0/14   ◄─┐
  02:00:00:aa:bb:cc   Gi0/9    ◄─┤  same address,
  02:00:00:aa:bb:cc   Gi0/14   ◄─┤  two ports,
  02:00:00:aa:bb:cc   Gi0/9    ◄─┘  ~40x/sec  ✗

I traced both ports. Port 14 and port 9 went to two units of a consumer multi-room speaker system — the kind that builds its own little wireless mesh between speakers so they stay in sync.

And somebody had wired both of them into the LAN.

The “aha”

There it was. Each speaker had an Ethernet drop. Each speaker was also bridging to its sibling over the proprietary wireless mesh. So a frame could leave the switch on port 14, ride into speaker A, hop the wireless mesh to speaker B, and come right back into the switch on port 9.

Copper out, radio across, copper back in. A loop with one leg made of Wi-Fi.

The reason it had run fine for months and then exploded? Doesn’t matter much. Someone re-cabled a closet, both speakers ended up plugged in, the mesh did its job, and nothing on those access ports was watching for a loop.

The fix

The speaker was the trigger. The real bug was a switch fabric with no loop protection. So we fixed both.

First, the immediate bleeding — pull one of the two speaker uplinks. One leg of copper gone, loop broken, network breathes.

Then, the actual fix. Turn on spanning tree, and pick the root on purpose instead of letting MAC addresses elect a random one:

# Core switch — make it the root, deliberately. Lowest priority wins.
spanning-tree mode rstp
spanning-tree vlan 1-4094 priority 4096

# Distribution — second in line
spanning-tree vlan 1-4094 priority 8192

Then lock down the edge so an end-device port can never become a transit path:

# Access port to the speaker / any end device
interface GigabitEthernet0/14
 spanning-tree portfast
 spanning-tree bpduguard enable     # any BPDU here = err-disable the port
 spanning-tree guard root            # this port may never be a path to root
 storm-control broadcast level 1.00  # cap broadcast at 1% of line rate
 storm-control action shutdown

Mesh-style managed gear is the same idea in different clothes — flip on STP, set tiered priorities so the core is root, and enable Loop Protection plus BPDU Guard on access ports.

edge port discipline
─────────────────────────────────
  PortFast      ✓  fast end-device link
  BPDU Guard    ✓  sees a switch → kills port
  Loop Guard    ✓  detects the loop
  Storm-control ✓  broadcast ceiling
─────────────────────────────────
  result: ▓▓▓▓▓ blast radius = one port

We re-cabled the speakers to a single drop each, verified one stayed on copper and the rest synced over mesh, and watched the broadcast counter flatline.

Why it happened

Consumer mesh gear is a loop machine wearing a friendly face. It will happily bridge two of its own units, and it has no idea it’s also sitting on your switched LAN. Plug two of them into copper and you’ve handed your network a second path it never agreed to.

The switch didn’t fail. It had no spanning tree configured, so it had no mechanism to notice the loop and break it. The default config trusted every port to behave. Edge ports never behave.

Takeaways

• Always run STP/RSTP. A modern loop-free topology still needs spanning tree on standby for the day someone hands you a loop.
• Set the root bridge on purpose. Lowest priority value wins; don’t let a random MAC address decide where your traffic converges.
• Guard the edge. BPDU Guard, Loop Protection, and storm-control on access ports turn a network-wide meltdown into one dead port.
• Treat consumer mesh gear as hostile to your switch fabric. Never wire two mesh-bridging devices into the same LAN.
• A flapping MAC is a loop until proven otherwise. The mac-address-table is your fastest path from “it’s slow” to “it’s port 9 and 14.”

The pager went off at 6:14 a.m. The morning sync hadn’t run.

I knew that script. I wrote that script. It was good. It pulled data, massaged it, pushed it where it needed to go. It worked flawlessly every single time I ran it.

That was the problem. It only worked when I ran it.

The scene

Here’s the embarrassing truth about that automation: it lived inside an SSH session. I’d connect to the box at sync-host.internal.example, fire it off, watch it succeed, and disconnect feeling like a wizard.

The moment my terminal closed, the kernel reaped it. SIGHUP, lights out. No process, no cron-shaped safety net, nothing.

On the Windows box it was worse. I’d “scheduled” it — except every run popped a console window that stole focus from whoever was using the machine. People started killing the window because it interrupted them. Can’t blame them.

So three different failure modes, one script:

   ┌──────────────────────────────────────────────┐
   │  "It works when I run it manually"            │
   ├──────────────────────────────────────────────┤
   │  close SSH session   ──►  SIGHUP, process dies│
   │  server reboots      ──►  nothing comes back  │
   │  scheduled on win    ──►  window steals focus │
   └──────────────────────────────────────────────┘
                    │
                    ▼
            NOT actually deployed

The investigation

I tailed the journal. Nothing — because nothing was logging anywhere. The script wrote to stdout, and stdout was attached to a terminal that no longer existed.

I checked the process list. Empty. I checked for a cron entry. None — I’d never made one, because “I’ll just run it” had quietly become the deployment strategy.

That’s the aha, and it’s a dumb one: manual execution was masquerading as deployment. The script was perfect and completely undeployed at the same time. Every successful manual run was me hand-holding a service that had no manager.

A long-running, scheduled, headless job has three needs I’d ignored: survive a logout, survive a reboot, and write its logs somewhere a human can read them later. A terminal gives you none of those.

The fix

Give it a service manager. On Linux, a systemd unit. On Windows, NSSM. Both get auto-restart, log redirection, and start-on-boot — and both run headless.

Linux — /etc/systemd/system/morning-sync.service:

[Unit]
Description=Morning data sync
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=svc-sync
WorkingDirectory=/opt/morning-sync
ExecStart=/opt/morning-sync/.venv/bin/python /opt/morning-sync/sync.py
Restart=on-failure
RestartSec=10
StandardOutput=append:/var/log/morning-sync/out.log
StandardError=append:/var/log/morning-sync/err.log

[Install]
WantedBy=multi-user.target

sudo mkdir -p /var/log/morning-sync
sudo systemctl daemon-reload
sudo systemctl enable --now morning-sync.service
systemctl status morning-sync.service
journalctl -u morning-sync.service -f

Windows — same job, NSSM does the babysitting and runs it with no visible window:

nssm install MorningSync "C:\apps\morning-sync\.venv\Scripts\python.exe" "C:\apps\morning-sync\sync.py"
nssm set MorningSync AppDirectory "C:\apps\morning-sync"
nssm set MorningSync AppStdout "C:\logs\morning-sync\out.log"
nssm set MorningSync AppStderr "C:\logs\morning-sync\err.log"
nssm set MorningSync AppExit Default Restart
nssm set MorningSync Start SERVICE_AUTO_START
nssm start MorningSync

Now I close my laptop and it keeps running. The box reboots at 3 a.m. for patches and the sync comes back on its own. The logs sit in a file instead of evaporating with my SSH session.

Why it happened

Because the script worked. That’s it. When something works on the first manual run, the brain files it under “done” and quietly skips the boring part — the part where a supervisor owns the process, restarts it when it dies, and persists across boots.

A terminal is not a runtime. Cron-with-a-popup is not headless. “Done” is not “deployed.”

Takeaways

• “It works when I run it manually” is not deployed. A successful manual run proves the logic, not the operation.
• Long-running automation belongs in a service manager — systemd on Linux, NSSM on Windows — not a terminal you forgot to close.
• Always set auto-restart and boot persistence. Restart=on-failure plus enable --now (or SERVICE_AUTO_START) means reboots and crashes self-heal.
• Redirect stdout/stderr to a file. Logs that live inside an SSH session die with the session, and you’ll be debugging blind at 6 a.m.
• Run it headless. Background jobs that pop windows get killed by annoyed users. No UI, no focus theft, no problem.

All the World’s Knowledge, and a Loading Spinner

You remember Encarta.

That one disc.
Silver. Heavier than it should’ve been.
Lived in a jewel case next to the family computer like it was a relic.

You remember the moment it loaded — the swirl, the little fanfare, the screen going dark for a second like the machine was bracing itself.

And then: everything.
Animals. Volcanoes. Mozart. The solar system.
All of it, right there, glowing.

You were eight.
You did not understand what you were holding.
You just knew the vibes:

• It was the smart disc — the one Dad said cost real money
• It had a video of a frog, and you watched that frog forty times
• There was a globe you could spin and it felt like piloting a spaceship
• The narrator voice was so calm it was basically a wizard
• Typing your report? No. You copied it word for word and called it research

It felt like the entire library got crushed down into a coaster.
Forbidden, almost. Too much power for one kid and one beige tower.

Here’s what it actually was.

That disc was a CD-ROM — about 650 megabytes of read-only storage, pressed once at a factory and never changed again. (Your phone’s wallpaper is bigger than that now.)

To fit a whole encyclopedia on it, they pulled off some real magic:
the text was compressed down tiny, the photos were squeezed into early JPEGs, and the audio and video were chopped into low-res clips just big enough to feel amazing on a 14-inch CRT.

The frog video? Maybe two seconds of footage at a resolution you’d laugh at today. But it was moving pictures, on demand, from a disc. In 1995 that was witchcraft.

The whole thing ran off your CD drive at a blistering 1.2 megabytes a second — which is why the globe took a moment to think before it spun.

        _.--""""--._
      .'  _    _   '.
     /   (_)  (_)    \      ENCARTA '95
    |    .-.  .-.     |    .-------------.
    |   (   )(   )    |    | [o] CD-ROM  |
     \   '-'  '-'    /     '-------------'
      '._        _.'        all of human
         '------'           knowledge™
                            (insert disc 1)

The wild part?
It never really left.

You just stopped noticing.

The frog video became YouTube.
The spinning globe became Maps.
The calm narrator wizard became the little box you type questions into at 2 a.m.

The whole pressed-on-a-disc encyclopedia got vaporized into something that lives nowhere and everywhere at once, updates itself while you sleep, and never makes you swap to disc 2.

You don’t hold all the world’s knowledge in a jewel case anymore.

You hold it in your pocket.
And it still takes a second to load.

The Sacred Sticker That Ran Your Whole Weekend

You remember the tape.

That fat black brick with a label half peeled off.
The Blockbuster case that clicked shut like it meant business.
And the sticker — always the sticker —

BE KIND, REWIND.

You didn’t know who you were supposed to be kind to.
A stranger? The future? The tape itself?

You just knew it felt like a rule from on high.

The whole ritual was holy and a little terrifying.

The vibes:

• The late-fee dread that lived in your stomach all weekend
• Sliding the tape in and hearing that hungry mechanical gulp
• The screen going blue, then snowy, then… somebody else’s movie still playing
• That deep machine whirrrrr of rewinding, the counter ticking backward
• The guilt if you returned it un-rewound, like you’d left a mess at a friend’s house

You thought the movie lived inside the box.
Like a tiny theater you were renting for one night only.
You were not allowed to be late. You were not allowed to be unkind.

Here’s what was actually going on.

That tape was a long ribbon of magnetic film, wound between two reels inside the shell.

The picture and sound weren’t a file — they were stripes of magnetism painted onto that ribbon, read by a spinning drum of tiny heads as the tape dragged past.

And rewinding mattered for a real reason.
The tape played from one reel to the other, so when the movie ended, all the film was bunched on the wrong side.

The next person had to wait for it to spin all the way back before they could even press play.

So “be kind, rewind” wasn’t sentiment.
It was etiquette for shared, sequential hardware.
You weren’t streaming a movie. You were borrowing a physical line and you owed the next kid a fresh start.

        BE KIND, REWIND
   ___________________________
  |  [ o ]            [ o ]   |
  |   ===              ===    |
  |  /   \   VHS      /   \   |
  | (     )--========(     )  |
  |  \___/  |||||||||  \___/  |
  |_____________________  ____|
  |#####################|    |
  |#####################|____|
  |_________________________ |

The tapes are gone.
The little machine that rewound them for you — gone.

But the etiquette never died.
It just went invisible.

Now the platform rewinds for you. Silently. Instantly.
You scrub the timeline and the whole movie is already there, every second equally close, no wrong side to bunch up on.

No counter. No whir. No guilt.

And honestly? Something’s a little lost.

Because you’ll never again hand someone a thing and quietly hope —

you left it better than you found it.

The Yellow Brick That Played Your Whole Heart

You remember the Walkman.

That chunky little slab of plastic clipped to your waistband like a deputy’s badge.
Orange foam headphones that pinched your ears just enough to feel official.
And the clunk — that satisfying mechanical clunk when the cassette door snapped shut.

You hit play.
And for the length of one side, the world belonged to you.

It felt important. A little forbidden. The first thing that was yours and not the family’s.

The vibes were immaculate:

• Hitting record off the radio and screaming at the DJ for talking over the intro
• The terror of low batteries — when the song slowed down and dropped an octave, like the music was dying
• Rewinding to your favorite part and counting in your head because there was no rewind-to-track, just raw guesswork
• That ONE tape so loved it sounded like it was recorded underwater
• And the pencil. Always the pencil.

Because here’s the thing every 90s kid knew by instinct.
When the tape went slack and spat out a loop of shiny brown ribbon — you grabbed a pencil, jammed it in the little spoked hole, and wound it back in.

You didn’t know why it worked.
You just knew it was magic, and you were the wizard.

Here’s what was actually happening.

That brown ribbon was mylar tape coated in iron oxide — basically rust, lovingly arranged. The play head was a tiny electromagnet “reading” the magnetic pattern stamped into that rust as it dragged past at exactly 1⅞ inches per second.

Two reels. One feeding, one taking up. When the take-up reel stuck, the player kept pulling tape out — and you got The Spaghetti Incident.

The pencil? Its six-sided hole was a dead-perfect match for the cassette’s reel hub. You weren’t doing magic. You were doing manual tape transport — the exact job the motor did, just powered by a bored 9-year-old’s wrist.

And the dying-batteries voice drop? Real physics. Weak power meant the motor spun slower, so the tape crawled past the head and every pitch sagged. You were literally hearing your AA batteries run out of breath.

   ___________________________
  |  .---.             .---.  |
  | (  o  )           (  o  ) |
  |  '---'   [ A ]     '---'  |
  |    \_______________/      |
  |   |::|::|::|::|::|::|      |
  |___|_TYPE_I_·_60_min_|_____|
       \___ pencil goes ___/
            here →  O

You can’t kill a format that taught you patience.

The Walkman never really died — it just lost the moving parts. The iPod was a Walkman that finally stopped eating your tapes. Your phone is a Walkman with the whole record store crammed inside it.

But somewhere a teenager just paid forty bucks for a brand-new cassette because the algorithm told them it was vintage.

And somewhere, an old pencil is so proud.

A Clock That Never Learned What Time It Was

You remember the VCR.

That big black slab under the TV.
Heavier than it had any right to be.
A mouth that ate tapes and a single, stubborn green 12:00 blinking out into the dark living room.

12:00.
12:00.
12:00.

Forever.

Nobody set it. Nobody could. It just blinked, all night, like a tiny lighthouse warning ships away from the entertainment center.

And you, small and certain, knew exactly what it was about:

• vibes:
• it was the brain of the TV, obviously
• the blinking meant it was thinking
• pushing EJECT felt like defusing a bomb
• the rented tape had to be rewound or something bad happened (you weren’t told what)
• the little tracking lines were a storm you could fix by waving your hand at it
• that flap was a mouth and you were NOT putting your fingers near it

It was important. It was a little forbidden. It was a little magic.

Here’s the reveal.

That blinking 12:00 wasn’t broken. It was just honest. The VCR had a tiny digital clock inside, and it had no idea what time it was because nobody ever told it. Every power flicker — every summer storm, every blown fuse, every time Dad vacuumed and hit the wrong outlet — wiped its memory clean and it woke up at the only number it knew: midnight. Square one. Again.

Inside that heavy box was genuinely wild stuff. Magnetic tape, a half-inch wide, dragged past a spinning head drum tilted at a slight angle so it could cram diagonal stripes of video onto the tape — helical scan, they called it. That whir you heard when you hit play was the drum spinning up to speed, threading the tape around itself like a tiny robot doing origami in the dark.

And the rewinding? Real. Tape is a line, not a grid. To see the start, you physically wound it back to the start. There was no “skip to beginning.” There was only patience and that zzzzzzWP of the tape snapping home.

        ____________________________
       |  [] SONY      VHS HQ        |
       |                            |
       |   .------------------.     |
       |   |  __          __  |     |
       |   | (__)  ::::   (__)|     |  <- the tape
       |   |______________ ___|     |
       |   '------------------'     |
       |                            |
       |    [<<] [|>] [>>] [#]       |
       |        __                  |
       |       |12:00|  *blink*     |
       |       |_:_:_|              |
       |____________________________|

It never really went away.

That blinking clock just moved. Now it’s the microwave you never set after daylight saving. The oven that flashes after a brownout. The router light that means something, you’re pretty sure. The little devices in your life still wake up confused, still ask you what time it is, still get ignored.

And rewinding? You do it every time you drag the scrubber bar back thirty seconds because you missed a line of dialogue.

Be kind. Rewind.

You still do. You just don’t have to get up for it anymore.

The Box That Kept Our Voices

You remember the answering machine.

It lived on the little phone table in the hallway.
Beige. Always beige.
A red light that blinked when something was waiting for you.

One blink. Two blinks. Three blinks?!
Three blinks meant something happened today.

And you were NOT allowed to touch it.

You’d come home from school and that red light would be going,
and the whole house would change.

Mom hits PLAY.
A beep.
Then a voice — disembodied, a little too loud, a little too slow —
floating out of a box into the hallway like a tiny ghost.

The vibes:

• the outgoing message rehearsed nineteen times before anyone got it right
• the cousin who didn’t know how it worked and just breathed for forty seconds
• the telemarketer’s awkward pause, then click
• pressing your ear to it because you weren’t allowed to turn it up
• the cold terror of accidentally erasing all messages

It felt like a vault.
A little machine guarding voices that belonged to grown-ups.

Here’s the reveal.

Inside that beige box?

Tape.

Sometimes two tiny cassettes — one looping your greeting on a couple feet of magnetic film, one recording whatever came in. Later models cheated and used a microchip with a few minutes of grainy memory, but the classic ones were pure analog.

The “magic” was just physics and a phone line.

The machine counted rings. After four, it picked up — closed the circuit so the caller stopped hearing ring, played your greeting off the loop tape, then dropped a record head onto the second tape and let it roll. The beep wasn’t decoration. It was the cue: the tape is moving NOW, say your thing.

The blinking light? A counter ticking up each time the record head fired.

No cloud. No server.
Voices physically scratched into a ribbon of rust, sitting on a hallway table.

        _________________________
       |   ___________________   |
       |  |  ANSWERING MACHINE | |
       |  |   ___    ___   ___ | |
       |  |  (o o)  | 3 |  msg | |   <- blinking
       |  |___________________| |
       |   [<<] [>] [>>]  [REC]  |
       |    ___________________  |
       |   ||| | |||| | | || |   |   <- tape window
       |   |___________________| |
       |_________________________|

The funny part?

It never left.

We just stopped owning the beige box.
Now the tape lives on a server farm somewhere, the greeting is robot-voiced, and the blinking red light became a little number bubble on your phone screen you swipe away without listening.

Same loop. Same dread of a full mailbox.

Except now nobody calls — they text “call me?” — and the only voicemails left are the dentist and a scam about your car’s extended warranty.

That box held the voices of everyone you loved.

Four rings, then a beep.

You still flinch a little when something blinks red.

It was supposed to be a five-line automation. Read a value, tweak it, write it back. The kind of thing you knock out between coffees.

I shelled into a client’s pfSense box — a clean, current 24.11 install — and ran my little script through pfSsh.php. It exited without a peep. No errors. No warnings. Just a fresh prompt blinking back at me.

Then the SSH session died.

Then the VPN dropped.

Then my phone lit up.

The investigation

I got back in through the console. The dashboard loaded, and my stomach went cold. No WAN config. No LAN rules. No NAT, no VLANs, no DHCP scopes. The box was sitting there with the bright-eyed innocence of a router that had never met a network.

The config wasn’t corrupted. It was gone. Replaced with defaults, like someone had hit the reset button — except nobody had touched the hardware.

I pulled up my script. Five lines. The offending logic looked like ancient, trustworthy pfSense scripting lore:

global $config;
$config['system']['something'] = 'newvalue';
write_config("scripted tweak");

Every old forum thread, every crusty wiki page from a decade ago, does exactly this. Grab the $config global, poke a key, call write_config(). It’s the canonical move.

So why did it nuke the box?

I added one debug line before the write:

global $config;
var_dump($config);

The output told the whole story in one ugly word: empty.

The “aha”

On modern pfSense, that legacy $config global is no longer the living, fully-populated config tree the old guides assume. In the pfSsh.php scripting context it came back essentially empty — a hollow shell.

And write_config() doesn’t merge. It doesn’t patch. It takes whatever is in memory right now and serializes that as the new, authoritative config. The whole thing.

So my script read an empty global, set one key on top of nothing, and then confidently wrote that “nothing plus one key” back as the complete configuration. write_config() did exactly what it was told. It overwrote a fully working firewall with a near-empty husk.

┌──────────────────────────────────────────────────────────┐
│  WHAT I THOUGHT HAPPENED          WHAT ACTUALLY HAPPENED  │
├──────────────────────────────────────────────────────────┤
│  $config = { full tree }     ►    $config = { } (EMPTY)   │
│        │                                  │               │
│        ▼ set one key                      ▼ set one key   │
│  { full tree + key }              { almost nothing }      │
│        │                                  │               │
│        ▼ write_config()                   ▼ write_config()│
│   ✓ saved correctly               ✗ overwrote everything  │
└──────────────────────────────────────────────────────────┘

The modern config backend wants you to read and write specific paths — config_get_path() and config_set_path() — never the whole global at once.

The fix

First, recover. Restore the last known-good config from a backup and reload:

# Restore from a saved backup and reload
config_restore /cf/conf/backup/config-<latest-good>.xml
/etc/rc.reload_all

pfSense keeps automatic backups under /cf/conf/backup/ — list them with ls -lt /cf/conf/backup/ and pick the newest one from before your script ran. That brought the firewall back to life.

Then, fix the script. Use the path API and stop touching the global entirely:

// Read a specific key (with a default), modify, write that key back.
$value = config_get_path('system/something', 'default');
config_set_path('system/something', 'newvalue');
write_config("scripted tweak via path API");

And — non-negotiable — snapshot the config before any scripted write, so recovery is one command instead of a panic:

cp /cf/config.xml "/cf/conf/backup/config-$(date +%Y%m%d-%H%M%S)-prewrite.xml"

Why it happened

Platforms migrate their internals quietly. pfSense moved to a path-based config backend, and the old global $config got demoted from “the config” to “a vestigial variable that may or may not be populated, depending on context.”

The docs got updated. The thousand forum posts from 2014 did not.

So the trap is perfect: the legacy pattern still compiles, still runs, still exits clean. It just operates on a ghost of the real config — and write_config() is happy to make that ghost permanent.

A function that overwrites everything from in-memory state is a loaded gun. When the thing loading that memory changes underneath you, the gun is now pointed at your whole config and you don’t even know it.

Takeaways

• A global write_config() rewrites the entire config from in-memory state. If that state is stale, partial, or empty, you’ve just factory-reset your box. Treat it as a full-config replace, never a patch.
• When a platform migrates its config backend, the old global is a footgun. It still runs clean — that’s what makes it dangerous. Verify what it actually contains before trusting it.
• Use the scoped API. config_get_path() / config_set_path() touch exactly one key. That’s the blast radius you want.
• Snapshot before any scripted write. cp /cf/config.xml /cf/conf/backup/...-prewrite.xml turns a disaster into a 30-second restore.
• var_dump() your assumptions. One debug line showing an empty global would’ve saved the whole incident. When in doubt, print what you’ve actually got — not what the old guides promise you have.

The Save Icon You Could Hold In Your Hand

You remember the floppy disk.

Not the bendy black ones your older cousin had.
The hard ones. The beige squares with the little silver shutter on top.

You remember the click it made going into the slot.
That confident, mechanical chunk that told you something important was happening.

You’d slide it in.
The drive light would blink.
And the whole computer would make that grinding, churning noise like it was thinking really hard.

To a kid, a floppy was basically magic.

• The metal shutter. You were NOT supposed to slide it open. So obviously you slid it open. There was shiny brown film inside. You touched it. You doomed the disk.
• The write-protect tab. A tiny plastic switch that felt like arming a nuclear device.
• The label. You wrote “GAMES” on it in marker like a tiny librarian.
• The hoard. A whole rainbow box of them, and you had no idea what was on 80% of them.

It felt forbidden and fragile and yours.

You knew, somehow, that if you put a magnet near it, you’d kill it. Nobody told you the rules. You just knew. Floppy disk law was passed down kid to kid like folklore.

Here’s the reveal.

That beige square held 1.44 megabytes.

That’s it. That’s the whole thing.
Less than a single phone photo today. A blurry one.

Inside that shutter was an actual spinning disk of magnetic-coated mylar — that shiny brown film you weren’t supposed to touch. The drive head physically pressed against it and read magnetic charges, like a tiny record player for ones and zeros.

The shutter? That slid open automatically when you inserted it, so the head could reach the disk. You opening it by hand was just you, a small barbarian, exposing the delicate part to dust and fingerprints.

The write-protect tab actually worked by letting light through a hole. Open the hole, a sensor saw it, and the drive physically refused to write. It wasn’t software. It was a window. Genuinely un-hackable by a kid, which is the only security that ever truly held.

        ________________________
       |  __                    |
       | |  |  <- write-protect |
       | |__|     tab (a hole!) |
       |  ____________________  |
       | |####################| |  <- metal shutter
       | |####################| |     (do NOT slide it)
       |_|____________________|_|
       |                        |
       |   [ GAMES ]            |  <- your handwriting
       |   ( in marker )        |
       |________________________|

The funny part?

The floppy never really left.

Look at the Save button in basically any app you’ve used in the last thirty years. That little square with the shutter? That’s a floppy disk. A piece of 1980s hardware, frozen forever as an icon, saving documents to drives that floppy disks couldn’t dream of holding.

There are kids today hitting Save who have never seen the object the button is shaped like. To them it’s just “the save shape.” A glyph. A rune with no origin story.

But you know.

You know it went chunk.
You know you weren’t supposed to touch the brown part.
And you know that somewhere in a closet, there’s still a rainbow box of them, holding 1.44 megabytes of something you’ll never read again.

And honestly? That’s plenty.

The message came in the way these always do. No timestamp, no context, just adrenaline.

“The payment shows zero applied. The customer paid. Where did the money GO?”

Self-hosted invoicing app. A real payment, recorded against a real invoice. The UI said $0.00 applied. To everyone looking at that screen, that meant one thing: a customer handed over money and the system ate it.

That’s a five-alarm fire in billing. Money you can’t account for is money you have to refund, re-bill, or explain to an auditor. So I did the thing you do when the building is allegedly burning. I stopped trusting the screen.

The investigation

The UI is a story the app tells you. The database is what actually happened. When the two disagree, the database wins — every time.

So I went to the source of truth. Payments table first.

SELECT id, invoice_id, amount, applied, status, created_at
FROM payments
WHERE invoice_id = 4815;

 id  | invoice_id | amount  | applied | status    | created_at
-----+------------+---------+---------+-----------+---------------------
 162 |       4815 | 1200.00 |  NULL   | completed | 2026-03-07 16:42:11

There it is. amount = 1200.00. Status completed. The money was recorded. It did not vanish. It was sitting in the table exactly where it should be.

applied was NULL — interesting, but not “missing money” interesting. Stored zero would’ve been one thing. NULL is the smell of a column nobody populates.

Now the invoice. Did the balance actually move?

SELECT id, amount, balance, status
FROM invoices
WHERE id = 4815;

 id  | amount  | balance | status
-----+---------+---------+--------
4815 | 1200.00 |    0.00 | paid

Balance dropped to zero. Status flipped to paid. The ledger was correct. The accounting was correct. The money was where it belonged.

So why was the UI screaming?

        WHAT THE SCREEN SAID            WHAT THE DB SAID
       ┌──────────────────────┐       ┌──────────────────────┐
       │  Payment #162        │       │  payments.amount      │
       │  Applied:  $0.00  ⚠  │  vs   │    = 1200.00  ✓       │
       │  "money gone?!"      │       │  invoices.balance     │
       └──────────────────────┘       │    = 0.00 (paid) ✓    │
                  │                    └──────────────────────┘
                  │                              │
                  └──────────► SAME ◄────────────┘
                         UNDERLYING TRUTH

The “aha”

The “Applied” figure on that payment screen wasn’t reading payments.amount. It was reading a computed display field — a value the UI derived at render time from a relationship that, for this payment, came back empty.

The applied column existed in the schema but was never written by this payment path. It lived only in the database and was never even exposed through the API. The renderer reached for it, found NULL, coerced it to 0.00, and printed a number that looked like a catastrophe.

Nothing was lost. A blank field got rendered as a zero, and a zero in a money column reads like a heist.

The fix

There was no data to fix. The fix was confirming the ground truth and proving the money was accounted for, then knowing which field to never trust again.

-- Prove the payment is reconciled against the invoice
SELECT i.id,
       i.amount        AS invoice_total,
       i.balance,
       COALESCE(SUM(p.amount), 0) AS paid_total,
       i.amount - COALESCE(SUM(p.amount), 0) AS expected_balance
FROM invoices i
LEFT JOIN payments p ON p.invoice_id = i.id AND p.status = 'completed'
WHERE i.id = 4815
GROUP BY i.id;

 id  | invoice_total | balance | paid_total | expected_balance
-----+---------------+---------+------------+------------------
4815 |       1200.00 |    0.00 |    1200.00 |             0.00

balance matches expected_balance. Books are square. Crisis cancelled — because there was never a crisis, only a bad pixel.

Why it happened

The app had two notions of “applied”: the real stored amount on the payment, and a derived field the UI computed from a relation that this particular payment flow never filled in. When the relation was empty, the computed value fell through to zero instead of throwing or hiding itself.

A zero in a dollar column is the most alarming value a billing system can show. The renderer printed it with full confidence and zero context, and a correct ledger looked like a robbery.

Takeaways

• The UI is a rendering of the truth, not the truth. When a number looks wrong, query the database before you assume data loss.
• Most “missing money” is a display bug. Check the ledger and the invoice balance first — the books are usually fine.
• Know which fields are stored vs computed. A value derived at render time can lie even when every stored byte is correct.
• NULL coerced to 0 is a landmine in money columns. A blank field and an actual zero mean wildly different things; don’t let the UI conflate them.
• Schema fields can exist without ever being populated or exposed. Confirm the data path actually writes the column before you trust what reads it.

The Box On Top Of The Box

You remember the cable box.

Not the TV. The other thing.
The squat plastic brick that lived on top of the TV, where it had no business being.

It had a red LED clock that glowed like a tiny alarmed eye.
It got warm. Always warm.
And it had a wired remote — a remote on a leash — that snaked across the carpet like it was afraid to leave home.

You knew, in your kid bones, that this box was Important.

• It hummed faintly, like it was thinking
• The channel number flipped on it with a soft clunk you could almost feel
• Dad told you not to unplug it or “we’ll lose the channels”
• The remote cord was always one stomped-on foot away from being yanked out
• There was a little sliding door on the front, and behind it, forbidden buttons

You were pretty sure the box was negotiating with the TV on your behalf.
And you weren’t entirely wrong.

What it actually was

Here’s the nerdy reveal.

Your TV in the 90s could only really speak one fluent language: channels 2 through 13, the old VHF band.
But the cable company was pumping dozens of channels down that one coax wire, all stacked on different frequencies way up high where your TV couldn’t tune.

The cable box was a set-top converter. Its whole job:

Grab the channel you wanted from way up the frequency dial
and re-broadcast it down to channel 3 (or 4),
the one frequency your TV trusted.

That’s why you set the TV to 3 and left it there forever.
The TV was just a dumb window.
The box did the choosing.

The wired remote? That was infrared and pricey, so the cheap units just ran a cable straight to the box — no batteries, no line-of-sight, no losing it in the couch. A leash, yes. But an honest one.

        _________________________
       |   [ 0 3 ]        CABLE   |
       |   .----.                 |
       |   | 03 |   o o o o o     |
       |   '----'   POWER  ~~~    |
       |_________________________|
              |||  coax in
              |||
        ======O======>  TV ch.3
       (  "leashed"  )
        \  remote   /
         '~~~~~~~~~'

It never really left

The leash got cut. The box got smaller, then it got a guide, then a DVR, then a slot for movies you paid eight bucks to “rent” with a button.

But the idea never died.

Every streaming stick jammed into the back of your TV today is the exact same trick:
a little warm box that does the real work, while the TV just sits there being a dumb, beautiful window.

You still set the input and leave it.
You still don’t unplug it, or “we’ll lose everything.”

The box on top of the box won.
It just learned to hide.

The ticket was boring. Cleanup. A client wanted some stale files gone from their primary share. I deleted them. Coffee. Done.

Twenty minutes later the same nine files vanished from the backup node too.

That’s the part that made my stomach drop. The backup node existed precisely so that “oops, deleted it on the primary” was a non-event. It was supposed to be the safety net. Instead it watched me cut the trapeze and then helpfully cut its own rope to match.

The investigation

First instinct: ransomware, a rogue cron, a fat-fingered teammate. All wrong. The truth was dumber and worse.

I’d inherited this setup and never read the sync config closely. I assumed “backup node” meant one-way. It didn’t.

grep -i 'mode\|direction\|propagat' /etc/syncthing/config.xml

<folder id="primary-share" type="sendreceive" ...>

sendreceive. Bidirectional. The backup node wasn’t a backup at all — it was a mirror. And a mirror does exactly one job: make the other side look like this side. Including the absences.

I deleted nine files on the primary. The sync engine saw nine deletions, decided they were authoritative changes, and faithfully replicated them downstream. No malice. No bug. It did exactly what I configured it to do.

┌──────────────┐   delete 9 files    ┌──────────────┐
│   PRIMARY    │ ──────────────────► │  "BACKUP"    │
│  (source)    │   sendreceive       │  node        │
└──────────────┘   propagates ✗      └──────────────┘
       │                                    │
       ▼                                    ▼
   files gone                          files gone too
                                       ▓ safety net: GONE

The “aha” wasn’t a clever diagnostic. It was reading one attribute and feeling like an idiot. Sync is not backup. A sync engine has no concept of “retain this.” Its entire worldview is converge. Deletion is just another change to converge on.

The fix

Recover first, redesign second. The deleted files came back from a daily snapshot that — by luck, not design — lived off the sync path. Then I changed the node from a mirror into an actual backup.

# 1) flip the backup node to PULL-ONLY: it receives, never pushes
#    Syncthing: set this folder to "Receive Only"
#    config.xml -> type="receiveonly"

# 2) route deletions to a versioned trash instead of hard-deleting
#    <versioning type="trashcan"> with a retention window
#    (deletes land in .stversions/, not the void)

# 3) shorten the sync interval to bound the blast radius
#    rescanIntervalS: 3600 -> 300   # 5 min, not 1 hour

# 4) confirm the daily snapshot job runs OUTSIDE the synced tree
systemctl list-timers | grep snapshot
ls -la /var/snapshots/   # snapshots here, NOT under the synced folder

After the flip, the verification was the satisfying part. Delete a test file on the primary, watch the backup node not care:

# on primary
touch /srv/share/CANARY.txt
rm  /srv/share/CANARY.txt

# on backup node, a few minutes later
ls /srv/backup/share/CANARY.txt
# CANARY.txt is still there ✓  (delete did not propagate)

Receive-only means upstream deletions are treated as conflicts to ignore, not commands to obey. The node now keeps things the primary throws away. That’s the whole point of a backup.

Why it happened

Because “sync” and “backup” use the same verbs — copy, mirror, replicate — people treat them as the same noun. They aren’t.

Sync optimizes for agreement: every node looks identical, fast. Backup optimizes for survival: keep a copy even when the source loses it. Those goals conflict the instant you delete something. A mirror resolves the conflict by deleting too. A backup resolves it by holding on.

I’d labeled a mirror “backup” and trusted the label. The config said sendreceive the whole time. I just never looked.

Takeaways

• Sync is not backup. Bidirectional sync replicates deletions perfectly — that’s a feature, and it will eat your data.
• A backup node must be one-way. Receive-only / pull-only: it accepts changes but never pushes or propagates deletes upstream-to-down.
• Route deletions to trash, not the void. A versioned trashcan with retention turns “gone forever” into “gone for now.”
• Keep snapshots off the sync path. If your point-in-time recovery lives inside the synced tree, the sync can delete your recovery too.
• Read the config before you trust the label. grep the one attribute that defines behavior — sendreceive vs receiveonly is the whole story.

The Coil That Stretched From the Kitchen to the Edge of Forever

You remember the cord.

That curly, springy, impossibly tangled umbilical between the wall and the phone you were absolutely not supposed to be using right now.

You remember stretching it.

Kitchen, around the corner, down the hall — pulled taut, vibrating like a guitar string, the handset pressed to your ear while you sat on the floor of the only room with a door that closed.

You remember the stretch limit.
The exact spot where one more inch would yank the whole phone off the wall.

It felt like more than a wire.

The vibes:

• A leash. Mom could find you by following it like a treasure map.
• A weapon. Whip it and it cracked like a tiny lightning bolt.
• A mystery. Why was it ALWAYS knotted? You hung up straight. It tangled out of spite.
• Forbidden tech. Long-distance calls cost real money, and you could feel the adults watching the clock.
• A portal. Somehow your friend’s voice was in your hand and they were across town.

You’d stretch it as far as it would go just to whisper. That extra ten feet of cord was the closest thing your house had to privacy.

Here’s what it actually was.

That coiled cord carried two tiny copper wires — a circuit — and your voice rode it as a wobbling electrical current, a copy of the sound pressure hitting the mouthpiece.

The curl wasn’t decoration. It was strain relief: coil it and you get slack without a long trip-wire dangling everywhere. The price of that clever spring? It stored twist, and every time you flipped the handset it wound up a little more. The tangle was physics, not spite.

And the wall jack? Behind it, a pair of wires ran out of your house, down the street, to a central office full of switches — once human operators with patch cords, later electromechanical relays clacking in the dark — quietly connecting your line to anyone’s, anywhere.

You were holding the end of a network that wrapped the entire planet. You just used it to ask if anyone was home.

        _.-._
       / ___ \
      | /   \ |     "...around the corner...
      | \___/ |          one more inch..."
       \_____/
          |
        __|__
       (_____)
       /  |  \____
      |   |       )___
       \  |   ___/    )___
        \ |__/    ___/    )___
         \|      /    ___/    )
          \_____/    /    ___/
                 \__/____/
              wall ---> the whole world

The cord is gone now. You carry the phone and the central office in your pocket, and it never tangles.

But the stretch never left you.

You still wander the house looking for the quiet room. You still pace to the edge of the WiFi. You still find the one spot — by the window, on the stairs — where the signal holds and nobody can hear you.

The leash got invisible.

You’re still pulling it as far as it’ll go.

The dashboard was green. That’s the part that still bugs me.

A customer calls because a workstation died and they need yesterday’s file back. Easy day. I pull up the backup server, find the machine, and go to restore. Except there’s nothing to restore. Last successful backup: never. Not “stale.” Not “failed.” Never.

But the agent was online. Heartbeat OK. Service running OK. Sitting there reporting in like a good little soldier. It just hadn’t backed up a damn thing the entire time it had been installed.

The investigation

First instinct: blame the agent. So I remoted in and read the client logs. The agent was happily completing runs. No errors. It thought it was doing its job.

So now I’ve got an agent that says “I backed up” and a server that says “this machine has never backed up.” Both swear they’re telling the truth. When two systems disagree about reality, the disagreement is almost always about identity.

I checked what name the agent was registered under on the server. Then I checked the machine’s actual OS hostname.

They didn’t match.

┌──────────────────────────┐        ┌──────────────────────────┐
│      THE WORKSTATION      │        │      BACKUP SERVER        │
│                           │        │                           │
│  hostname:  WS-FRONTDESK  │        │  client:  frontdesk-pc    │
│  agent runs OK            │        │  keyed by name ───┐       │
│  "backup complete" OK     │        │                   ▼       │
└───────────┬──────────────┘        │   ┌───────────────────┐   │
            │  reports as            │   │ frontdesk-pc      │   │
            │  WS-FRONTDESK ─────────┼──►│ (PHANTOM CLIENT)  │   │
            │                        │   │ last backup: none │   │
            ▼                        │   └───────────────────┘   │
   bytes go ►►► nowhere              │   WS-FRONTDESK: not found │
└──────────────────────────┘        └──────────────────────────┘

The agent identified itself by the machine’s real hostname. The server had the client filed under a different, hand-typed name from install day. So the agent checked in as WS-FRONTDESK, the server looked for a client called WS-FRONTDESK, found nobody, and the actual data went into the void. Meanwhile the frontdesk-pc entry someone created manually just sat there forever, eternally “waiting for first backup,” looking close enough to green that nobody clocked it.

The “aha”

Backup identity is the hostname. Not a label you pick. Not a friendly name. The hostname.

The server keys everything — auth, storage path, history — off the client name. If that name doesn’t equal what the OS reports as hostname, the agent and server are two ships passing in the night, both convinced they’re docked.

The fix

Stop guessing. Ask the machine its own name, on the box itself:

# On the workstation — what does the OS actually call itself?
hostname

# Verify what the agent will report
[System.Net.Dns]::GetHostName()

That gave me WS-FRONTDESK. The server had frontdesk-pc. So I deleted the phantom client, re-registered the machine under its real hostname, and pushed the server’s preconfigured installer — the one that bakes the correct identity in instead of trusting a human to type it right:

# Remove the dead phantom entry, register the machine by its true hostname,
# then build the preconfigured client installer the server signs.
backup-admin client remove   --name "frontdesk-pc"
backup-admin client add      --name "WS-FRONTDESK"
backup-admin installer build --client "WS-FRONTDESK" --out ./install-WS-FRONTDESK.exe

Ran the signed installer on the box, kicked off a manual run, and watched the server. First full backup completed. Restore point exists. The phantom is gone.

(The other valid fix is to rename the machine to match the registered name — but renaming a production workstation is a bigger blast radius than fixing a backup entry. Match the agent to the host, not the host to a typo.)

Why it happened

Someone registered the client by hand and typed a “nicer” name than the ugly auto-generated hostname. The installer was generic, so identity was never enforced — it trusted the typed name. Two sources of truth, no reconciliation, and a monitoring view that treated “client exists and pings” as “client is protected.” Existence isn’t protection. A backup you can’t restore is just a process eating CPU.

Takeaways

• Backup identity is the hostname. The registered client name must equal hostname on the machine — verify it, don’t assume it.
• A green agent proves nothing. Online does not mean backing up. The only real green is a recent, restorable recovery point.
• Use the server’s preconfigured installer. Hand-typed client names are how phantom clients are born. Bake identity in.
• Alert on silence. Add a “no successful backup in N hours” check so an empty backup history screams instead of sitting there looking calm.
• When two systems disagree about reality, suspect identity first. Auth, naming, and keys cause more “impossible” failures than disk or network ever will.

A client kept their notes vault in git. Years of markdown. Meeting notes, runbooks, half-finished ideas — the kind of stuff you don’t think about until it’s gone.

They wanted to stop tracking one big folder. Reasonable. It didn’t belong in version control.

So they ran the obvious command:

git rm -r attachments/

Git chewed on it for a second and printed a wall of rm 'attachments/...' lines. Looked fine. Then git status showed something that made my stomach drop.

It wasn’t just the attachments/ folder.

deleted:    notes/2024/standup.md
deleted:    notes/2024/oncall.md
deleted:    runbooks/dns.md
... 30,206 more ...

The working tree was empty. ls came back nearly bare. 30,209 markdown files were gone from disk. Not staged-for-deletion-someday. Gone. Off the filesystem.

The investigation

The panic version of this story ends with a restore from last night’s backup and a lost afternoon. The calm version starts with one question: where does git actually delete from?

People think git rm means “stop tracking this file.” It doesn’t. git rm removes a file from two places at once: the index (staging area) and the working tree — your actual disk.

┌──────────────┐   ┌──────────────┐   ┌──────────────┐
│ Working Tree │   │    Index     │   │   HEAD/Repo  │
│  (your disk) │   │  (staging)   │   │ (last commit)│
└──────┬───────┘   └──────┬───────┘   └──────┬───────┘
       │                  │                  │
  git rm ──► ✗ DELETES ──►│ ✗ stages delete  │ ✓ STILL HERE
       │                  │                  │
  git rm --cached ─────►  │ ✗ untracks only  │ ✓ STILL HERE
       ▼                  ▼                  ▼
   file gone          marked rm          recoverable

That last column is the whole ballgame. git rm touches the working tree and the index, but it does not touch your last commit until you actually commit the deletion.

Nobody had committed yet.

The files were sitting safe in HEAD the entire time, even as the disk looked like a crime scene.

The aha

If HEAD still has every file, recovery is just “copy them back out of the last commit.” No backup tape, no remote clone, no heroics.

The fix

First, confirm the damage scope. Trust nothing, count everything:

git status --porcelain | grep -c '^D'
# 30209

30,209 staged deletions. Matches the missing files exactly. Good — nothing weird, just one bad command.

Now restore the working tree and reset the index straight from HEAD:

git restore --staged --worktree .
# or, on older git:
git checkout HEAD -- .

Verify the files actually came back on disk before believing it:

find . -name '*.md' | wc -l
# 30209

All present. git status clean. If HEAD had been corrupt, the same files were still in the remote — git fetch && git restore --source=origin/main --worktree . would have done it from there.

Then we did the thing they meant to do — untrack without deleting:

git rm -r --cached attachments/
echo 'attachments/' >> .gitignore
git commit -m "Stop tracking attachments/ (keep on disk)"

--cached removes the folder from the index only. The disk is never touched. That one flag is the entire difference between a config tidy-up and a disaster.

Why it happened

git rm carries a default that doesn’t match most people’s mental model. The intent was “git, please forget this exists.” Git heard “delete this from the index and the disk,” because that’s what it does. The -r made it recursive, and the glob pulled in far more than expected.

There was no malice and no bad command syntax. It did exactly what it’s documented to do. The gap was understanding that the working tree is a target, not a bystander.

Takeaways

• git rm deletes from your disk. Use git rm --cached to untrack a file while keeping it on the filesystem — that’s almost always what you actually want.
• Count before, count after. find . -name '*.md' | wc -l on both sides of any bulk git op. A number that drops by 30,209 is not a surprise you want at commit time.
• Test on ONE file first. Run the destructive command against a single path, check git status and ls, then scale up. One file lost is a shrug; thirty thousand is a bad day.
• Don’t run destructive git in a live data directory without a backup. A notes vault is data, not just a repo. Treat it like one.
• Know your three trees. Working tree, index, HEAD. Recovery is trivial when you know which one still has your files — and panic when you don’t.

The Sky Was the Cable Box

You remember the antenna on the roof.

That big aluminum bone-cage bolted up by the chimney.
Pointing somewhere. Pointing at something.
You never knew what.

It just lived up there.
Catching things out of the air.

And you knew the rules without anybody teaching you.

When the picture went to static, you didn’t call a repairman.
You didn’t reboot anything.

You sent a person to stand in a doorway and not move.

The vibes:

• the screen fuzzed and Dad yelled “HOLD IT RIGHT THERE”
• somebody’s arm became part of the TV
• a storm was coming and you could feel the channels getting nervous
• the wind made the picture breathe
• adjusting it on the roof was a forbidden, slightly heroic act

It felt like the antenna was alive.
Like it had moods.
Like it was listening to the sky and sometimes the sky talked back in snow.

Here’s what it actually was.

That metal skeleton wasn’t decoration. It was a receiving array — a careful arrangement of metal rods cut to specific lengths, each one tuned to grab a slice of the radio spectrum where TV stations lived (VHF down low, UHF up high).

The stations broadcast their signal as electromagnetic waves from a tower miles away. Those waves washed over your whole neighborhood, invisible. The antenna’s job was to let those waves nudge a tiny voltage into the metal — and your TV amplified that whisper into Saturday morning cartoons.

The reason your cousin had to freeze in the doorway? Multipath. The signal bounced off hills, buildings, even people, and the bounced copy arrived a hair late, smearing the picture into ghosts and snow. Your cousin’s body was a wall. Moving them changed the math. You were, no exaggeration, hand-tuning a radio-frequency interference pattern with a human being.

The big rotator boxes that spun the antenna? Those folks were literally re-aiming at a different tower in a different town.

        |
       =|=        <- the little rods caught
      ==|==          UHF (the high channels)
     ===|===
        |
   =====|=====   <- the long rods caught
        |            VHF (2 through 13)
   =====|=====
        |
        |  ~ ~ ~ ((( signal from the tower )))
   _____|_____
  /  rooftop  \

It never really left, you know.

We buried it in coax and called it cable. We beamed it down from dishes. We streamed it over fiber.

But cut the cord today and put up a cheap flat antenna, and the local channels still come down out of the sky — free, crisp, digital — same towers, same physics, no human in the doorway.

The sky’s still broadcasting.

We just stopped looking up.