The ticket said: “The server is possessed.”

That’s a direct quote. An app had been rock-solid for two years, then one Tuesday it “just stopped” — no config change, no deploy, no nothing. The on-call engineer had already rebooted twice, blamed cosmic rays, and was halfway to blaming the building’s wiring.

It wasn’t possessed. It never is.

After enough years you learn that 95% of “haunted network” tickets are the same seven mundane bastards wearing bedsheets. None of them are mysterious. All of them are one check away from being solved. The trick is running the check before you start theorizing about ghosts.

Here’s the scene I keep coming back to.

  THE GHOST                        THE CHECK              VERDICT
  ─────────────────────────        ──────────────────     ──────
  "service won't start"     ──►    who owns the port?     ✓ orphan
  "site loads wrong server" ──►    what does dig say?     ✓ stale DNS
  "one client 500s"         ──►    force --http1.1        ✓ h2 quirk
  "big transfers hang"      ──►    ping -M do -s 1472     ✓ MTU/MSS
  "intermittent dropouts"   ──►    arp -a (two MACs?)     ✓ dup IP
  "works one direction"     ──►    firewall state table   ✓ asymmetric
  "localhost can't connect" ──►    loopback fw rule        ✓ blocked pipe

The Investigation, Compressed Into a Checklist

1. The service that “can’t start.” Symptom: it crashed, you restart it, and now it refuses to bind. The aha: the crashed process orphaned the port and the kernel still thinks it’s owned. The check — find the squatter:

# Linux
ss -ltnp 'sport = :8080'
# Windows
netstat -ano | findstr :8080

Kill the orphan PID, the port frees, the service starts. No exorcism required.

2. DNS serving a corpse. Symptom: you migrated a site, half the world sees the new box, this one client sees the old one. The check — ask what’s actually resolving, not what should:

dig +short app.example.com @1.1.1.1
nslookup app.example.com
# flush the local liar:
sudo resolvectl flush-caches      # systemd
ipconfig /flushdns                # Windows

Stale cache, gone.

3. The one client that 500s while everyone else is fine. That’s an HTTP/2 downgrade or framing incompatibility. Pin the protocol and watch it heal:

curl -v --http1.1 https://app.example.com/health

If --http1.1 works and the default doesn’t, you found your ghost.

4. Small requests fine, big ones hang. Classic MTU/MSS mismatch — pings and tiny payloads sail through, the TLS handshake or a large transfer stalls forever. Probe for the real path MTU:

# 1472 + 28 overhead = 1500. Shrink until it stops fragmenting.
ping -M do -s 1472 192.0.2.1

If 1472 fails but 1400 works, clamp your MSS and the hangs vanish.

5. The intermittent flapper. Symptom: connectivity that drops for seconds at random — a rogue DHCP server or a static IP collision. Check the ARP table for one IP claimed by two MACs:

arp -a | sort        # two different MACs on the same IP = your culprit
ip neigh show

6. Works one way, dies the other. Asymmetric routing where the return path takes a different door and a stateful firewall drops it for having no matching session. Look at the state table:

# Linux conntrack
conntrack -L | grep 192.0.2.50
# pf
pfctl -ss | grep 192.0.2.50

No state entry for the return flow = there’s your “ghost.”

7. The app that “should be local” but can’t reach itself. A host firewall rule or blocked named pipe is eating loopback. Confirm the listener and that localhost itself is allowed:

ss -ltnp | grep 127.0.0.1
sudo iptables -L INPUT -n -v | grep -E 'lo|127.0.0.1'

That Tuesday “possession”? Number one. Crashed worker, orphaned port, ss -ltnp named the PID in four seconds. Ghost dispelled.

Why It Happens

None of these are exotic. They’re the natural failure modes of stateful systems: processes die untidily, caches outlive their truth, two devices want the same address, a firewall remembers a flow that no longer exists. The supernatural feeling comes entirely from partial symptoms — it works for some and not others, or for small things and not big ones. Partial failure reads as spooky. It’s just state you haven’t looked at yet.

Takeaways

• Run the seven-item checklist before you theorize. The boring answer is almost always right.
• “Some clients work” is a clue, not a contradiction — it points straight at DNS, HTTP/2, or MTU.
• “Small works, big hangs” is MTU/MSS until proven otherwise. Don’t waste an hour on app logs.
• Check state, not config. ss, dig, arp, and the firewall state table tell you what is, not what should be.
• Reboots hide orphans; they don’t explain them. Find the squatter PID before you cycle power and lose the evidence.

The ticket landed with a number attached, and the number was wrong.

A monitoring report — the one we generate so nobody has to think too hard — claimed a site was running 99 wireless access points. Ninety-nine. For a building that, last I checked, you could walk end to end during a coffee that hadn’t gone cold yet.

I’d been in that building. I knew the ceilings. There were not 99 APs up there. There weren’t even a quarter of that.

So either the site had quietly become an enterprise campus overnight, or the report was hallucinating. Spoiler: reports don’t hallucinate. They count exactly what you tell them to count, which is a different and worse problem.

The investigation

First rule when a number looks insane: don’t argue with the number. Go find ground truth and make the number argue with that.

I pulled the controller’s live device list — the actual adopted, talking-to-me-right-now hardware:

# adopted APs the controller is actively managing
curl -sk -b /tmp/controller.cookie \
  https://controller.example.internal/proxy/network/api/s/default/stat/device \
  | jq '[.data[] | select(.type=="uap")] | length'

12

Twelve. Twelve adopted access points, online, real, with MAC addresses I could go put my hand on.

So where did the report get 99?

I went and read the thing that built the report instead of the report itself. The collector wasn’t querying live adopted devices. It was scraping a broader chunk of the controller database — and it was counting rows, not radios you could hang on a wall.

The aha

Three lies were stacked on top of each other, and the report added them all up with a straight face.

  WHAT THE REPORT COUNTED            WHAT WAS ACTUALLY THERE
  ─────────────────────────          ───────────────────────
  ┌───────────────────────┐
  │ adopted APs        12 │ ◄──────── ✓ real, on the ceiling
  ├───────────────────────┤
  │ pending / forgotten   │
  │ ghost DB rows      ~30│ ◄──────── ✗ stale, never cleaned
  ├───────────────────────┤
  │ per-radio entries     │
  │ (2.4 / 5 / 6 GHz)     │ ◄──────── ✗ one AP counted 3×
  ├───────────────────────┤
  │ per-SSID rows         │ ◄──────── ✗ counted broadcasts
  └───────────────────────┘            as devices
        Σ = 99                          Σ (real) = 12

Every tri-band AP showed up as three radio entries. Every SSID broadcast left its own footprint. And the controller DB still held a graveyard of devices that had been swapped out, RMA’d, or “forgotten” in the UI months ago — rows that never actually leave the database, they just stop mattering. To a human. Not to a COUNT(*).

Stack the real twelve on top of the radio multiplication on top of the ghost rows, and you arrive — with perfect arithmetic and zero truth — at 99.

The fix

The fix wasn’t clever. It was narrowing the question. Count adopted UAP-type devices, deduplicated by MAC, and exclude anything pending or forgotten:

curl -sk -b /tmp/controller.cookie \
  https://controller.example.internal/proxy/network/api/s/default/stat/device \
  | jq '[.data[]
         | select(.type=="uap")
         | select(.adopted==true)
         | select(.state==1)          # 1 = connected
         | .mac]
        | unique | length'

12

Then I did the thing the database can’t: I walked the building and counted ceilings. Twelve.

The report now pulls that single reconciled number, and it carries a footnote: adopted, connected, deduped by MAC. If it ever disagrees with the live device list again, it’s the report that has to explain itself.

Why it happened

Because counting feels trivial, so nobody guards it. length on a JSON array looks like truth. But the controller DB isn’t a list of access points — it’s an event log of everything that ever claimed to be one: every radio, every SSID broadcast, every device that got swapped and never purged. Counting rows in that pile and calling it “APs” is like counting fingerprints and reporting the number of burglars.

The artifact survived for months because 99 was plausible-adjacent. Nobody who’d been on-site ever read the report; nobody who read the report had ever been on-site. The number floated in the gap between them, fat and unchallenged, until somebody finally looked up.

Takeaways

• Never report an inventory number you can’t reconcile to ground truth. A count needs two witnesses: the live device list and a physical walk.
• Rows are not things. Per-radio and per-SSID entries multiply one AP into many. Filter by device type and dedupe by MAC before you trust a total.
• Controller databases keep ghosts. Forgotten, pending, and RMA’d devices linger in the DB long after they leave the ceiling. Scope your query to adopted and connected.
• Plausible-adjacent is the dangerous zone. A wrong number that’s obviously insane gets caught. One that’s merely “a bit high” rides for months.
• Read the query, not the report. When a metric lies, the bug is upstream in what you told it to count — go fix the question.

The ticket said “internet is down.” Then, four minutes later, “never mind, it’s back.” Then, twenty minutes after that, “it’s down again.”

That pattern is a special kind of hell. A clean outage you can chase. A flapping one that heals itself before you SSH in just laughs at you.

I got a client on the phone. Their whole site would blackhole — every desktop, every phone, the printer, all of it — for one to five minutes, then snap back like nothing happened. No schedule. No trigger anyone could name. The ISP swore the circuit was clean, and for once the ISP was right.

The investigation

I started where you start: at the edge. The gateway was up. WAN was up. The firewall logs were boring in the way you want them to be boring. No flapping interface, no CPU spike, no DHCP storm in the leases.

So I sat on a workstation and just hammered the gateway while I waited for the next outage.

ping -i 0.5 192.0.2.1

It ran fine for ten minutes. Then — dead air. Eight, nine, ten dropped replies. Total blackhole. Then it came back on its own.

The instant it died, I dumped the ARP table.

arp -a 192.0.2.1

And there it was. The gateway’s IP was mapped to a MAC address I did not recognize. Not the firewall’s MAC. Some OUI I had to look up by hand.

I watched it flap in real time:

┌─────────────────────────────────────────────────────────────┐
│  arp -a 192.0.2.1   (watched over ~30s)                      │
├─────────────────────────────────────────────────────────────┤
│  t+0s   192.0.2.1  ►  aa:bb:cc:11:22:33   (firewall)   ✓     │
│  t+6s   192.0.2.1  ►  de:ad:be:ef:00:99   (???)        ✗     │
│  t+9s   192.0.2.1  ►  de:ad:be:ef:00:99   (???)        ✗     │
│  t+14s  192.0.2.1  ►  aa:bb:cc:11:22:33   (firewall)   ✓     │
│  t+19s  192.0.2.1  ►  de:ad:be:ef:00:99   (???)        ✗     │
└─────────────────────────────────────────────────────────────┘
         ▼ when the IP pointed at de:ad:..:99, traffic vanished

When the gateway IP pointed at the firewall, the network worked. When it pointed at that mystery MAC, every packet bound for the internet got handed to a device that did absolutely nothing with it. Classic ARP poisoning. The only question was who.

The “aha”

I looked up the OUI. The mystery MAC belonged to a Tuya-based Wi-Fi module — the kind that ships inside cheap smart-home gadgets by the millions.

Then I cross-referenced the DHCP leases for that MAC and found a hostname that made me laugh out loud: a smart window blind controller. Somebody had put a $30 motorized blind on the corporate flat network.

That little NIC was broadcasting unsolicited ARP replies claiming to be 192.0.2.1. Every client on the segment believed it, updated its ARP cache, and started shipping its default route straight into a window covering.

The fix

First, stop the bleeding. I pinned the gateway’s real MAC on the affected machines so they’d stop trusting the lie while I worked.

# Linux
sudo ip neigh replace 192.0.2.1 lladdr aa:bb:cc:11:22:33 dev eth0 nud permanent

Then the real fix — get IoT off the flat network entirely and turn on the switch protections that should’ve been on from day one.

BEFORE                              AFTER
──────                              ─────
VLAN 1 (everything)                 VLAN 10  desktops/servers
  ├─ desktops                       VLAN 20  voice
  ├─ servers                        VLAN 90  IoT  ◄── blind lives here,
  ├─ phones                                          firewalled, no L2
  └─ smart blind  ◄── poisoner                       reach to clients

# Cisco-style switch hardening (concept)
# Trust only the uplink for DHCP; drop rogue offers elsewhere
ip dhcp snooping
ip dhcp snooping vlan 90
interface Gi1/0/24
 ip dhcp snooping trust

# Dynamic ARP Inspection — validate ARP against snooping bindings
ip arp inspection vlan 90
interface Gi1/0/24
 ip arp inspection trust

Last step: I yanked the blind, pushed its vendor firmware update, and parked it on the isolated IoT VLAN where it can lie about being the gateway all it wants — and nobody who matters will hear it.

Why it happened

Cheap Tuya firmware is a grab bag. Some builds have buggy network stacks that emit malformed or unsolicited ARP — gratuitous ARP gone feral, sometimes triggered by a Wi-Fi reconnect, which explains the random timing. Could also be straight-up malicious; with no-name IoT you genuinely can’t tell, and it doesn’t change the remediation.

Either way, a flat L2 network trusts every device equally. ARP has no authentication. Whatever shouts loudest and last, wins. Put one mouthy gadget on that wire and it can pretend to be anything — including the door to the internet.

Takeaways

• On weird, intermittent, self-healing outages, check the gateway’s ARP entry FIRST. arp -a <gateway-ip> flapping to an unknown MAC is a five-second smoking gun.
• Match a rogue MAC to its OUI and your DHCP leases. That’s how a “mystery device” became “the smart blind in the break room.”
• IoT does not belong on your flat network. Ever. Its own VLAN/SSID, firewalled off from clients, no L2 reach to anything that matters.
• Turn on DHCP Snooping and Dynamic ARP Inspection. They exist precisely to kill rogue gateways and ARP lies at the switch.
• Cheap IoT is an untrusted host by default — bug or malice, you treat it the same: isolate, update, and never let it speak for the gateway.

The Night We Carried Our Computers Into the Basement

You remember the LAN party.

Not a download.
Not a lobby.
A pilgrimage.

You unplugged your entire desktop — the beige tower, the CRT the size of a microwave, the rat’s nest of cables — and you carried it, in your arms, like a wounded soldier, to somebody’s basement.

The monitor weighed forty pounds.
You did this on purpose.
For fun.

And when you got there, there was already a folding table, an extension cord doing something it was definitely not rated for, and a bowl of Doritos that would last exactly nine minutes.

Here’s what it felt like as a kid:

• Forbidden — six computers in one room felt like a hacker movie
• Sacred — nobody touched the host’s machine. He was the king.
• Slightly dangerous — that many power bricks in one outlet? bold
• A little magic — your screen and his screen showing the same world
• Eternal — it was 2am and nobody’s mom had come downstairs yet

You’d hear someone yell “WHO HAS THE GRAY CABLE” and the whole night hinged on it.

The reveal

That gray cable was the whole trick.

It was a Cat-5 Ethernet cable, and it ran to a little blinking box in the middle of the table — a hub (later, the fancy kids had a switch). Every computer plugged into it, and that box let your machines gossip with each other at a blistering 10 megabits a second.

No internet involved. None. The whole party was a tiny private island of computers — a Local Area Network — talking only to each other.

Before someone bought the hub, you did it the cursed way: a coaxial cable daisy-chained machine to machine, with a little metal terminator screwed onto each end. If one person tripped over the line, the entire network died — and everyone knew whose fault it was.

And the game finding everyone? That was your machine literally shouting into the wire — “anybody out there hosting?” — and a host shouting back. That’s why someone always had to “make the server.” He wasn’t being dramatic. He was being the server.

   .------.   .------.   .------.
   |  PC  |   |  PC  |   |  PC  |
   '--||--'   '--||--'   '--||--'
      ||         ||         ||
   ===++=========++=========++===
      |     [ H U B ]   .|.|.     |
      '------ blink blink --------'
        every box hears every box

Where it went

It never really died — it just stopped weighing forty pounds.

That hub in the middle of the table? It moved into the little plastic router blinking in your hallway right now. That “shout into the wire to find a host”? That’s matchmaking, except now the wire is the whole planet and the host is a warehouse in Virginia.

You still play with your friends.
You just don’t carry your computer to do it.

But some part of you misses it — the cables, the chaos, the bowl of Doritos, the gray cable that held the whole night together.

We didn’t have multiplayer.

We built it, in a basement, every single time.

Someone pinged me with the kind of bug that makes your skin crawl: “I save a note, it saves fine, no error. But when I come back later it’s gone. And sometimes I open a file and it’s the wrong file entirely.”

No errors. No crash. Data just… wrong. The worst kind of ticket — because “wrong data, no error” means the system is lying to you with a straight face.

I assumed corruption. Maybe a bad sync, a stale cache, a half-written file on disk. I was wrong about all of it.

The investigation

First thing I did was stop trusting the app and start trusting the wire. The app talked to itself over a local REST API on a fixed port. So I asked the only question that matters when data is “wrong”:

Am I even talking to the thing I think I’m talking to?

The user ran two separate profiles of the same application — two distinct workspaces, two configs, two processes. Different data, different windows. But same binary, same defaults.

I checked what was actually listening.

        Profile A (started first)        Profile B (started second)
        ┌──────────────────────┐         ┌──────────────────────┐
        │  app instance A      │         │  app instance B      │
        │  workspace: WORK     │         │  workspace: PERSONAL │
        └──────────┬───────────┘         └──────────┬───────────┘
                   │ bind :8200 OK                  │ bind :8200 FAIL
                   ▼                                ▼
            ┌─────────────┐                  (silent: port taken)
            │  TCP :8200  │ ◄──── requests land HERE
            └─────────────┘        regardless of which UI you used
                   ▲
                   │
            edits in B's UI → API call → answered by A

Both instances were configured to expose their REST API on the same hard-coded port. Whichever process won the startup race grabbed the socket. The loser tried to bind, failed, and — because the failure was non-fatal — just shrugged and kept running its UI as if nothing happened.

So you’d type into Profile B’s window. The UI would fire an API write. That write went to port 8200, which Profile A owned. Profile A happily saved it into the wrong workspace. Reads came back from A too. Profile B’s UI was a puppet with cut strings.

The “aha”

The tell was netstat. One PID owned the port. Not two. Not a conflict logged anywhere — just one quiet winner.

# Windows: who actually holds the port?
netstat -ano | findstr :8200
#   TCP    127.0.0.1:8200    0.0.0.0:0    LISTENING    18044

tasklist /fi "PID eq 18044"
#   app.exe    18044   ...   <- this is Profile A, not B

# Linux/macOS equivalent
lsof -nP -iTCP:8200 -sTCP:LISTEN
ss -ltnp 'sport = :8200'

One listener. Two apps. The data wasn’t corrupt — it was just going to the wrong house, and the mailman never complained.

The fix

Stop them from racing for the same socket. Give each instance its own port.

# 1. Edit each profile's config — distinct ports per instance
#    Profile A
"api": { "enabled": true, "port": 8200 }
#    Profile B
"api": { "enabled": true, "port": 8201 }

# 2. Fully stop both instances (no orphan holding the old socket)
taskkill /f /im app.exe        # Windows
# pkill -f app                 # Linux/macOS

# 3. Confirm the ports are actually free before restart
netstat -ano | findstr "8200 8201"   # expect: nothing

# 4. Start both, then verify ownership — two PIDs, two ports
netstat -ano | findstr "8200 8201"
#   TCP  127.0.0.1:8200  LISTENING  20110   <- Profile A
#   TCP  127.0.0.1:8201  LISTENING  20338   <- Profile B

Two listeners, two PIDs, two ports. Writes to B stayed in B. Reads told the truth. The “disappearing edits” were gone because they’d never disappeared — they’d just been filed in the wrong cabinet.

Why it happened

Defaults. The app shipped with a single hard-coded API port and assumed exactly one instance would ever run. Reasonable for the common case, fatal for power users running multiple profiles. There was no port-conflict warning because a failed bind on a convenience API isn’t fatal — the app degrades silently instead of refusing to start. Silent degradation is how a config bug masquerades as data loss.

Takeaways

• When data looks “wrong” but nothing errors, question the endpoint first. Confirm you’re talking to the instance you think you are before you go disk-spelunking for corruption.
• netstat -ano / lsof -i is ground truth for “who owns this port.” One listener where you expected two is the whole bug, right there.
• Multiple instances of the same app will knife-fight over hard-coded ports. Give every instance a unique port in its config — don’t trust defaults to coexist.
• A non-fatal bind failure is a trap. Silent degradation turns a 5-minute config fix into a week of phantom data-loss tickets.
• Pre-flight before restart: kill all instances, verify the port is actually free, then start — and read back the listeners to prove it stuck.

The bot worked. Then it didn’t. Then it worked again.

It was a small Node.js Telegram bot running on a client’s server. Its whole job was to fire off outbound API calls — sendMessage, getUpdates, the usual. Most of the time it was instant. But every so often a call would just… stop. No exception. No 500. No DNS error. The promise sat there until the request timeout finally put it out of its misery seconds later.

Intermittent. Silent. Multi-second. The worst kind of bug there is.

The investigation

First instinct was the obvious one: the upstream API is flaky. So I logged timing around every call. The hangs weren’t correlated with anything — not load, not time of day, not a specific endpoint. Same call, same payload, same host. Sometimes 80ms, sometimes 5,000ms-then-timeout.

I pulled strace on the process during a hang and watched the syscalls. The bot wasn’t busy. It was sitting in a connect() that never came back. It wasn’t waiting on the API — it was waiting on a socket that was never going to open.

That reframed everything. This wasn’t a slow server. This was a connection attempt going into the void.

   bot ──► undici (happy-eyeballs)
              │
              ├─► resolve host
              │     ├─ AAAA (IPv6)  ─► ::  ►  connect() ──► ✗ black hole (no route)
              │     └─ A    (IPv4)  ─► .   ►  connect() ──► ✓ would work fine
              │
              ▼
        races both... but stalls on the v6 attempt

The host had IPv6 addresses configured on its primary NIC. It just had no working IPv6 route to the outside world. Addresses present, connectivity dead. The packets went out and nothing ever came back.

The “aha”

The bot used undici as its HTTP client. Modern undici does Happy Eyeballs (RFC 8305) — it resolves both A and AAAA records and races IPv4 and IPv6 connection attempts, preferring the v6 path.

On a healthy host that’s great. On this host it was poison. undici would get an AAAA record, try the IPv6 path first, and that connect() would sail into a black hole. The race timer is supposed to fall back to IPv4 — but with a route that silently drops packets instead of cleanly rejecting them, the fallback timing slipped and the whole call stalled until the app-level timeout.

A quick curl proved it cold:

# hangs forever — there's an AAAA record and broken v6 routing
curl -v https://api.example.com/

# instant — forced down IPv4
curl -4 -v https://api.example.com/

-6 hung. -4 flew. That’s your smoking gun.

The fix

Two layers. First, kill IPv6 on the one NIC that had it broken — while leaving it alone on the VPN and VM interfaces that genuinely use it.

# disable IPv6 on the offending interface only (e.g. eth0), keep it elsewhere
sudo sysctl -w net.ipv6.conf.eth0.disable_ipv6=1

# make it stick across reboots
echo 'net.ipv6.conf.eth0.disable_ipv6 = 1' | sudo tee /etc/sysctl.d/99-disable-ipv6-eth0.conf
sudo sysctl --system

Second, belt and suspenders at the app layer — force the resolver IPv4-first so even a half-broken host can’t bite us again:

// top of the entrypoint, before any outbound calls
const dns = require('node:dns')
dns.setDefaultResultOrder('ipv4first')

curl -4 already proved the IPv4 path was healthy, so this was a safe, deterministic pin. Restarted the bot, hammered it with calls for ten minutes. Zero hangs. Every request back under 200ms.

Why it happened

“Broken but present” IPv6 is worse than no IPv6 at all. If the host had no AAAA-resolvable path, the client never tries v6 and you never notice. But give it IPv6 addresses with no real route, and every modern HTTP client cheerfully tries the dead path first — then waits on a connect() that silently drops instead of refusing.

No RST means no fast failure. No fast failure means a stall. And because Happy Eyeballs only sometimes loses the race badly, the stall is intermittent. That’s why it looked like a flaky upstream when it was a flaky local NIC the whole time.

Takeaways

• If something hangs only sometimes on outbound HTTP, suspect IPv6. Test it in one command: curl -4 vs curl -6. If -6 hangs, you found it.
• Addresses ≠ connectivity. A NIC can hold a perfectly valid IPv6 address and still have zero working route to the internet.
• Disable IPv6 per-interface, not globally. Kill it on the broken NIC; leave it on VPN/VM links that actually need it.
• Pin IPv4-first at the app layer too (dns.setDefaultResultOrder('ipv4first')) — defense in depth for hosts you don’t fully control.
• A connect() that never returns is a routing problem, not a server problem. strace the hang before you blame the API.

There’s a box at a remote site. It lives behind NAT, behind a router I don’t control, on a connection that hands out a new public IP whenever the ISP feels like it.

To reach it, I had a ritual. SSH into a bastion. From the bastion, SSH into the target. Type two passphrases, pray nothing changed, and hope the WAN IP I memorized last month was still the WAN IP.

ssh -J bastion target

That one flag hid a small empire of fragility.

The investigation

It broke on a Tuesday, the way these things do. ssh: connect to host ... port 22: Connection timed out.

First instinct: the target is down. It wasn’t — a colleague on-site confirmed it was humming along, recording, serving, fine.

Second instinct: the bastion is down. It wasn’t either. I could land on the bastion clean.

ssh bastion 'echo alive'
alive

So the bastion was up, the target was up, and the hop between them was dead. I jumped onto the bastion and tried the second leg by hand.

ssh me@198.51.100.x
ssh: connect to host 198.51.100.x port 22: No route to host

There it was. The remote site’s WAN IP had rotated overnight. The 198.51.100.x baked into my SSH config — and into the bastion’s known_hosts, and into a port-forward rule, and into my muscle memory — now pointed at some stranger’s DHCP lease.

The “aha”

I’d built a chain of three brittle links to solve one problem: the target has no stable address I can reach from outside.

Every fix I’d ever applied — dynamic DNS, a forwarded port, a second hop — was just another way of chasing an address that refused to hold still. The bastion wasn’t a solution. It was a workaround for a missing fact.

What if the box just had a stable address? One that didn’t care about NAT, ISP whims, or which coffee shop my laptop was sitting in?

   BEFORE                                AFTER
 ┌─────────┐                          ┌─────────┐
 │ laptop  │                          │ laptop  │
 └────┬────┘                          └────┬────┘
      │ ssh                                │ ssh
      ▼                                    │ (encrypted mesh)
 ┌─────────┐                               │
 │ bastion │  ◄── public IP that          │
 └────┬────┘      keeps rotating          │
      │ ssh                                ▼
      ▼                                ┌─────────┐
 ┌─────────┐   NAT / CGNAT             │ target  │
 │ target  │   no inbound port         │ 100.x.. │ stable tailnet IP
 └─────────┘                           └─────────┘

That address is exactly what a WireGuard-based mesh VPN hands you. Put the laptop and the target on the same tailnet, and each device gets a private IP that follows it everywhere — through NAT, through CGNAT, through IP rotations — because the mesh handles NAT traversal for you.

The fix

Install the mesh on both ends. (Generic Tailscale-style flow below.)

# On the target — behind NAT, no inbound ports needed
curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --ssh --hostname remote-target

# On my laptop
tailscale up

# Confirm the target shows a stable tailnet IP
tailscale status
# 100.x.y.z   remote-target   linux   active; direct

Now SSH goes straight there. No bastion, no -J, no rotating address.

ssh me@100.x.y.z
# or, by MagicDNS name
ssh me@remote-target

Lock it down with an ACL so only my devices can reach port 22 on that box:

{
  "acls": [
    { "action": "accept", "src": ["tag:admin"], "dst": ["tag:remote:22"] }
  ]
}

Then I deleted the bastion’s forward rule, pulled the stale host key, and retired the jump chain entirely.

ssh-keygen -R 198.51.100.x

Why it happened

The bastion was never the point. It existed because the target had no reachable, durable address — so I borrowed one from a machine that did, then chained a second SSH session on top.

Every link in that chain depended on a public IP staying put. One of them rotated, and the whole thing collapsed. That’s not a bug in SSH. That’s the architecture telling you it was held together with assumptions.

A mesh VPN moves the addressing problem off the public internet entirely. The tailnet IP is yours, it’s stable, and it doesn’t care what the ISP does at 3 a.m.

Takeaways

• If you’re chaining ssh -J to reach a NAT’d box, that’s a missing stable address, not a routing puzzle. Fix the address, delete the chain.
• Mesh VPNs (WireGuard-based) give every device a durable private IP that survives NAT, CGNAT, and ISP rotations — no inbound ports to forward.
• Stable addressing kills a whole class of failures: stale known_hosts, dead port-forwards, dynamic-DNS lag, memorized WAN IPs.
• Gate it with ACLs. A flat tailnet where everything reaches everything is just a bigger blast radius. Tag, scope, restrict to port 22.
• The most reliable hop is the one you deleted. Fewer moving parts, fewer 3 a.m. timeouts.

The call came in the way these always do: “the whole network is slow.”

Not down. Slow. Which is worse, because slow means it’s still trying.

I pulled up the switch stats from the site and the picture was ugly. Broadcast counters spinning like a gas pump. Port LEDs across the access switches strobing in unison — that synchronized, hypnotic blink that should make the hair on your neck stand up. CPU on the core switch pinned. Every wired device crawling, half the VLANs timing out.

That’s not congestion. That’s a storm.

The investigation

A broadcast storm has a signature. One frame gets flooded out every port, comes back in another port, gets flooded again, forever, at line rate. The switches aren’t broken — they’re doing exactly what they’re told, infinitely.

So the question is never “what’s slow.” It’s “where’s the loop.”

I started where loops live: the edge. I pulled the MAC address table and watched a handful of addresses flapping between ports — the same MAC showing up on port 14, then port 9, then port 14, dozens of times a second. A stable MAC doesn’t teleport. A looped one does.

show mac address-table | the same MAC, two ports, flapping
─────────────────────────────────────────────────────────
  02:00:00:aa:bb:cc   Gi0/14   ◄─┐
  02:00:00:aa:bb:cc   Gi0/9    ◄─┤  same address,
  02:00:00:aa:bb:cc   Gi0/14   ◄─┤  two ports,
  02:00:00:aa:bb:cc   Gi0/9    ◄─┘  ~40x/sec  ✗

I traced both ports. Port 14 and port 9 went to two units of a consumer multi-room speaker system — the kind that builds its own little wireless mesh between speakers so they stay in sync.

And somebody had wired both of them into the LAN.

The “aha”

There it was. Each speaker had an Ethernet drop. Each speaker was also bridging to its sibling over the proprietary wireless mesh. So a frame could leave the switch on port 14, ride into speaker A, hop the wireless mesh to speaker B, and come right back into the switch on port 9.

Copper out, radio across, copper back in. A loop with one leg made of Wi-Fi.

The reason it had run fine for months and then exploded? Doesn’t matter much. Someone re-cabled a closet, both speakers ended up plugged in, the mesh did its job, and nothing on those access ports was watching for a loop.

The fix

The speaker was the trigger. The real bug was a switch fabric with no loop protection. So we fixed both.

First, the immediate bleeding — pull one of the two speaker uplinks. One leg of copper gone, loop broken, network breathes.

Then, the actual fix. Turn on spanning tree, and pick the root on purpose instead of letting MAC addresses elect a random one:

# Core switch — make it the root, deliberately. Lowest priority wins.
spanning-tree mode rstp
spanning-tree vlan 1-4094 priority 4096

# Distribution — second in line
spanning-tree vlan 1-4094 priority 8192

Then lock down the edge so an end-device port can never become a transit path:

# Access port to the speaker / any end device
interface GigabitEthernet0/14
 spanning-tree portfast
 spanning-tree bpduguard enable     # any BPDU here = err-disable the port
 spanning-tree guard root            # this port may never be a path to root
 storm-control broadcast level 1.00  # cap broadcast at 1% of line rate
 storm-control action shutdown

Mesh-style managed gear is the same idea in different clothes — flip on STP, set tiered priorities so the core is root, and enable Loop Protection plus BPDU Guard on access ports.

edge port discipline
─────────────────────────────────
  PortFast      ✓  fast end-device link
  BPDU Guard    ✓  sees a switch → kills port
  Loop Guard    ✓  detects the loop
  Storm-control ✓  broadcast ceiling
─────────────────────────────────
  result: ▓▓▓▓▓ blast radius = one port

We re-cabled the speakers to a single drop each, verified one stayed on copper and the rest synced over mesh, and watched the broadcast counter flatline.

Why it happened

Consumer mesh gear is a loop machine wearing a friendly face. It will happily bridge two of its own units, and it has no idea it’s also sitting on your switched LAN. Plug two of them into copper and you’ve handed your network a second path it never agreed to.

The switch didn’t fail. It had no spanning tree configured, so it had no mechanism to notice the loop and break it. The default config trusted every port to behave. Edge ports never behave.

Takeaways

• Always run STP/RSTP. A modern loop-free topology still needs spanning tree on standby for the day someone hands you a loop.
• Set the root bridge on purpose. Lowest priority value wins; don’t let a random MAC address decide where your traffic converges.
• Guard the edge. BPDU Guard, Loop Protection, and storm-control on access ports turn a network-wide meltdown into one dead port.
• Treat consumer mesh gear as hostile to your switch fabric. Never wire two mesh-bridging devices into the same LAN.
• A flapping MAC is a loop until proven otherwise. The mac-address-table is your fastest path from “it’s slow” to “it’s port 9 and 14.”

It was supposed to be a five-line automation. Read a value, tweak it, write it back. The kind of thing you knock out between coffees.

I shelled into a client’s pfSense box — a clean, current 24.11 install — and ran my little script through pfSsh.php. It exited without a peep. No errors. No warnings. Just a fresh prompt blinking back at me.

Then the SSH session died.

Then the VPN dropped.

Then my phone lit up.

The investigation

I got back in through the console. The dashboard loaded, and my stomach went cold. No WAN config. No LAN rules. No NAT, no VLANs, no DHCP scopes. The box was sitting there with the bright-eyed innocence of a router that had never met a network.

The config wasn’t corrupted. It was gone. Replaced with defaults, like someone had hit the reset button — except nobody had touched the hardware.

I pulled up my script. Five lines. The offending logic looked like ancient, trustworthy pfSense scripting lore:

global $config;
$config['system']['something'] = 'newvalue';
write_config("scripted tweak");

Every old forum thread, every crusty wiki page from a decade ago, does exactly this. Grab the $config global, poke a key, call write_config(). It’s the canonical move.

So why did it nuke the box?

I added one debug line before the write:

global $config;
var_dump($config);

The output told the whole story in one ugly word: empty.

The “aha”

On modern pfSense, that legacy $config global is no longer the living, fully-populated config tree the old guides assume. In the pfSsh.php scripting context it came back essentially empty — a hollow shell.

And write_config() doesn’t merge. It doesn’t patch. It takes whatever is in memory right now and serializes that as the new, authoritative config. The whole thing.

So my script read an empty global, set one key on top of nothing, and then confidently wrote that “nothing plus one key” back as the complete configuration. write_config() did exactly what it was told. It overwrote a fully working firewall with a near-empty husk.

┌──────────────────────────────────────────────────────────┐
│  WHAT I THOUGHT HAPPENED          WHAT ACTUALLY HAPPENED  │
├──────────────────────────────────────────────────────────┤
│  $config = { full tree }     ►    $config = { } (EMPTY)   │
│        │                                  │               │
│        ▼ set one key                      ▼ set one key   │
│  { full tree + key }              { almost nothing }      │
│        │                                  │               │
│        ▼ write_config()                   ▼ write_config()│
│   ✓ saved correctly               ✗ overwrote everything  │
└──────────────────────────────────────────────────────────┘

The modern config backend wants you to read and write specific paths — config_get_path() and config_set_path() — never the whole global at once.

The fix

First, recover. Restore the last known-good config from a backup and reload:

# Restore from a saved backup and reload
config_restore /cf/conf/backup/config-<latest-good>.xml
/etc/rc.reload_all

pfSense keeps automatic backups under /cf/conf/backup/ — list them with ls -lt /cf/conf/backup/ and pick the newest one from before your script ran. That brought the firewall back to life.

Then, fix the script. Use the path API and stop touching the global entirely:

// Read a specific key (with a default), modify, write that key back.
$value = config_get_path('system/something', 'default');
config_set_path('system/something', 'newvalue');
write_config("scripted tweak via path API");

And — non-negotiable — snapshot the config before any scripted write, so recovery is one command instead of a panic:

cp /cf/config.xml "/cf/conf/backup/config-$(date +%Y%m%d-%H%M%S)-prewrite.xml"

Why it happened

Platforms migrate their internals quietly. pfSense moved to a path-based config backend, and the old global $config got demoted from “the config” to “a vestigial variable that may or may not be populated, depending on context.”

The docs got updated. The thousand forum posts from 2014 did not.

So the trap is perfect: the legacy pattern still compiles, still runs, still exits clean. It just operates on a ghost of the real config — and write_config() is happy to make that ghost permanent.

A function that overwrites everything from in-memory state is a loaded gun. When the thing loading that memory changes underneath you, the gun is now pointed at your whole config and you don’t even know it.

Takeaways

• A global write_config() rewrites the entire config from in-memory state. If that state is stale, partial, or empty, you’ve just factory-reset your box. Treat it as a full-config replace, never a patch.
• When a platform migrates its config backend, the old global is a footgun. It still runs clean — that’s what makes it dangerous. Verify what it actually contains before trusting it.
• Use the scoped API. config_get_path() / config_set_path() touch exactly one key. That’s the blast radius you want.
• Snapshot before any scripted write. cp /cf/config.xml /cf/conf/backup/...-prewrite.xml turns a disaster into a 30-second restore.
• var_dump() your assumptions. One debug line showing an empty global would’ve saved the whole incident. When in doubt, print what you’ve actually got — not what the old guides promise you have.