A customer called on a Tuesday. A critical service had been down for hours. Their first question was the one that ruins your week:

“Why didn’t we get an alert?”

I had built the alerting. I knew the rules existed. I’d watched them fire in testing months ago. So my honest, confident, completely wrong answer was: “You should have.”

That’s the trap. “Should have” is what you say right before you learn something humbling about your own stack.

The investigation

First instinct: the rule didn’t fire. Easy theory, easy to check. So I pulled the alert history.

The rule fired. On time. With the right severity, the right labels, the right everything. It did its job flawlessly and then handed the alert off to… something.

┌──────────┐    ┌────────────┐    ┌──────────┐    ┌────────┐
│  RULE    │──► │  RECEIVER  │──► │ CHANNEL  │──► │ HUMAN  │
│ ✓ fired  │    │ ✓ matched  │    │  ✗ none  │    │  ✗ —   │
└──────────┘    └────────────┘    └──────────┘    └────────┘
                                   ▲
                                   │
                          the chain dies right here

The rule routed to a receiver. The receiver had a name, a config block, a place in the tree. It looked alive. And its list of channels — the email addresses, the webhook URLs, the push targets, whatever was supposed to carry the message to a person — was empty.

Not broken. Not misconfigured. Empty. The alert was delivered, perfectly, to nobody.

The “aha”

Here’s the part that stung. Everyone — me included — had validated the wrong layer.

We tested that rules existed. We tested that rules fired. We never tested that a fired alert reached a phone, an inbox, a pager, a chat channel — a person. The chain has four links; we’d verified two and assumed the other two by vibes.

An alert is not a notification. A rule firing is not someone finding out. The gap between those two facts is exactly where this outage lived — silent, for months — manufacturing a warm, false confidence that the system “had alerting.”

It did. The alerting just talked to a wall.

The fix

Audit every receiver. Every single one needs at least one channel, and that channel has to actually reach a human or a device. No empties, no dead webhooks, no stale email lists.

Then — and this is the part people skip — add a heartbeat. A “dead-man” alert that is supposed to fire on a schedule. If it stops arriving, your delivery path is broken and you find out on purpose instead of during a customer call.

# 1. Find every receiver with an empty / missing channel list.
amtool config routes show
amtool config show | yq '.receivers[] | select(
  ([.email_configs, .webhook_configs, .pushover_configs,
    .slack_configs, .opsgenie_configs] | flatten | length) == 0
) | .name'

# 2. End-to-end test: fire a synthetic alert and confirm it lands on a device.
amtool alert add deadmans_switch \
  severity=info service=alerting \
  --annotation summary="heartbeat — if you can read this, delivery works"

# 3. The dead-man heartbeat: a rule that is ALWAYS firing on purpose.
groups:
  - name: heartbeat
    rules:
      - alert: DeadMansSwitch
        expr: vector(1)
        labels: { severity: heartbeat }
        annotations:
          summary: "Alerting pipeline is alive. Silence = it isn't."

DELIVERY CHAIN ── post-fix
  RULE ─► RECEIVER ─► CHANNEL ─► DEVICE
   ✓        ✓          ✓ (≥1)     ✓ buzzed
        + DeadMansSwitch heartbeat every 60s ►► if it goes quiet, you know

Flip the logic. A normal alert proves a problem exists. A heartbeat proves the path itself still works. You want both, because the path is the thing that fails silently.

Why it happened

Someone — possibly past me — created the receiver as a placeholder, fully intending to fill in the channels later. Later never came. The config was syntactically valid, so nothing complained. There’s no error on an empty channel list; it’s a legal, boring state.

And because the rules were visibly there, every review after that pattern-matched “alerting: configured ✓” and moved on. Nobody walked the whole chain to a buzzing phone. The placeholder calcified into production.

Takeaways

• An alert with no destination is worse than no alert. No alerting at least keeps you honest. A dead receiver hands you false confidence and bills you for it during an outage.
• Validate the entire chain: rule → receiver → channel → device. Verifying any single link tells you almost nothing about the link after it.
• Test delivery, not configuration. “The rule exists” and “a human got paged” are different claims. Only the second one matters at 2 a.m.
• Add a dead-man heartbeat. A signal that should arrive on schedule turns silent failure into a loud, on-purpose one.
• Audit receivers for empties on a cadence. Placeholders, decommissioned inboxes, and rotted webhooks accumulate. Grep them out before they grep you.

The cron job ran at 11 PM. By the time I poured coffee the next morning, it had called a paid LLM API a few thousand times.

Nobody touched it. Nobody approved it. It just sat there in the dark, retrying, retrying, retrying — like a vending machine eating a stuck dollar bill, except the dollar bill was billable tokens.

I didn’t find it because of a clever alert. I found it because the provider dashboard had a graph that looked like a cliff face.

The scene

I run a small fleet of unattended automations for a client. One of them enriches records overnight by sending them through a metered LLM endpoint. Cheap per call. Boring. The kind of job you set up once and forget — which is exactly the problem.

That night, the upstream service it depended on got flaky. A handful of calls returned errors. The loop did what badly-written loops always do.

It tried again. Immediately. Forever.

┌────────────────────────────────────────────────────┐
│  THE LOOP (as written)                             │
│                                                    │
│   ┌──────────┐   error   ┌──────────────┐          │
│   │ call API │ ────────► │ retry now    │          │
│   └────┬─────┘           └──────┬───────┘          │
│        ▲                        │                  │
│        └────────────────────────┘                  │
│        no backoff · no cap · no kill switch        │
│                                                    │
│   result: ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓  thousands of hits │
└────────────────────────────────────────────────────┘

No sleep. No max-retry counter. No circuit breaker. Every failure became an instant request, and every instant request became another line on the invoice.

The investigation

First thing: confirm it’s us and not a stolen key. I pulled the provider’s usage view and bucketed by hour.

# usage exploded between 23:00 and 07:00 — exactly the cron window
# logs from the job host told the same story
journalctl -u record-enrich.service --since "yesterday 23:00" \
  | grep -c "POST /v1/"
# -> 4,300-something. Overnight. For a job that should make ~80 calls.

Same source IP. Same user-agent. Same service. Not a breach — a self-inflicted wound. Somehow that’s worse, because it means the call was coming from inside the house.

The “aha”

The smoking gun was four lines of code. Paraphrased:

while not done:
    try:
        resp = client.complete(payload)
        done = True
    except Exception:
        continue   # <-- the whole disaster, right here

continue. No delay. No ceiling. The instant the upstream hiccuped, this turned into a tight spin loop firing paid requests as fast as the network would carry them. A retry without backoff isn’t resilience — it’s a denial-of-wallet attack you launch against yourself.

The fix

I treated the key as compromised even though it wasn’t, because the behavior was indistinguishable from a leak. Containment first, blame later.

# 1. Rotate the key immediately — old one dies on the spot
provider keys rotate --name record-enrich --revoke-old

# 2. Disable the API entirely at the provider while I fix the code.
#    Kill the bleeding before patching the artery.
provider api disable --service record-enrich

# 3. Re-enable WITH a hard cap + budget alert. The cap is the real fix.
provider budget set --service record-enrich \
  --hard-limit-usd 25 --period monthly
provider alerts set --service record-enrich \
  --notify-at 50% --notify-at 90% --channel telegram

Then the code got the guardrails it should have shipped with:

import time

MAX_RETRIES = 5

for attempt in range(MAX_RETRIES):
    try:
        resp = client.complete(payload)
        break
    except TransientError:
        time.sleep(min(2 ** attempt, 30))   # exponential backoff, capped
else:
    raise RuntimeError("gave up after retries — failing loud, not looping")

Backoff. A retry ceiling. And a loud failure instead of a silent infinite spin. The provider cap is the seatbelt; this is actually steering the car.

Why it happened

It happened because “it’s just a small overnight job” is the exact mindset that ships uncapped loops at paid endpoints. The cost-per-call was trivial, so nobody did the multiplication. Trivial times infinity is still a number you have to pay.

The code had no concept of “too many.” The provider had no concept of “enough.” With neither a ceiling in the app nor a ceiling at the wallet, the only limiter left was how fast the network could move — and that’s a throttle on the damage, not a budget.

Takeaways

• Put a hard spend cap at the provider. It’s the only limit a runaway loop can’t out-code. Set it before you write the first request.
• Backoff and a max-retry count are not optional. A retry without delay or ceiling is a self-DoS. continue on an exception is a loaded gun.
• Wire usage alerts at 50% and 90%. You want a phone buzz at midnight, not a graph shaped like a cliff at breakfast.
• Treat weird usage as a leak until proven otherwise. Rotate the key and disable the endpoint first; debug the code second. Containment beats curiosity.
• “Small overnight job” is a smell. Anything unattended that touches a metered API gets a kill switch, a cap, and an alert — or it doesn’t get deployed.

The pager went off at 6:14 a.m. The morning sync hadn’t run.

I knew that script. I wrote that script. It was good. It pulled data, massaged it, pushed it where it needed to go. It worked flawlessly every single time I ran it.

That was the problem. It only worked when I ran it.

The scene

Here’s the embarrassing truth about that automation: it lived inside an SSH session. I’d connect to the box at sync-host.internal.example, fire it off, watch it succeed, and disconnect feeling like a wizard.

The moment my terminal closed, the kernel reaped it. SIGHUP, lights out. No process, no cron-shaped safety net, nothing.

On the Windows box it was worse. I’d “scheduled” it — except every run popped a console window that stole focus from whoever was using the machine. People started killing the window because it interrupted them. Can’t blame them.

So three different failure modes, one script:

   ┌──────────────────────────────────────────────┐
   │  "It works when I run it manually"            │
   ├──────────────────────────────────────────────┤
   │  close SSH session   ──►  SIGHUP, process dies│
   │  server reboots      ──►  nothing comes back  │
   │  scheduled on win    ──►  window steals focus │
   └──────────────────────────────────────────────┘
                    │
                    ▼
            NOT actually deployed

The investigation

I tailed the journal. Nothing — because nothing was logging anywhere. The script wrote to stdout, and stdout was attached to a terminal that no longer existed.

I checked the process list. Empty. I checked for a cron entry. None — I’d never made one, because “I’ll just run it” had quietly become the deployment strategy.

That’s the aha, and it’s a dumb one: manual execution was masquerading as deployment. The script was perfect and completely undeployed at the same time. Every successful manual run was me hand-holding a service that had no manager.

A long-running, scheduled, headless job has three needs I’d ignored: survive a logout, survive a reboot, and write its logs somewhere a human can read them later. A terminal gives you none of those.

The fix

Give it a service manager. On Linux, a systemd unit. On Windows, NSSM. Both get auto-restart, log redirection, and start-on-boot — and both run headless.

Linux — /etc/systemd/system/morning-sync.service:

[Unit]
Description=Morning data sync
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=svc-sync
WorkingDirectory=/opt/morning-sync
ExecStart=/opt/morning-sync/.venv/bin/python /opt/morning-sync/sync.py
Restart=on-failure
RestartSec=10
StandardOutput=append:/var/log/morning-sync/out.log
StandardError=append:/var/log/morning-sync/err.log

[Install]
WantedBy=multi-user.target

sudo mkdir -p /var/log/morning-sync
sudo systemctl daemon-reload
sudo systemctl enable --now morning-sync.service
systemctl status morning-sync.service
journalctl -u morning-sync.service -f

Windows — same job, NSSM does the babysitting and runs it with no visible window:

nssm install MorningSync "C:\apps\morning-sync\.venv\Scripts\python.exe" "C:\apps\morning-sync\sync.py"
nssm set MorningSync AppDirectory "C:\apps\morning-sync"
nssm set MorningSync AppStdout "C:\logs\morning-sync\out.log"
nssm set MorningSync AppStderr "C:\logs\morning-sync\err.log"
nssm set MorningSync AppExit Default Restart
nssm set MorningSync Start SERVICE_AUTO_START
nssm start MorningSync

Now I close my laptop and it keeps running. The box reboots at 3 a.m. for patches and the sync comes back on its own. The logs sit in a file instead of evaporating with my SSH session.

Why it happened

Because the script worked. That’s it. When something works on the first manual run, the brain files it under “done” and quietly skips the boring part — the part where a supervisor owns the process, restarts it when it dies, and persists across boots.

A terminal is not a runtime. Cron-with-a-popup is not headless. “Done” is not “deployed.”

Takeaways

• “It works when I run it manually” is not deployed. A successful manual run proves the logic, not the operation.
• Long-running automation belongs in a service manager — systemd on Linux, NSSM on Windows — not a terminal you forgot to close.
• Always set auto-restart and boot persistence. Restart=on-failure plus enable --now (or SERVICE_AUTO_START) means reboots and crashes self-heal.
• Redirect stdout/stderr to a file. Logs that live inside an SSH session die with the session, and you’ll be debugging blind at 6 a.m.
• Run it headless. Background jobs that pop windows get killed by annoyed users. No UI, no focus theft, no problem.

It was supposed to be a five-minute job.

A client’s backup server needed one field bumped on a handful of backup clients. Retention. One value. I had a Python wrapper sitting right there with a method named exactly what I wanted: change_client_setting(). Tab-complete practically begged me to use it.

So I looped over the clients and called it. Twice, because I tweaked a second field on the second pass.

Then the dashboard started showing clients with no backup schedule, no retention, no paths — like they’d never been configured. Not one client. All of them.

The investigation

First instinct: did I fat-finger the loop? No. The script was boring. Read a name, set a field, move on.

Second instinct: did the service choke? Logs were clean. The server was happily applying the garbage I’d handed it.

So I pulled the raw settings blob the API was actually POSTing. And there it was — the settings object, wrapped inside another settings object, wrapped inside another one. A JSON turducken.

┌─────────────────────────────────────────────┐
│ POST #1   { retention: 30, schedule: ... }   │  ✓ looks fine
├─────────────────────────────────────────────┤
│ POST #2   { settings: {                      │
│              settings: {                      │
│                retention: 30, schedule: ...   │  ✗ nested once
│              } } }                            │
├─────────────────────────────────────────────┤
│ POST #3   { settings: { settings: {          │
│              settings: { ...buried... } } } } │  ✗ nested twice → unreadable
└─────────────────────────────────────────────┘
                       ▼
        server reads top level → finds nothing it recognizes
        every real field is now three layers down → effectively gone

The “aha”

Here’s the part that made my stomach drop.

change_client_setting() does not set one field. It round-trips the entire settings object: GET the whole blob, mutate the one key in memory, then re-serialize and POST the whole thing back.

That’s already more blast radius than the name implies. But the killer was the encoder. Its re-serialization had a recursive-nesting bug — each round-trip tacked the existing object inside a fresh settings wrapper instead of replacing it.

One call: subtly wrong but survivable. Two calls: the real fields are buried deep enough that the server can’t find them anymore. The defaults win. Every client looks wiped.

I didn’t corrupt one setting. I corrupted the container every setting lives in — for every client — by changing a single number.

The fix

Stop trusting the convenience method. Write a minimal-POST helper that submits only the field that changed, never the whole object — so there’s no full blob to re-encode and no encoder to trip over.

def safe_set_client_setting(server, client_id, key, value, dry_run=False):
    # Snapshot the current settings BEFORE touching anything.
    current = server.get_client_settings(client_id)
    snapshot_path = f"snapshots/{client_id}-{int(time.time())}.json"
    with open(snapshot_path, "w") as f:
        json.dump(current, f, indent=2)

    if dry_run:
        print(f"[dry-run] {client_id}: {key} {current.get(key)!r} -> {value!r}")
        return

    # Minimal POST: ONLY the one field. No round-trip of the whole blob.
    server.post_setting(client_id, {key: value})

    # Read-back validation. Trust nothing.
    after = server.get_client_settings(client_id)
    if str(after.get(key)) != str(value):
        raise RuntimeError(
            f"VALIDATION FAILED {client_id}: {key} is {after.get(key)!r}, "
            f"expected {value!r} — restore from {snapshot_path}")
    print(f"✓ {client_id}: {key} = {value}  (snapshot {snapshot_path})")

Recovery itself was unglamorous — restore each client from the JSON snapshot I now wish I’d had before the first run:

for f in snapshots/*-pre-incident.json; do
  id=$(basename "$f" | cut -d- -f1)
  python restore_settings.py --client "$id" --from "$f"
done

Four guardrails came out of this, and they’re now non-negotiable on that box:

Why it happened

The method name lied — or rather, I read a name and assumed a behavior. “Change one client setting” sounded surgical. Under the hood it was a full read-modify-write of a complex object, handed to a serializer nobody had stress-tested across repeated calls.

A buggy encoder on a single-field write is annoying. A buggy encoder on a whole-object round-trip is a data-integrity bomb, because every write re-touches every field. The bug didn’t need to corrupt the value I changed. It corrupted the structure wrapping everything.

And I had no snapshot, no read-back, and no dry-run — so the first time I learned any of this was when the dashboard went blank.

Takeaways

• Read what the API actually sends, not what the method is named. “Set one field” frequently means “GET, mutate, re-serialize, POST everything.” That round-trip is your real blast radius.
• Prefer minimal, targeted writes. If you can POST just the changed key, do it — you never hand a fragile encoder the whole object to mangle.
• Validate by reading back. A write that “succeeded” tells you the server accepted bytes, not that the right value landed. Read it again and compare.
• Snapshot before you mutate. A pre-write JSON dump turns a catastrophe into a one-line restore. It costs milliseconds and buys you the whole night back.
• Ship a --dry-run for anything that touches production config. Seeing the diff before it’s real would have caught the nesting on call number one.