It started as a slow-afternoon habit: scan yourself the way an attacker would. Pull up Shodan, type in your own public IP, and brace.

I expected a clean sheet. UFW was configured. ufw deny on everything that wasn’t 80, 443, and the VPN. I’d checked it a dozen times. Green across the board.

Shodan disagreed.

There it was, indexed and timestamped: an admin/management panel answering on a port that was supposed to be firewalled into oblivion. Banner, title, the works. Anyone with a browser could’ve found it. Some of them probably had.

The investigation

First reaction: Shodan’s cache is stale. It happens. So I scanned from the outside, from a box that had no business reaching anything internal.

nmap -Pn -p 9000 192.0.2.10 --open   # from a cloud box, not my network
# (really: nmap -Pn -p 9000 <my-public-ip>)
nmap -Pn -p 9000 192.0.2.10

PORT     STATE  SERVICE
9000/tcp open   http

Open. From the public internet. Not stale.

So I SSH’d in and asked UFW what it thought it was doing.

sudo ufw status verbose | grep 9000
# (nothing — port 9000 is not allowed)
sudo ufw status | head
# Status: active
# 22/tcp   ALLOW   Anywhere
# 443/tcp  ALLOW   Anywhere

UFW was adamant: 9000 is closed. The internet was equally adamant: 9000 is open. One of them was lying, and firewalls don’t lie. They just get bypassed.

The “aha”

Then I remembered what was actually listening on 9000.

A Docker container.

docker ps --format '{{.Names}}\t{{.Ports}}'
# admin-panel   0.0.0.0:9000->9000/tcp

0.0.0.0:9000->9000/tcp. There it is. That -p 9000:9000 in the compose file. Docker doesn’t ask UFW for permission — it writes its own iptables rules, straight into the DOCKER chain, and they get evaluated before your tidy UFW INPUT rules ever run.

Your ufw deny is a polite note on the front door. Docker built a second door around back and propped it open.

  Inbound packet → :9000
        │
        ▼
  ┌───────────────────────────┐
  │ iptables nat/PREROUTING   │
  │  DOCKER chain (DNAT)  ◄────┼── Docker put this here.
  │  → 172.17.0.2:9000        │     UFW never gets a vote.
  └───────────────────────────┘
        │            ╲
        ▼             ╲ (never reached)
   container :9000     ┌────────────────────┐
                       │ ufw-user chain     │
                       │  DENY 9000  (moot) │
                       └────────────────────┘

The fix

The publish target was the bug. 0.0.0.0 means “the whole world.” It almost never should.

I bound the panel to loopback only and put it behind the reverse proxy with auth, reachable just through the VPN.

# docker-compose.yml — before
    ports:
      - "9000:9000"        # = 0.0.0.0:9000, published to the internet

# after
    ports:
      - "127.0.0.1:9000:9000"   # localhost only; proxy/VPN reaches it

docker compose up -d
docker ps --format '{{.Names}}\t{{.Ports}}'
# admin-panel   127.0.0.1:9000->9000/tcp

For anything that genuinely must survive the firewall, lock it at the DOCKER-USER chain — the one place Docker promises not to clobber:

# allow only the tailnet, drop everyone else for published container ports
sudo iptables -I DOCKER-USER -i eth0 -s 100.64.0.0/10 -j RETURN
sudo iptables -I DOCKER-USER -i eth0 -p tcp --dport 9000 -j DROP

Re-scan from outside:

nmap -Pn -p 9000 192.0.2.10
# PORT     STATE    SERVICE
# 9000/tcp filtered http

filtered. Dark. The panel only answers behind the tunnel now.

Why it happened

Nobody screwed up the firewall. The firewall did exactly what it was told. The trap is the mental model: people assume UFW sits at the front gate and inspects everything inbound. Docker doesn’t go through that gate — it cuts its own keyhole in iptables and your host rules never see the packet.

-p 9000:9000 reads like “expose this locally.” It actually means “publish this to 0.0.0.0.” One missing 127.0.0.1: prefix and an internal admin tool was on the public internet for who-knows-how-long, indexed by Shodan, waiting.

Takeaways

• Scan yourself the way an attacker would. Shodan plus an external nmap from a box outside your network is the only ground truth. Your config file is a hypothesis, not a result.
• Docker port publishing ignores your host firewall. -p PORT:PORT writes its own iptables rules and bypasses UFW entirely. ufw deny does nothing for published container ports.
• Bind admin services to 127.0.0.1 or a VPN, never 0.0.0.0. Always prefix the publish: 127.0.0.1:9000:9000. Default-public is a footgun.
• Use Docker-aware firewalling. If a container port truly needs outside reach, gate it in the DOCKER-USER chain — the one chain Docker won’t overwrite.
• Admin panels belong on localhost or a tunnel, behind a reverse proxy and auth. Convenient-but-public is just public.

The ticket said “internet is down.” Then, four minutes later, “never mind, it’s back.” Then, twenty minutes after that, “it’s down again.”

That pattern is a special kind of hell. A clean outage you can chase. A flapping one that heals itself before you SSH in just laughs at you.

I got a client on the phone. Their whole site would blackhole — every desktop, every phone, the printer, all of it — for one to five minutes, then snap back like nothing happened. No schedule. No trigger anyone could name. The ISP swore the circuit was clean, and for once the ISP was right.

The investigation

I started where you start: at the edge. The gateway was up. WAN was up. The firewall logs were boring in the way you want them to be boring. No flapping interface, no CPU spike, no DHCP storm in the leases.

So I sat on a workstation and just hammered the gateway while I waited for the next outage.

ping -i 0.5 192.0.2.1

It ran fine for ten minutes. Then — dead air. Eight, nine, ten dropped replies. Total blackhole. Then it came back on its own.

The instant it died, I dumped the ARP table.

arp -a 192.0.2.1

And there it was. The gateway’s IP was mapped to a MAC address I did not recognize. Not the firewall’s MAC. Some OUI I had to look up by hand.

I watched it flap in real time:

┌─────────────────────────────────────────────────────────────┐
│  arp -a 192.0.2.1   (watched over ~30s)                      │
├─────────────────────────────────────────────────────────────┤
│  t+0s   192.0.2.1  ►  aa:bb:cc:11:22:33   (firewall)   ✓     │
│  t+6s   192.0.2.1  ►  de:ad:be:ef:00:99   (???)        ✗     │
│  t+9s   192.0.2.1  ►  de:ad:be:ef:00:99   (???)        ✗     │
│  t+14s  192.0.2.1  ►  aa:bb:cc:11:22:33   (firewall)   ✓     │
│  t+19s  192.0.2.1  ►  de:ad:be:ef:00:99   (???)        ✗     │
└─────────────────────────────────────────────────────────────┘
         ▼ when the IP pointed at de:ad:..:99, traffic vanished

When the gateway IP pointed at the firewall, the network worked. When it pointed at that mystery MAC, every packet bound for the internet got handed to a device that did absolutely nothing with it. Classic ARP poisoning. The only question was who.

The “aha”

I looked up the OUI. The mystery MAC belonged to a Tuya-based Wi-Fi module — the kind that ships inside cheap smart-home gadgets by the millions.

Then I cross-referenced the DHCP leases for that MAC and found a hostname that made me laugh out loud: a smart window blind controller. Somebody had put a $30 motorized blind on the corporate flat network.

That little NIC was broadcasting unsolicited ARP replies claiming to be 192.0.2.1. Every client on the segment believed it, updated its ARP cache, and started shipping its default route straight into a window covering.

The fix

First, stop the bleeding. I pinned the gateway’s real MAC on the affected machines so they’d stop trusting the lie while I worked.

# Linux
sudo ip neigh replace 192.0.2.1 lladdr aa:bb:cc:11:22:33 dev eth0 nud permanent

Then the real fix — get IoT off the flat network entirely and turn on the switch protections that should’ve been on from day one.

BEFORE                              AFTER
──────                              ─────
VLAN 1 (everything)                 VLAN 10  desktops/servers
  ├─ desktops                       VLAN 20  voice
  ├─ servers                        VLAN 90  IoT  ◄── blind lives here,
  ├─ phones                                          firewalled, no L2
  └─ smart blind  ◄── poisoner                       reach to clients

# Cisco-style switch hardening (concept)
# Trust only the uplink for DHCP; drop rogue offers elsewhere
ip dhcp snooping
ip dhcp snooping vlan 90
interface Gi1/0/24
 ip dhcp snooping trust

# Dynamic ARP Inspection — validate ARP against snooping bindings
ip arp inspection vlan 90
interface Gi1/0/24
 ip arp inspection trust

Last step: I yanked the blind, pushed its vendor firmware update, and parked it on the isolated IoT VLAN where it can lie about being the gateway all it wants — and nobody who matters will hear it.

Why it happened

Cheap Tuya firmware is a grab bag. Some builds have buggy network stacks that emit malformed or unsolicited ARP — gratuitous ARP gone feral, sometimes triggered by a Wi-Fi reconnect, which explains the random timing. Could also be straight-up malicious; with no-name IoT you genuinely can’t tell, and it doesn’t change the remediation.

Either way, a flat L2 network trusts every device equally. ARP has no authentication. Whatever shouts loudest and last, wins. Put one mouthy gadget on that wire and it can pretend to be anything — including the door to the internet.

Takeaways

• On weird, intermittent, self-healing outages, check the gateway’s ARP entry FIRST. arp -a <gateway-ip> flapping to an unknown MAC is a five-second smoking gun.
• Match a rogue MAC to its OUI and your DHCP leases. That’s how a “mystery device” became “the smart blind in the break room.”
• IoT does not belong on your flat network. Ever. Its own VLAN/SSID, firewalled off from clients, no L2 reach to anything that matters.
• Turn on DHCP Snooping and Dynamic ARP Inspection. They exist precisely to kill rogue gateways and ARP lies at the switch.
• Cheap IoT is an untrusted host by default — bug or malice, you treat it the same: isolate, update, and never let it speak for the gateway.

It was supposed to be a quick audit.

A client had asked me to sweep a private repo before they handed access to a new contractor. Routine. I cloned it, ran a scanner over the working tree, and started reading. Five minutes in, the scanner lit up red on a config helper.

A reusable credential. Plaintext. In the repo.

I felt that familiar little spike of righteous sysadmin adrenaline — who the hell committed THIS — opened the blame, and read the author line.

It was me.

The investigation

Here’s the part everyone gets wrong, and the part I almost got wrong too.

My first instinct was to delete the line, commit “remove secret,” and move on. Crisis averted, right?

Wrong. The file in the working tree is the least important place that secret lives. Git is a time machine. Deleting it today does nothing about the dozens of commits where it sits, fat and happy, in the history. Anyone with git log and ten seconds can walk straight to it.

I checked how deep it went:

git log --all --oneline -S 'the_secret_string' -- path/to/config
git rev-list --all --count

It went deep. The credential had been in the repo for months, across dozens of commits, surviving file renames and a refactor. Deleting the file would leave it perfectly intact in every parent commit.

  the working tree (what you see)
        │
        ▼
   ┌─────────┐   ┌─────────┐   ┌─────────┐   ┌─────────┐
   │ commit  │◄──│ commit  │◄──│ commit  │◄──│ commit  │
   │  HEAD   │   │  ~12    │   │  ~31    │   │  ~40    │
   │  ✗ gone │   │ ▓ secret│   │ ▓ secret│   │ ▓ secret│
   └─────────┘   └─────────┘   └─────────┘   └─────────┘
        ▲              └──────────────┬──────────────┘
   you "fixed" it          still right here, forever

The “aha”

The realization isn’t “I need to scrub this.” It’s earlier and uglier than that:

A secret is compromised the instant it’s committed. Not when the repo goes public. Not when someone clones it. At commit time. Every backup, every fork, every laptop that ever pulled, every CI cache — assume the secret is out. You cannot un-ring that bell by editing history.

So scrubbing history is necessary, but it is not step one. Step one is to make the leaked value worthless.

The fix

1. Rotate, immediately. Before anything else, I logged into the provider, revoked the credential, and issued a fresh one. The old value is now a dead string. Whoever has it has nothing.

2. Purge it from history. With the value already dead, I scrubbed it so it stops scaring the next auditor (me, in six months). git filter-repo is the modern tool; BFG works too.

# install once: pip install git-filter-repo
echo 'the_secret_string==>REDACTED' > replacements.txt
git filter-repo --replace-text replacements.txt

# then force the rewritten history out
git push --force --all
git push --force --tags

Every collaborator re-clones after this. A force-push that rewrites history will wreck anyone’s local copy — warn them first.

3. Move secrets out of the repo for good. The config now reads from the environment, injected at runtime from a secrets manager. Nothing sensitive touches the tree.

# .env stays OUT of git
echo '.env' >> .gitignore

# app reads from the environment, value lives in the secrets manager
export API_TOKEN="$(vault kv get -field=token secret/app)"

4. Make it impossible to repeat. A pre-commit hook scans every staged change so a secret never reaches a commit in the first place.

# .pre-commit-config.yaml
#  - repo: https://github.com/gitleaks/gitleaks
#    rev: v8.18.0
#    hooks: [{ id: gitleaks }]
pre-commit install
gitleaks detect --source . --verbose

Why it happened

No villain here. Just the most ordinary mistake in the trade.

Months ago, mid-deploy, I needed the thing to work now. I hardcoded the credential “just to test,” told myself I’d pull it out before committing, got interrupted, and committed everything with a git add .. The TODO never came back. The repo was private, so it felt safe — and “private” is exactly the lie that lets these things rot in history for half a year.

Private isn’t a control. It’s a setting someone can flip.

Takeaways

• A secret is burned the moment it’s committed. Rotate first, always. Treat the value as public from commit time forward — because effectively it is.
• Deleting the file is not remediation. The credential lives in every old commit. You must rewrite history (git filter-repo / BFG) and force-push, or it’s still there.
• Secrets belong in a manager, not the tree. Env-injected values, .gitignore your .env, and never let credentials and code share a home.
• Automate the catch. A gitleaks/trufflehog pre-commit hook stops the next leak before it’s a commit — humans forget, hooks don’t.
• “Private repo” is not a security boundary. Audit your own repos like you’d audit a stranger’s. The author line might surprise you.

It started with a boring task. Quarterly DNS audit. Coffee, a zone file, and a vague sense that everything was fine.

It was not fine.

Halfway down the export I hit a CNAME — a customer-facing subdomain — pointing at a cloud host we’d decommissioned back in the winter. The instance was gone. The DNS record had never gotten the memo.

I sat there for a second. That subdomain still resolved. It still answered. It just answered nothing — a hosting account that no longer existed, on infrastructure we’d long since stopped paying for.

That’s not a typo. That’s a loaded gun pointed at your own brand.

The investigation

A dangling DNS record is a pointer with no backing store. The record says “this name lives over there.” But “over there” got freed months ago — the VM was destroyed, the cloud bucket released, the hosting account closed.

The danger isn’t that it’s broken. The danger is that the resource is now available for anyone to claim.

I pulled the full zone and started checking targets one by one. Does the A record’s IP still belong to us? Is the CNAME target a hostname we still control? Or is it a now-orphaned cloud resource sitting in a provider’s free pool, waiting for the next person to grab it?

        Your DNS                      The Internet
   ┌──────────────────┐         ┌────────────────────────┐
   │ app.example.com  │         │  cloud host (DELETED)   │
   │  CNAME ──────────┼────────▶│  account closed, freed  │
   └──────────────────┘         │  ▒▒▒ up for grabs ▒▒▒  │
                                └───────────┬────────────┘
                                            │  attacker
                                            ▼  re-claims it
                                ┌────────────────────────┐
                                │  serves THEIR content   │
                                │  on YOUR subdomain      │
                                └────────────────────────┘

The “aha”

Here’s the part that turns a stale record into a breach.

If an attacker registers that freed cloud resource — same provider, same dangling target — the provider hands them the keys. Now your subdomain serves their content. And because it’s your subdomain, the blast radius is ugly:

• Pixel-perfect phishing on a domain your users already trust.
• Cookies scoped to *.example.com get harvested — including session cookies from your real apps.
• A free, fully valid TLS certificate, because the takeover host can pass the domain-control check for a name you literally pointed at it.

No CVE. No exploit chain. Just a DNS record that outlived its server.

The fix

First, inventory everything. Pull the whole zone, not just the records you remember.

# Dump every A / AAAA / CNAME in the zone
dig @ns1.example.com example.com AXFR \
  | awk '$4 ~ /^(A|AAAA|CNAME)$/ {print $1, $4, $5}'

Then resolve each one and check whether the target still answers — and whether it’s still yours.

while read -r name type target; do
  echo "=== $name ($type) -> $target"
  dig +short "$name"
  curl -sS -o /dev/null -w "  http=%{http_code} ip=%{remote_ip}\n" \
    --max-time 8 "https://$name" || echo "  DEAD / no response"
done < zone-records.txt

For the dangling ones, the rule is simple: if you don’t control the target, the record dies or gets repointed.

# Delete the stale record (RFC 2136 dynamic update)
nsupdate -k /etc/dns/update.key <<'EOF'
server ns1.example.com
zone example.com
update delete app.example.com. CNAME
send
EOF

Repoint anything still in use at infrastructure you own — a host behind your own reverse proxy, not a third-party resource you might let lapse again.

Finally, make it a standing job. A nightly check that flags any record whose target stops resolving or stops belonging to you.

# cron: nightly dangling-DNS sweep, alert on dead targets
0 4 * * *  /opt/dns/dangle-check.sh && \
  curl -fsS https://hc-ping.test/dns-sweep

Why it happened

DNS outlives servers. Always.

When you spin a box down, you delete the box. You raise the ticket, you confirm it’s gone, you move on. The DNS record — living in a different system, owned by a different runbook — just sits there pointing at a ghost. Nobody’s job ends with “and remove the CNAME.”

Multiply that by a few years of “temporary” subdomains and deprovisioned cloud accounts, and you’ve got a quiet pile of doors with no locks.

Takeaways

• Inventory every record. You can’t audit a zone you’ve only half-remembered. Pull the whole thing.
• Verify the target, not just the name. A record that resolves isn’t safe — confirm the thing it points at is still alive and still yours.
• Prune the moment a server dies. Decommissioning a host isn’t done until its DNS is gone too. Bake it into the runbook.
• Prefer targets you control. Point subdomains at your own infrastructure, not third-party resources you might forget to renew.
• Monitor continuously. Dangling DNS is created by routine. Catch it with routine — a nightly sweep beats a quarterly surprise.

🔐 Have You Enabled 2FA Everywhere? Changed Your Password This Decade?

Let’s start with a gut check.

Do you reuse the same password across multiple sites?

Is your email password the same one you used in 2013?

Have you enabled two-factor authentication (2FA) on your most important accounts?

Or… are you still getting text codes (yikes) instead of using an authenticator app?

If any of the above made you uncomfortable — good. That’s the point.

🧠 Why This Still Matters

Every week we clean up the mess after someone gets locked out of an account, or worse, loses access due to a breach. And 99% of the time? It could’ve been prevented with a simple change:

✅ Stronger, unique passwords

✅ Two-Factor Authentication (2FA)

Still think you’re safe because “nobody would hack me”? Newsflash: they don’t target you. They target everyone, and your password is probably already floating around out there. Just check https://haveibeenpwned.com if you don’t believe me.

🛡️ Do This Now:

Enable 2FA (Everywhere)

Start with:

1. Email

2. Bank

3. cloud storage

4. Social media

5. Any sites with your personal data or billing info

Use an authenticator app (like Authy, Microsoft Authenticator, or Google Authenticator). Not SMS. Text-based 2FA can be hijacked.

2. Use a Password Manager

Bitwarden, 1Password, or even the built-in tools from Apple or Google are far better than your sticky note or Excel sheet.

3. Change Old Passwords

If your password is:

Over 5 years old

Shared across more than one site

Or contains your pet’s name and birth year…

Change. It. Now.

Use 12+ characters, mix it up, and stop using “!” at the end to feel secure. Hackers are wise to that game.

⚙️ Bonus Tip: Turn on Login Alerts

Most sites let you enable notifications when someone logs in from a new device. Do that. It’s like a smoke alarm for your accounts.

Final tip: listen to your IT people, ensure you have working backups, and someone monitoring the security of your devices, servers, and other online assets.

And update your end of life networking devices and computers!

🔚 Final Thoughts

Cybersecurity isn’t just for IT nerds. It’s for everyone. The smallest effort (like enabling 2FA) goes a long way in preventing a total digital meltdown.

And if it’s too much? That’s what we’re here for.

!Need help securing your accounts or deploying 2FA for your business? Contact us mypueblopc.com