🏠 Homelab Known Fixes

Symptoms

• ntfy fires 'DDNS: DNS record(s) corrected' (priority 3) for *.compellinglylowbrow.org every 5-minute cron cycle for hours, always the exact same direction (stale IP -> current IP), never converging
• This is the priority-3 'corrected' path, NOT the priority-5 duplicate-detected alert -- so the 2026-07-21 duplicate guard is NOT tripping (DUP_COUNT == 1: a single wildcard record, not two)
• bin/ddns-update --dry-run reports 'would update *.compellinglylowbrow.org: <stale> -> <current>' with no duplicate warning, run after run
• dig against all four authoritative Porkbun nameservers returns ONE consistent stale IP (not two round-robining) -- the public record is genuinely stuck, not duplicated
• The wildcard's TTL is stuck at an old value (e.g. 600) while headscale/headplane -- edited by the same script, same run -- show the TTL ddns-update sets (300); the wildcard's editByNameType/%2A calls were never landing

Symptom

Root cause

Diagnostic path that worked

Remediation

Lessons worth keeping

Symptom

level=error err="opening storage failed: /var/snap/prometheus/common/chunks_head/000465: invalid magic number 0"
...
snap.prometheus.prometheus.service: Start request repeated too quickly.

Cause

Investigation before fixing

Fix

sudo snap remove prometheus

Prevention / notes for next time

Symptom

Details

Status

Symptoms

• Generated collected/runbooks/YYYYMMDD-group1.sh fails 'bash -n' with 'syntax error near unexpected token' pointing at an echo or _log line
• A pre_update/post_update/verify desc: field containing a contraction ("it's", "gen-runbook's", "won't") breaks the runbook it's embedded in

Symptom

Root Cause

Fix

# before
lines.append(f"echo '  post: {desc}'")
after
lines.append(f"echo {sh_single_quote('  post: ' + desc)}")

Verification

# 1. Syntax check against real hosts-config.yaml data (not synthetic) — this is
   what actually caught the bug; a scratch-dir test with clean desc text will
   NOT catch this class of bug.
python3 bin/gen-runbook   # with a real pending -summary.txt in place
bash -n collected/runbooks/<latest>-group1.sh

2. Adversarial regression test — desc text WITH an apostrophe in every step
   type (pre_update, post_update, verify), confirmed bash -n clean:
python3 -c "
from importlib.machinery import SourceFileLoader
gr = SourceFileLoader('gr', 'bin/gen-runbook').load_module()
... build a synthetic host with pre_update/post_update/verify desc: fields
each containing an apostrophe, call gr.generate_sh(), bash -n the output
"

Symptoms

• after git pull, bin/collect-homelab returns Permission denied
• scripts in bin/ lost execute bit after push
• bin/ scripts permanently show as modified in git status / VS Code, even after pulling

Cause

Fix

chmod +x bin/collect-homelab bin/resolve-hosts bin/generate-hosts-yaml
git add bin/collect-homelab bin/resolve-hosts bin/generate-hosts-yaml
git commit -m "chore: restore execute bit on scripts"
git push

Hook setup (one-time on developer-env)

cat > ~/projects/homelab/.git/hooks/post-merge << 'EOF'
#!/bin/bash
v3: detects scripts by shebang (#!) rather than by directory (bin/ only),
so execute bits get restored regardless of where a script lives in the repo.
changed=$(git diff-tree -r --name-only --no-commit-id ORIG_HEAD HEAD 2>/dev/null)
to_fix=""
for f in $changed; do
    [ -f "$f" ] || continue
    if head -c 2 "$f" 2>/dev/null | grep -q '^#!'; then
        to_fix="$to_fix $f"
    fi
done
if [ -n "$to_fix" ]; then
    chmod +x $to_fix
    git add $to_fix
    if ! git diff --cached --quiet; then
        git commit -m "chore: restore execute bit on scripts [post-merge]" --quiet
        git push --quiet
        echo "[post-merge] restored execute bits and committed:$to_fix"
    fi
fi
EOF
chmod +x ~/projects/homelab/.git/hooks/post-merge

Also set globally

git config --global rebase.autoStash true

Symptoms

• curl from raw.githubusercontent.com returns 404 page
• script fails with 'line 1: 404:: command not found'
• file is visible in repo browser but not via raw URL

Cause

Diagnosis

head -3 /root/<script>
Shows HTML or "404: Not Found" → curl fetched the 404 page

Fix

cat > /root/<script>.sh << 'HEREDOC'
#!/usr/bin/env bash
... paste full script content ...
HEREDOC
chmod +x /root/<script>.sh
head -3 /root/<script>.sh   # verify

Prevention

Symptoms

• ntfy alert: Update FAILED: grafana
• update_cmd exited 100 / dpkg returned an error code (1)
• dpkg-preconfigure: unable to re-open stdin (stage 1)
• cannot copy extracted data ... failed to write (No space left on device) (stage 2)
• mv: cannot overwrite '/var/lib/grafana/plugins-bundled': Directory not empty (stage 3)

Summary

Stage 1 — debconf preconfigure, no stdin (the automated failure)

Stage 2 — rootfs out of space during unpack

cannot copy extracted data for './usr/share/grafana/bin/grafana' to
'.../grafana.dpkg-new': failed to write (No space left on device)

# on proxmox-nuc (live, non-destructive; runs resize2fs for you):
pct resize 101 rootfs +4G          # 4G → 8G

Stage 3 — postinst can't replace a non-empty plugins-bundled dir

mv: cannot overwrite '/var/lib/grafana/plugins-bundled': Directory not empty
dpkg: error processing package grafana (--configure): ... exit status 1

mv /var/lib/grafana/plugins-bundled /var/lib/grafana/plugins-bundled.old
dpkg --configure grafana          # resumes the interrupted configure
verify BEFORE cleanup:
grafana --version                 # 13.1.1
curl -s http://localhost:3000/api/health   # "database": "ok"
dpkg -l grafana | tail -1         # must be 'ii', not 'iF'
rm -rf /var/lib/grafana/plugins-bundled.old
apt-get clean

Two standing facts this incident nailed down

Related

Symptom

Root cause -- three independent mechanisms, not one bug

Fix

Symptom

Root cause

Fix

Symptom

  verify: Health endpoint returns pass
  FAILED: expected '{"status":"pass"}' in output
  got:
[ABORT: verify failed: headscale — Health endpoint returns pass]

Root cause

Fix

_verify() {
    local desc="$1" cmd="$2" expected="$3" run_from="$4" user="$5" host="$6"
    echo "  verify: $desc"
    local out attempt=0
    while : ; do
        if [ "$run_from" = "local" ]; then out=$(eval "$cmd" 2>&1) || true
        else out=$(_ssh "$user" "$host" "$cmd" 2>&1) || true; fi
        if [ -z "$expected" ] || echo "$out" | grep -qF "$expected"; then break; fi
        attempt=$((attempt + 1))
        if [ "$attempt" -ge "$_VERIFY_MAX_ATTEMPTS" ]; then
            echo "  FAILED after ${_VERIFY_MAX_ATTEMPTS} attempts: expected '$expected' in output"
            echo "  got: $out"; return 1
        fi
        sleep "$_VERIFY_RETRY_SLEEP"
    done
    echo "  ✓ $out"
}

Deliberately NOT done

Applies to

Latent gap noted (not fixed here)

Also observed: group1-preflight false-FAIL on developer-env :5000 (2026-08-03)

Symptoms

• Headplane UI returns 404
• Cannot connect to Headscale API
• ECONNREFUSED 127.0.0.1:8080

Causes

Correct `/etc/headplane/config.yaml`

headscale:
  url: http://192.168.42.177:8080    # LAN IP, NOT 127.0.0.1
  api_key: <key>
server:
  host: 0.0.0.0
  port: 3000
  cookie_secret: "<32-char-secret>"  # must be quoted
  base_url: https://headplane.compellinglylowbrow.org

Correct docker run command

docker stop headplane && docker rm headplane
mkdir -p /var/lib/headplane

docker run -d \
  --name headplane \
  --restart unless-stopped \
  -p 3000:3000 \
  -v /etc/headplane:/etc/headplane \
  -v /var/lib/headplane:/var/lib/headplane \
  -v /etc/headscale/config.yaml:/etc/headscale/config.yaml \
  -v /var/run/headscale/headscale.sock:/var/run/headscale/headscale.sock \
  ghcr.io/tale/headplane:latest

Verify

docker logs headplane 2>&1 | tail -5
curl -v http://192.168.42.177:3000/admin/
Should return 302 redirect to /admin/machines

Symptoms

• headplane.compellinglylowbrow.org returns 502
• docker ps -a shows headplane Exited (127) after a host/LXC reboot
• restart policy is unless-stopped but container is not running
• headscale itself (systemd, port 8080) is healthy -- only headplane (docker, port 3000) is down

Cause

Diagnosis

ssh headscale "sudo ss -tlnp | grep 3000"          # nothing listening = down
ssh headscale "docker ps -a | grep headplane"       # check Exited state/code
ssh headscale "docker logs headplane --tail 50"     # look for SIGTERM near reboot time, then a gap
ssh headscale "docker inspect headplane --format '{{.HostConfig.RestartPolicy.Name}}'"

Manual fix (fallback, if the systemd unit below is ever missing/disabled)

ssh headscale "docker start headplane"
ssh headscale "docker logs headplane --tail 20"   # confirm clean startup, no immediate exit

Permanent fix (2026-07-10) -- systemd unit, survives reboot

ssh headscale "sudo tee /etc/systemd/system/headplane-start.service" <<'EOF'
[Unit]
Description=Ensure headplane docker container is running
After=docker.service network-online.target
Requires=docker.service
Wants=network-online.target

[Service]
Type=oneshot
ExecStart=/usr/bin/docker start headplane
RemainAfterExit=yes
ExecStartPre=/bin/sleep 5

[Install]
WantedBy=multi-user.target
EOF

ssh headscale "sudo systemctl daemon-reload && sudo systemctl enable headplane-start.service"

Automated diagnosis (2026-07-06)

Resolved: open question / follow-up

python3 bin/docker-exited-sweep

Symptom

Root cause (working theory for the 2026-07-03 failures)

Fix

Testing gotchas that caused false alarms during the 2026-07-12 retry

Result

Symptoms

• HA addon connects but never authenticates
• changing login_server in addon config has no effect

Cause

Fix

# From HA terminal / SSH addon
rm /mnt/data/supervisor/addons/data/a0d7b954_tailscale/tailscaled.state

Symptoms

• Home Assistant Companion App won't load on iPhone, both on and off LAN
• Safari via homeassistant.compellinglylowbrow.org works fine on the same iPhone
• Direct LAN-IP access to HA (192.168.42.113:8123) works fine
• Issue persists identically regardless of physical network (Wi-Fi vs cellular)

Cause

Diagnosis

ssh developer-env
grep -A2 '"hosts"' -A40 headscale/acl.hujson | grep homeassistant

Fix

"homeassistant": "100.64.0.110/32"   // added to hosts{}

{
  // Rule 9: HA Companion App's "External URL" connects directly via
  // Tailscale IP (100.64.0.110:8123), bypassing Caddy entirely -- not
  // just control/browsing via homeassistant.compellinglylowbrow.org
  // (rule 1). homeassistant was never declared in this policy at all
  // (not in hosts{}, not a dst anywhere) -- same missing-host pattern
  // as seeder-daemon/lyrionmusicserver, this time triggered by a
  // personal device's app config rather than a service dependency.
  "action": "accept",
  "src": ["iphone"],
  "dst": ["homeassistant:8123"]
}

Broader lesson

Unresolved side note

Symptoms

• tailscale up runs without error but node never appears in Headscale
• node does not show after tailscale up

Cause

Fix

sudo rm /var/lib/tailscale/tailscaled.state
sudo systemctl restart tailscaled
sudo tailscale up --login-server https://headscale.compellinglylowbrow.org --auth-key <key>

Symptom

FAIL  adguard recovery hatch adguard.compellinglylowbrow.org → '100.64.0.4' (expected 192.168.42.27)
WARN  adguard missing adguard2.compellinglylowbrow.org recovery hatch (got '100.64.0.4', expected 192.168.42.89)

Root cause

Fix

# 3. adguard's own hostname — no dedicated recovery hatch BY DESIGN (2026-07-18).
AG_RESULT=$(dns_query "$ADGUARD_IP" "adguard.${DOMAIN}")
if [[ "$AG_RESULT" == "100.64.0.4" ]]; then
    pass "adguard adguard.${DOMAIN} → 100.64.0.4 (wildcard fallthrough, by design)"
else
    warn "adguard adguard.${DOMAIN} → '${AG_RESULT}' (expected wildcard 100.64.0.4; a LAN-IP hatch here was removed 2026-07-18)"
fi

Note (still open, deliberately untouched)

Symptoms

• bin/health-check backend-reachability section shows a lone '000' on its own line, right after a PASS/FAIL entry for a backend that returned 4xx/5xx or was unreachable
• a genuinely-unreachable backend can show as PASS with 'HTTP 000' instead of FAIL -- this was a real accuracy bug, not cosmetic (see CORRECTION below)

CORRECTION (2026-07-12, same day as the original entry)

Cause

Fix

caddy_curl() {
    local proto="$1" hostport="$2"
    local url="${proto}://${hostport}/"
    local flags="-sf --max-time 5 -o /dev/null -w %{http_code}"
    [[ "$proto" == "https" ]] && flags="-sf -k --max-time 5 -o /dev/null -w %{http_code}"
    local code
    code=$(ssh_cmd "$CADDY_IP" "curl ${flags} ${url}" 2>/dev/null \
        | grep -oE '[0-9]{3}' | head -1 || true)
    echo "${code:-000}"
}

Verify

Follow-up

Symptom

Root cause

Fix

Generalizes to

Symptoms

• bin/health-check reports stirling-pdf as FAIL/not-found even though it was decommissioned (LXC 105 now hosts alpine-it-tools)
• bin/health-check reports vaultwarden, alpine-it-tools, watchdog, raspi4, ntfy, uptime-kuma, and nas as Headscale 'orphans' or missing backend checks even though all are real, active, hosts-config.yaml-tracked services
• bin/health-check permanently WARNs on raspi5, debian-test, debian-test-onboard -- none of which exist anywhere in hosts-config.yaml
• moslaptop (health-check's hardcoded spelling) / mospclaptop (hosts-config.yaml's headscale_devices spelling) / MOSLaptop (the live Headscale registration) are three different spellings of the same device -- still unresolved, see Follow-up

Cause

Fix

Verify

Follow-up

Symptoms

• watchdog ntfy: 'Watchdog: manual intervention required -- Monitors down: Proxmox, nastynas — Web UI -- Pattern does not match any known failure mode'
• nastynas and pibox unreachable on LAN (ping/SSH) while individually healthy -- nastynas still reachable and fine over its Headscale IP
• every other LAN host (proxmox-nuc, watchdog, developer-env, and initially the switch's own management IP) stays reachable
• after power-cycling the switch: nastynas/pibox forwarding recovers, but the switch's management IP stops answering ping/curl entirely -- 'Destination Host Unreachable' (ARP failure), not a timeout
• the switch's management IP doesn't appear in the gateway's wired-client list under any address
• TP-Link's Omada Discovery Utility (L2 broadcast) can't find the device either

Symptom

Root cause

Fix

Diagnosis commands

# Forwarding wedge signature: multiple specific hosts down, everything
else on the LAN fine, hosts individually healthy over Headscale
ping -c 3 192.168.42.200   # nastynas
ping -c 3 192.168.42.117   # pibox
ssh root@100.64.0.113 "uptime"   # nastynas over Headscale -- proves the host itself is fine

Confirm it's ARP failure, not a drop -- "Destination Host Unreachable"
from your OWN source IP means the kernel never got an ARP reply, vs. a
plain 100% packet loss timeout which is a silent drop somewhere else
ping -c 3 192.168.42.184

caddy validate needs the same env the systemd unit injects
ssh root@<caddy_ip> 'set -a; for _ef in $(systemctl cat caddy | sed -n "s/^EnvironmentFile=-\?//p"); do [ -f "$_ef" ] && . "$_ef"; done; caddy validate --config /etc/caddy/Caddyfile'

Prevention

Related

Symptoms

• git pull on a host with a repo clone aborts: 'Your local changes to the following files would be overwritten by merge'
• A service appears in a host's live compose file and in prose documentation, but grep finds it nowhere in the tracked config, and docker ps shows no such container
• docker compose config fails with 'env file ... not found' while targeted commands like docker compose up -d <service> keep working normally
• A host's repo clone is many commits behind HEAD, so a real divergence shows up mixed in with ordinary staleness and is easy to wave through

Symptom

error: Your local changes to the following files would be overwritten by merge:
        watchdog/compose.yaml
Aborting

The actual lesson: the same symptom, two opposite correct fixes

How this entry got it wrong the first time

Procedure

# 1. Read the diff. For each hunk, ask: is this live?
ssh <host> "cd ~/homelab && git log --oneline -1 && git status --short && git diff <path>"

2. PROVE it, per divergent item. Documentation does not count.
ssh <host> "docker ps"                     # containers
ssh <host> "systemctl is-active <unit>"    # services
ssh <host> "test -f <every referenced path> && echo present"

3. Back up the live file BEFORE discarding anything
ssh <host> "cp ~/homelab/<path> /tmp/<file>.live.bak"

4a. If live and untracked -> reconcile INTO git, push, then:
4b. If not live -> fix in git (disable/delete), push, then:
ssh <host> "cd ~/homelab && git checkout <path> && git pull"

5. Gate on the diff: it must show ONLY what you intended
ssh <host> "diff /tmp/<file>.live.bak ~/homelab/<path>"

6. For compose files specifically, confirm the file still parses.
   NOTE the single quotes -- with double quotes, $? expands on YOUR shell,
   not the remote one, and reports the ssh client's status instead. This
   exact mistake produced a misleading `exit=0` during this incident.
ssh <host> 'cd ~/homelab/watchdog && docker compose config >/dev/null 2>&1; echo "exit=$?"'

Prevention — implemented 2026-07-25

Scope

Related non-issue, recorded so it is not re-investigated

Symptoms

• A multi-line YAML folded scalar (cmd: >-) with a wrapped ssh/curl invocation splits into two invalid commands
• run-update's no_auto_reboot guard blocks a step that never actually reboots
• A dist-upgrade update_cmd works fine as root but fails/does nothing for a non-root ssh_user host

Symptom 1: a wrapped command inside `cmd: >-` silently splits into two commands

cmd: >-
  for i in $(seq 1 30); do
    ssh -o BatchMode=yes -o ConnectTimeout=5 -o StrictHostKeyChecking=accept-new
    root@192.168.42.63 'echo up' 2>/dev/null && exit 0;
    sleep 5;
  done; exit 1

Root Cause

Fix

python3 -c "
import yaml
cfg = yaml.safe_load(open('inventory/hosts-config.yaml'))
... find the entry, print repr(entry['version']['verify'][0]['cmd'])
"

Symptom 2: `no_auto_reboot`'s guard blocks a step that never reboots

Root Cause

Fix

post_update:
  - desc: "Reboot if the update left one pending, otherwise no-op"
    reboots: true   # required for run-update's no_auto_reboot guard to catch this
    cmd: >-
      if [ -f /var/run/reboot-required ]; then nohup bash -c 'sleep 2 && reboot' >/dev/null 2>&1 & echo triggered; else echo none; fi

Symptom 3: `apt-get` works fine over SSH as root, but silently does nothing (or fails) for a non-root `ssh_user` host

Fix

if ssh_user != "root":
    # sudo env VAR=val cmd -- env sets its OWN child's environment directly,
    # independent of sudo's env-filtering policy. Safer than a bare export.
    noninteractive_cmd = f"sudo env DEBIAN_FRONTEND=noninteractive bash -c {shlex.quote(update_cmd)}"
else:
    noninteractive_cmd = f"export DEBIAN_FRONTEND=noninteractive; {update_cmd}"

Symptoms

• A batch of unrelated-looking HA integrations all broken at once: enphase_envoy stuck in ConfigEntryState.SETUP_IN_PROGRESS for days (bootstrap log: 'Waiting for integrations to complete setup' with an elapsed time in the hundreds of thousands of seconds), wiim/ipp/brother in SETUP_RETRY
• Every entity under the affected integration shows unavailable/unknown -- not a handful, the whole device tree (Envoy: 32 entities; WiiM: media_player + button; Brother: printer status sensors)
• ping/curl/nc from any Servers-VLAN host (developer-env, or HA itself) to the device's LAN IP times out -- no refusal, a hang, same signature as ufw default-deny or a missing Headscale ACL entry
• The device's IP is on 192.168.12.0/24 (Trusted) even though it's a Wi-Fi IoT-class device (solar monitor, printer, multiroom speaker) that was never meant to live there long-term

Context

Root cause

ssh ucg-fiber "timeout 20 tcpdump -ni any 'host <ha-ip> and host <device-ip>' -c 60"
then, in HA:
homeassistant.reload_config_entry  # entry_id: <the stuck integration>

Fix -- six new UCG-Fiber zone-firewall rules

Outstanding, not fixed

Prevention

Symptoms

• iPhone on home WiFi (GDTRFB, Trusted VLAN): general internet felt slow, some external sites failed to load entirely
• iOS Wi-Fi settings showed a normal, correct LAN IP (192.168.12.240) and full signal -- looked nothing like a WiFi/AP problem
• Live tcpdump on the gateway showed the phone reaching ARP/mDNS instantly -- L2 connectivity was never the issue
• Gateway conntrack showed a genuine SYN_SENT/UNREPLIED entry: 192.168.12.240 -> 100.64.0.110:8123 (Home Assistant's Headscale IP), packets sent, zero replies, indefinitely
• Tailscale app on the iPhone was found disconnected; reconnecting it (status -> Connected) fixed browsing immediately, no other change needed

Symptom

Root cause

Diagnosis

ssh ucg-fiber "ip neigh show | grep 192.168.12.240"        # REACHABLE, not STALE
ssh ucg-fiber "conntrack -L | grep 192.168.12.240"          # live SYN_SENT/UNREPLIED to 100.64.0.110:8123
ssh ucg-fiber "tcpdump -i br10 -n host 192.168.12.240"      # confirmed ARP/mDNS fine, ruled out L2/WiFi

Fix

Follow-up: systemic fix landed same day

Prevention

Symptom

Root cause

Fix

{
  "action": "accept",
  "src": ["macbook", "iphone", "developer-env"],
  "dst": ["192.168.42.82:*"]
}

Prevention

Related

Symptoms

• ffmpeg exits with 'Unrecognized option C. Error splitting the argument list' after a Debian 12→13 upgrade
• LMS FLAC-route transcode (custom-convert.conf 'flc' rules) fails after OS upgrade; PCM/wav route still works
• bin/version-checker intermittently reports 'SSH failed or no output' for a host right after collect-homelab's parallel collection phase, even though the host is healthy

Context

Procedure (worked cleanly)

The real regression: ffmpeg `-C 0`

[/usr/bin/ffmpeg] -v 0 -i $FILE$ -f flac -C 0 -

Unrecognized option 'C'.
Error splitting the argument list: Option not found

Minor unresolved observation (not a real problem)

Symptom

Root cause

Fix

   pct set <vmid> -memory <new_MB>
   

   pct set <vmid> -swap <new_MB>
   

   pct reboot <vmid>
   

Symptoms

• Newly downloaded albums don't appear in LMS/lyrtui search or browse until a manual rescan
• autorescan pref shows 1 via CLI (`pref autorescan ?`) but new files still aren't picked up automatically
• Settings > Performance in the LMS web UI doesn't show an Autorescan checkbox at all
• scanner.log shows exactly one 'Discovering audio files' entry (from server startup / from toggling the pref), then nothing, even hours later with real new files added

Symptom

Cause -- three layers deep, only the last one actually mattered

Fix

sub canAutoRescan { 1 }

Making the patch durable across OS-package updates (added 2026-08-15)

Prevention / lesson

Symptoms

• After a normal reboot, Tailscale won't connect (stuck), nothing homelab-related resolves -- looks like a full Tailscale/DNS outage
• Claude Desktop app and general internet browsing work fine the whole time -- only homelab-specific reachability is broken
• networksetup -getairportnetwork en0 shows the laptop joined to Guest_GDTRFB, not GDTRFB (the household SSID, now on Trusted VLAN) or IoT_GDTRFB
• GDTRFB was already the top-priority (slot 0) entry in the Mac's preferred-networks list, yet a lower-priority SSID was joined instead

Symptom

Root cause

Fix

Diagnosis commands worth keeping

# Which SSID is actually joined right now
networksetup -getairportnetwork en0

Preferred-network order (first entry = highest priority, in theory)
networksetup -listpreferredwirelessnetworks en0

Remove a stale/test SSID from the known-networks list entirely
sudo networksetup -removepreferredwirelessnetwork en0 "Guest_GDTRFB"
sudo networksetup -removepreferredwirelessnetwork en0 "IoT_GDTRFB"

Prevention

Current impact

Symptoms

• One specific *.compellinglylowbrow.org hostname fails to load while every other one on the same domain works fine, from the same client, at the same time
• sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder does not change anything for the affected hostname
• curl -v shows the hostname resolving to Caddy's LAN IP (192.168.42.45) instead of its Headscale IP (100.64.0.4)
• The connection succeeds at the TCP/TLS layer but the server returns 403 Forbidden
• AdGuard's own query log shows zero queries for the affected hostname from this client, even though every other hostname is being queried normally

Symptom

Root cause

192.168.42.45 seeder-daemon.compellinglylowbrow.org

Diagnosis

grep -i <hostname> /etc/hosts

Fix

Prevention

Symptoms

• bulk transfer from nastynas to wildwood stalls at ~100-140 kB/s
• tailscale ping shows via public DERP relay
• nastynas to wildwood never achieves direct connection

Current state (2026-06-27)

Root causes

Decision (2026-06-27)

SUPERSEDED (2026-08-09) — "all offsite backups originate from proxmox-nuc" is no longer true

Verify DERP fix is intact

# Confirm HTTP/1.1 transport for /derp in Caddyfile
ssh caddy "grep -A8 'headscale.compellinglylowbrow.org {' /etc/caddy/Caddyfile"
Must show @derp matcher and versions 1.1

Confirm DERP responds to HTTP/1.1 from nastynas
ssh nastynas "curl -v --http1.1 --max-time 5 -H 'Upgrade: WebSocket' -H 'Connection: Upgrade' https://headscale.compellinglylowbrow.org/derp 2>&1 | grep 'HTTP/1'"
Must return HTTP/1.1 200 OK (not 426)

Symptoms

• curl/Prometheus scrape to an internal *.compellinglylowbrow.org hostname times out after the full timeout window instead of failing immediately
• DNS resolves correctly (dig/nslookup returns an answer), the connection itself hangs
• curl -v shows 'Trying <IP>...' then nothing until the timeout
• the target IP is in the 100.64.0.0/10 range (a Headscale/Tailscale address)
• `tailscale` command not found, or `tailscale status` shows not connected, on the host making the request

Cause

Diagnosis

# is this host even a tailnet peer?
tailscale status 2>&1 | head -5

what does its own resolver actually return?
dig +short <hostname>.compellinglylowbrow.org

the real, unabstracted symptom -- don't trust an abstraction layer
(Prometheus's target status page, in this case) as the first signal
curl -v --max-time 5 https://<hostname>.compellinglylowbrow.org/<path> 2>&1 | tail -20

Fix

echo "192.168.42.45 metrics.compellinglylowbrow.org" | sudo tee -a /etc/hosts

Note

Symptoms

• Script log says 'ntfy sent' but nothing arrives in the ntfy app/UI
• ntfy send failed: 'latin-1' codec can't encode character ... in position 0: ordinal not in range(256)
• A HOLD/CAUTION verdict with an emoji-prefixed title never shows up, but SAFE (no icon, or an icon that happens to work) does

Cause

Diagnosis

# Look for this specific error in a script's output/log:
grep "codec can't encode character" <script-output>

Confirm the title that was attempted contained non-ASCII:
python3 -c "print('<the title string>'.isascii())"
False => confirmed cause

Fix

Prevention

Symptom

Root cause

Fix

vfs object = recycle

vfs objects = fruit streams_xattr recycle
fruit:metadata = stream
fruit:posix_rename = yes
fruit:veto_appledouble = no
fruit:wipe_intentionally_left_blank_rfork = yes
fruit:delete_empty_adfiles = yes

ssh root@192.168.42.63
smbpasswd -a mos

Prevention

Related

Symptom

Root cause

Fix / prevention

Related incident, same session

Generalizes to

Symptoms

• apt-get update fails with BADSIG during Tailscale install
• sqv: No binding signature valid under the policy
• Tailscale install fails on Debian 13 trixie

Cause

Fix

. /etc/os-release
mkdir -p /usr/share/keyrings
curl -fsSL -4 "https://pkgs.tailscale.com/stable/${ID}/${VERSION_CODENAME}.noarmor.gpg" \
    -o /usr/share/keyrings/tailscale-archive-keyring.gpg
curl -fsSL -4 "https://pkgs.tailscale.com/stable/${ID}/${VERSION_CODENAME}.tailscale-keyring.list" \
    -o /etc/apt/sources.list.d/tailscale.list
apt-get update -qq
apt-get install -y tailscale

Note

Symptoms

• ssh root@<container-ip> returns Permission denied (publickey) after onboarding
• container is reachable by IP and Headscale registration succeeded

Cause

Fix

pub=$(cat ~/.ssh/id_ed25519.pub)
ssh root@<proxmox-node-ip> "pct exec <vmid> -- sh -c \
  'mkdir -p /root/.ssh && chmod 700 /root/.ssh && \
   echo \"$pub\" >> /root/.ssh/authorized_keys && \
   chmod 600 /root/.ssh/authorized_keys'"

Verify
ssh root@<container-ip> hostname

Note

Symptoms

• onboard-container Phase 5 stalls during Tailscale installation
• curl commands hanging indefinitely on secondary Proxmox node
• same container onboards fine on proxmox-nuc

Cause

Diagnosis

curl -v https://pkgs.tailscale.com/stable/debian/bookworm.noarmor.gpg 2>&1 | head -10
Hangs after "Trying 2620:..." → IPv6 no route

curl -4 -v https://pkgs.tailscale.com/stable/debian/bookworm.noarmor.gpg 2>&1 | head -5
Connects immediately → confirmed IPv4-only container

Fix

Summary

Fix

Two more bugs found while testing this fix, before it shipped

Verification

Prevention / lesson

Symptoms

• Proxmox UI task log shows 'Update package database' failing: command 'apt-get update' failed: exit code 100
• apt-get update shows: Err:N https://enterprise.proxmox.com/debian/pbs trixie InRelease 401 Unauthorized
• E: The repository 'https://enterprise.proxmox.com/debian/pbs trixie InRelease' is not signed
• unattended-upgrades has been failing for days/weeks with no other visible symptom -- the box otherwise looks healthy

Cause

Diagnosis

ssh nastynas "cat /etc/apt/sources.list.d/*.sources"
ssh nastynas "apt-get update"   # look for Err: ... 401 Unauthorized on an enterprise.proxmox.com URL

Fix

ssh nastynas "echo 'Enabled: false' >> /etc/apt/sources.list.d/pbs-enterprise.sources"
ssh nastynas "apt-get update"   # confirm all Hit:/Get:, no Err:

Fleet-wide sweep

for h in proxmox-nuc macbookpro-pve nastynas; do
  echo "=== $h ==="
  ssh $h "grep -L 'Enabled: false' /etc/apt/sources.list.d/*enterprise*.sources 2>/dev/null || echo none-missing"
done

Follow-up (not yet done)

Symptoms

• ntfy: 'nastynas offsite sync: FAILED', 'proxmox-nuc offsite sync: FAILED', and/or 'wildwood offsite sync: FAILED' (exit 127)
• sync.log shows: <script>.sh: line N: proxmox-backup-manager: command not found
• manually re-running the same script over SSH works fine
• local vzdump/backup jobs are unaffected -- only the offsite push fails

Cause

Diagnosis

ssh root@100.64.0.113 "tail -30 /var/log/nastynas-self-offsite-sync/sync.log"
ssh root@100.64.0.113 "tail -30 /var/log/proxmox-nuc-offsite-sync/sync.log"
Look for: "<script>.sh: line N: proxmox-backup-manager: command not found"

Confirm it's a PATH issue, not a missing/broken install:
ssh root@100.64.0.113 "which proxmox-backup-manager"   # -> /usr/sbin/proxmox-backup-manager (found)
ssh root@100.64.0.113 "env -i /bin/sh -c 'PATH=/usr/bin:/bin; proxmox-backup-manager version'"
-> not found, reproduces cron's exact failure

Resolution

Prevention checklist for any new cron-triggered script on nastynas, wildwood

Symptoms

• apt-get install proxmox-backup-server returns E: Unable to locate package

Cause

Fix

cat > /etc/apt/sources.list.d/pbs.sources << 'EOF'
Types: deb
URIs: http://download.proxmox.com/debian/pbs
Suites: trixie
Components: pbs-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF
apt-get update && apt-get install -y proxmox-backup-server

Note

Symptoms

• pvesm add pbs returns 401 Unauthorized
• correct credentials but still 401

Cause

Fix

# Wrong:
pvesm add pbs ... --username backup@pbs

Correct:
pvesm add pbs ... --username backup@pbs!nuc

Symptoms

• Cannot find datastore 'backups', check permissions and existence
• datastore exists but 403 on pvesm add

Cause

Fix

ssh nastynas "proxmox-backup-manager acl update /datastore/backups DatastoreBackup --auth-id backup@pbs!nuc"
ssh nastynas "proxmox-backup-manager acl update /datastore/backups DatastoreAudit --auth-id backup@pbs!nuc"

Verify
ssh nastynas "proxmox-backup-manager acl list"

Symptoms

• proxmox-backup-manager sync-job run fails
• Querying namespaces failed - HTTP error 403 Forbidden

Cause

Fix

ssh wildwood "proxmox-backup-manager acl update /datastore/pbs DatastoreBackup --auth-id sync@pbs!nastynas"
ssh wildwood "proxmox-backup-manager acl update /datastore/pbs DatastoreAudit --auth-id sync@pbs!nastynas"

Symptoms

• sync job completes but syncs wrong direction
• destination content appears in source

Cause

Implication

Setup

# On wildwood: create remote pointing at nastynas
proxmox-backup-manager remote create nastynas \
  --host 100.64.0.113 \
  --auth-id backup@pbs!nuc \
  --password <token-value> \
  --fingerprint <nastynas-fingerprint>

On wildwood: create sync job
proxmox-backup-manager sync-job create nastynas-nightly \
  --store pbs \
  --remote nastynas \
  --remote-store backups \
  --schedule 'daily' \
  --remove-vanished false

Symptom

Fix

Prevention / notes for next time

Symptom

Root cause

ExecStart=/opt/prometheus/prometheus --config.file=/opt/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.listen-address=192.168.42.119:9090

Fix

ssh root@192.168.42.119 "systemctl restart prometheus"

curl -s http://192.168.42.119:9090/api/v1/targets | python3 -m json.tool

Prevention / next time

Symptoms

• after apt upgrade and reboot, proxmox-nuc DNS reverts to stale Tailscale entries
• 100.100.100.100 appears as DNS after upgrade

Fix

sudo tailscale set --accept-dns=false

Symptoms

• LXC loses network or can't resolve DNS after host reboot

Check

Note

Symptoms

• ping/ssh to raspi4's documented LAN IP (192.168.42.83) fails from every Servers-VLAN host (developer-env, watchdog) -- 'Destination Host Unreachable' / 'No route to host'
• ip neigh show <ip> on developer-env shows the ARP entry as FAILED / (incomplete) -- not a firewall-style hang, a genuine no-reply
• daily-fleet-digest's 06:30 apt-get-update health check ntfy flags exactly one host as FAILED, all others OK
• the host is 100% physically fine -- pinging/SSHing it from the gateway's own segment (or any Trusted-VLAN host) works instantly

Symptom

Root cause

Diagnosis

ssh ucg-fiber "ip neigh show | grep <suspect-mac-OUI-or-recent-lease>"
ssh ucg-fiber "ping -c2 <candidate-ip-on-another-VLAN>"
ssh ucg-fiber "ssh -o BatchMode=yes mos@<candidate-ip> hostname"  # Permission denied = alive, just no key here

Fix

   ssh ucg-fiber "iptables -S | grep -i 192.168.12.74"
   ssh developer-env-or-watchdog "ssh -o ConnectTimeout=5 mos@192.168.12.74 echo up"
   

Current impact

Prevention

Symptoms

• backup-music.sh reports leg: pibox FAILED (exit=23)
• delete_file: unlink failed: Permission denied
• cannot delete non-empty directory
• files owned by different UID than SSH user

Cause

Diagnosis

ssh -i /root/.ssh/id_rsa pibox@100.64.0.116 \
  "ls -lan '/srv/dev-disk-by-uuid-.../PiBoxShared/shared/<failing-dir>/'"
Files owned by UID != 1000 → ownership mismatch

ssh -i /root/.ssh/id_rsa pibox@100.64.0.116 "id"
uid=1000(pibox)

Fix

ssh -i /root/.ssh/id_rsa pibox@100.64.0.116 \
  "sudo chown -R pibox:users '/srv/dev-disk-by-uuid-.../PiBoxShared/shared/'"

Note

Symptoms

• backup-music.sh reports leg: wildwood FAILED (exit=255)
• mos@100.64.0.3: Permission denied (publickey,password)
• rsync error: unexplained error (code 255)

Cause

Diagnosis

ssh -i /root/.ssh/id_ed25519 -o BatchMode=yes -o ConnectTimeout=10 mos@100.64.0.3 "whoami"
Permission denied → key not authorized

ssh -i /root/.ssh/id_rsa -o BatchMode=no mos@100.64.0.3 "cat ~/.ssh/authorized_keys"
Lists only ssh-rsa keys → id_ed25519.pub is missing

Fix

ssh -i /root/.ssh/id_rsa mos@100.64.0.3 \
  "echo '$(cat /root/.ssh/id_ed25519.pub)' >> ~/.ssh/authorized_keys"

ssh -i /root/.ssh/id_ed25519 -o BatchMode=yes -o ConnectTimeout=10 mos@100.64.0.3 "whoami"
Returns "mos" → fixed

Symptoms

• run-update <service> fails immediately: 'Cannot classify update_cmd ... use gen-runbook for manual updates'
• update-advisor auto-execute attempts for a docker-compose-managed service would fail every cycle and fire a false urgent 'Update FAILED' ntfy, if auto_update were ever flipped to true
• any SSH call in run-update/update-apt/update-binary hardcoded root@<host>, which is wrong for hosts whose ssh_user isn't root (e.g. watchdog)

Cause

runtee bin/run-update ntfy --dry-run   # confirm dispatch classification + ssh_user in the JSON first
runtee bin/run-update ntfy             # then the real thing

Symptom

Root cause

Fix

ssh proxmox-nuc 'pct exec 110 -- ufw allow from 192.168.42.31 to any port 22 comment "developer-env ssh"'

ssh proxmox-nuc 'pct exec 110 -- ufw allow from 192.168.42.229 to any port 8888 comment "uptime-kuma health check"'

[botdetection.ip_lists]
pass_ip = [
  "192.168.42.0/24",
  "100.64.0.0/10",
]

Prevention — apply to any future first-time `ufw enable` on a service host

Related

Symptom

Root cause

Fix

Prevention

Open / unconfirmed

Related

Symptoms

• squeezelite log shows: error opening portaudio stream: Invalid number of channels
• squeezelite log shows: unable to open output device: <AirPods name>, even though squeezelite -l lists that exact name as a valid output device
• Device name matches exactly (no typo/encoding mismatch) but the open still fails
• squeezelite -o <index> for the same device (from squeezelite -l) works fine, immediately

Symptom

test_open:238 error opening portaudio stream: Invalid number of channels
output_init_common:401 unable to open output device: Matt's AirPods

Cause

Matt's AirPods:
  Default Input Device: Yes
  Input Channels: 1
  Current SampleRate: 24000        <- HFP / mic side, mono, low quality

Matt's AirPods:
  Default Output Device: Yes
  Output Channels: 2
  Current SampleRate: 48000        <- A2DP / media side, stereo, high quality

Fix

resolve_output_index() {
    local device_name="$1"
    "$SQUEEZELITE" -l 2>&1 \
        | awk '/^Output devices:/{f=1;next} /^Input devices:/{f=0} f' \
        | grep -F "$device_name" \
        | head -1 \
        | sed -E 's/^[[:space:]]*([0-9]+).*/\1/'
}

Prevention / lesson

Symptoms

• Squeezelite player randomly stops appearing in LMS web UI
• Only fixed by manually running squeezelite from Terminal (e.g. with -d all=debug)
• squeezelite log shows: unable to open output device: default
• ps aux shows two squeezelite processes running simultaneously
• duplicate process reappears after every reboot despite removing Login Item

Cause (two independent bugs stacked together)

   output_init_common:401 unable to open output device: default
   

   /Applications/Squeezelite.app/Contents/MacOS/squeezelite -l
   

Fix (2026-07-06 attempt)

osascript -e 'tell application "System Events" to delete login item "Squeezelite"'
osascript -e 'tell application "System Events" to get the name of every login item'

<key>ProgramArguments</key>
<array>
    <string>/Applications/Squeezelite.app/Contents/MacOS/squeezelite</string>
    <string>-n</string>
    <string>MacBook</string>
    <string>-s</string>
    <string>100.64.0.8</string>
    <string>-o</string>
    <string>MacBook Air Speakers</string>
    <string>-a</string>
    <string>100:0</string>
    <string>-f</string>
    <string>/tmp/squeezelite.log</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>

launchctl bootout gui/$(id -u)/com.user.squeezelite 2>/dev/null
launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.user.squeezelite.plist
launchctl enable gui/$(id -u)/com.user.squeezelite

Verification (do a real reboot, not just a reload)

ps aux | grep -i squeezelite | grep -v grep     # exactly one process, with -o flag
cat /tmp/squeezelite.log                        # no "unable to open output device"
launchctl list | grep -i squeeze                 # single com.user.squeezelite entry
osascript -e 'tell application "System Events" to get the name of every login item'
                                                  # no "Squeezelite"

Prevention

Update 2026-07-15: fix did not hold up on macOS Tahoe; MacBook-as-player abandoned

Reopened 2026-08-15 — lyrtui made Mac-native playback relevant again

Symptoms

• newly added subdomain resolves to public IP instead of 100.64.0.4
• dig @192.168.42.27 returns correct answer but system resolution is wrong
• resolvectl flush-caches does not fix it
• resolvectl status shows Current DNS Server as 1.1.1.1 instead of an AdGuard IP
• resolvectl status shows a per-link (e.g. ens18) DNS Servers list containing 1.1.1.1 even after resolved.conf.d is fixed

Reproductions

Root cause — Part 1: global config (found 2026-07-16)

DNS=192.168.42.27 192.168.42.89 1.1.1.1
Domains=~.

Root cause — Part 2: DHCP was also injecting 1.1.1.1 (found same day, after Part 1's fix was verified incomplete)

Link 2 (ens18)
       DNS Servers: 192.168.42.27 1.1.1.1
     Default Route: yes
    Current Scopes: DNS

Fix (fully applied and verified 2026-07-16)

[Resolve]
DNS=192.168.42.27 192.168.42.89
Domains=~.

sudo systemctl restart systemd-resolved

# /etc/netplan/00-installer-config.yaml
network:
  ethernets:
    ens18:
      dhcp4: true
      dhcp4-overrides:
        use-dns: false
      dhcp6: true
      dhcp6-overrides:
        use-dns: false
      match:
        macaddress: bc:24:11:dc:22:28
      set-name: ens18
  version: 2

sudo netplan apply

Sibling: same class on watchdog (NetworkManager/glibc, not systemd-resolved), fixed 2026-08-05

Diagnosis (for reference / if this or a similar bug resurfaces)

dig <fqdn> +short                    # returns public IP → wrong upstream, not necessarily "stale"
dig <fqdn> @192.168.42.27 +short     # returns 100.64.0.4 → AdGuard is correct
dig <fqdn> @127.0.0.53 +short        # returns public IP → stub resolver is using the wrong server
resolvectl status                    # check EVERY link's "DNS Servers" and "Current Scopes",
                                      # not just Global — a per-link list (e.g. from DHCP) can
                                      # override the Global resolved.conf.d config entirely

Why this note previously recommended `systemctl restart` as "the fix"

Symptoms

• tailscale status shows connected but SSH times out
• VS Code can't connect despite node appearing online
• tunnel is stale on connecting machine

Cause

Fix

# macOS
sudo tailscale down && sudo tailscale up
or restart via the menu bar icon

Symptoms

• onboard-container Phase 5 hangs at tailscale up
• tailscale up never returns via pct exec
• works fine in Proxmox console but not non-interactive

Cause

Fix

tailscale up \
  --login-server https://headscale.compellinglylowbrow.org \
  --authkey <key> \
  --accept-dns=false \
  --hostname <name> \
  --force-reauth \
  --accept-risk=lose-ssh

Symptoms

• SSH (or any other port) over a host's Headscale IP times out
• headscale nodes list shows the node connected with a current last_seen timestamp anyway
• The container/VM itself is otherwise completely healthy -- other services on it are up, LAN access works fine
• tailscale status on the host itself shows: Health check: - Tailscale could not connect to any relay server
• journalctl -u tailscaled shows repeating: wg: [...] - Failed to send handshake response: magicsock: network down

Symptom

Root Cause

Diagnosis

pct exec <vmid> -- systemctl is-active ssh <service> tailscaled   # confirm container itself is healthy
pct exec <vmid> -- tailscale status
Health check:
    - Tailscale could not connect to any relay server. Check your Internet connection.
pct exec <vmid> -- curl -s -o /dev/null -w '%{http_code}\n' https://1.1.1.1   # confirm real internet works
pct exec <vmid> -- journalctl -u tailscaled -n 40 --no-pager
wg: [...] - Failed to send handshake response: magicsock: network down  (repeating)

Fix

pct exec <vmid> -- systemctl restart tailscaled

Second occurrence (2026-08-11, pibox) — now auto-remediated

Third occurrence (2026-08-15, nastynas) — different node class again, now generalized fleet-wide

Generalized fleet-wide (2026-08-15, same session as the third occurrence)

Symptoms

• SSH to a node times out (not refused) despite the node showing 'online' in headscale nodes list and tailscale status
• sshd on the target is healthy and listening; LAN-IP SSH logins to the same host succeed fine in its own journal
• journalctl -u tailscaled shows repeated 'wg: [nodekey] - Failed to send handshake response: magicsock: network down'
• tailscale status --peers on the affected node itself shows 'Tailscale could not connect to any relay server. Check your Internet connection.'
• netmap: suggested exit node: no preferred DERP, try again later, repeating

Symptom

wg: [nodekey] - Failed to send handshake response: magicsock: network down
netmap: suggested exit node: no preferred DERP, try again later

# Health check:
    - Tailscale could not connect to any relay server. Check your Internet connection.

False leads ruled out along the way (useful to skip faster next time)

Fix

pct exec <vmid> -- systemctl restart tailscaled
sleep 5
pct exec <vmid> -- tailscale status   # health check line should be clean

Why this is worth knowing

Symptoms

• Immediately after reassigning the GDTRFB WiFi network from the default LAN to the new Trusted network (VLAN 10), devices connected to it could no longer load the UCG-Fiber's own Network app UI at https://192.168.42.1
• Ping to 192.168.42.1 worked fine from the same client the whole time -- this was not a full connectivity loss
• Symptom looked identical across three different access attempts, each with a different (wrong) prevailing theory at the time: WiFi client (blank/timeout), then a wired Ethernet client plugged directly into the gateway (also timed out), then Firefox specifically showed a browser-level timeout error
• curl -kv https://192.168.42.1/ from the wired client hung at 'Trying 192.168.42.1:443...' and timed out -- a real TCP-level failure, not a browser rendering/cache issue

Root cause

Two false leads chased before the real fix (both real, neither the cause)

Fix

Prevention

Symptom

Root cause -- confirmed

Device U7 Pro: Connecting
Device U7 Pro: Connected
...
mos@U7-Pro:~#

Fix

Prevention / notes for next time

Symptom

headscale /health: FAIL (HTTP 000 -- curl: (28) Connection timed out after 15002 milliseconds)
dns /dns-query: FAIL (HTTP 000 -- curl: (28) Connection timed out after 15002 milliseconds)
openssl s_client failed to run: ... timed out after 15 seconds

Root cause

Fix

Prevention / notes for next time

Symptom

Root cause

Fix

Prevention / notes for next time

Symptom

Root cause

Fix

Prevention / notes for next time

Symptom

Root cause

Fix

Prevention / notes for next time

Symptom

Root cause

Fix

Prevention / notes for next time

Symptoms

• A newly-created zone-based firewall rule has 0 packets/0 bytes on every counter no matter how much matching traffic is generated -- it's silently landed in the wrong zone-pair chain (e.g. LAN->LAN / same-zone, when the intent was Trusted->Servers)
• A rule intended to scope access to specific devices (by IP) instead grants the entire source zone unrestricted access -- matching_target ended up ANY instead of IP
• A device's hostname/label in the UniFi client list (ace.user collection) doesn't match who actually owns it -- confirmed via the device's own network settings screen, not by trusting the controller's label

Context

Trap 1: a fresh rule defaults into the wrong zone pair, not just a "toggle you forgot"

ssh ucg-fiber "mongo --port 27117 --quiet ace --eval 'db.firewall_policy.find({name:\"<rule name>\"}).forEach(printjson)'"

ssh ucg-fiber "iptables -L <chain> -n -v | grep <rule's ipset name>"

Trap 2: fixing the zone pair can silently switch scoping from IP to ANY

Trap 3: UniFi's own client label can be stale or simply wrong

Fix, fully verified (2026-08-24)

source.matching_target: IP, ips: [192.168.12.42, 192.168.12.240]
source.zone_id: Trusted
destination.matching_target: ANY
destination.zone_id: Servers
protocol: all, port: any
create_allow_respond: true

Prevention

Symptoms

• update-advisor --force --execute gives qui a SAFE verdict but skips it with 'script-type updates always require manual approval', even with auto_update not set to false and the release well past the 7-day age gate
• run-update qui (called directly) classifies and dispatches fine — only update-advisor's own gate is wrong

Cause

Fix

if "curl" in cmd and ("tar" in cmd or "dpkg" in cmd):
    return "binary"

Verify

runtee python3 bin/update-advisor --force --execute

Follow-up

Symptom

grafana: update_cmd ✓
grafana: post [Restart grafana and wait for HTTP to come up] ✓
grafana: verify [Grafana HTTP responding] ✓
grafana: ✓ update complete (v13.1.1 → v13.1.1)
grafana: execution OK [age gate bypassed]
grafana: SECURITY | executed: True

Root cause

Fix applied this incident

Prevention — for any future CVE/SECURITY-verdict auto-execute

Recurrence (2026-08-10) — and the generalized fix it prompted

Symptoms

• ntfy alert: Update FAILED: <service>
• update_cmd exited 100
• dpkg-preconfigure: unable to re-open stdin: No such file or directory
• E: Sub-process /usr/bin/dpkg returned an error code (1)

Symptom

update_cmd failed: update_cmd exited 100: dpkg-preconfigure: unable to
re-open stdin: No such file or directory E: Sub-process /usr/bin/dpkg
returned an error code (1)

Cause

Fix

noninteractive_cmd = f"export DEBIAN_FRONTEND=noninteractive; {update_cmd}"
rc_up, out_up, err_up = ssh(ssh_ip, noninteractive_cmd, ssh_user=ssh_user, timeout=300)

Immediate recovery (on the affected LXC, as root)

export DEBIAN_FRONTEND=noninteractive
dpkg --configure -a
apt-get update && apt-get upgrade -y <service>
systemctl restart <service>   # e.g. grafana-server

Related

Symptoms

• deleted monitor still in /metrics with last known status
• watchdog keeps alerting on deleted monitor
• monitor gone from get_monitors() but present in /metrics

Cause

Fix

ssh watchdog@192.168.42.229 "sudo docker restart uptime-kuma"

ssh watchdog@192.168.42.229 'bash -s' << 'EOF'
KEY=$(grep UPTIME_KUMA_API_KEY ~/homelab/.env | cut -d= -f2)
curl -s -u ":$KEY" http://192.168.42.229:3001/metrics | grep -i <monitor-name>
EOF

Note

Symptoms

• UI shows one monitor but /metrics returns two entries with same name
• one monitor UP, one DOWN, same name
• duplicate monitor_name in /metrics with different URLs

Cause

Diagnosis

ssh watchdog@192.168.42.229 'bash -s' << 'EOF'
KEY=$(grep UPTIME_KUMA_API_KEY ~/homelab/.env | cut -d= -f2)
curl -s -u ":$KEY" http://192.168.42.229:3001/metrics | grep monitor_status
EOF
Look for duplicate monitor_name values with different monitor_url values

Fix

cd ~/projects/homelab
python3 - << 'PYEOF'
import getpass
from uptime_kuma_api import UptimeKumaApi

password = getpass.getpass("Uptime Kuma password: ")
with UptimeKumaApi("http://192.168.42.229:3001") as api:
    api.login("mos", password)
    monitors = api.get_monitors()
    for m in monitors:
        print(f"ID={m['id']} name={m['name']!r} url={m.get('url','?')!r}")
    mid = int(input("Enter ID to delete: "))
    print(api.delete_monitor(mid))
PYEOF

Note

Symptoms

• version-checker shows uptime-kuma perpetually 'behind' against GitHub's /releases/latest, with update_cmd unable to close the gap
• compose.yaml image tag floating at louislam/uptime-kuma:1 only ever pulls new v1.x point releases, never v2

Cause

Why this can't go through `run-update`/`update-advisor` automation

Fix — migration procedure

   ssh watchdog
   docker run --rm --volume uptime-kuma-data:/app/data --volume $(pwd):/backup \
     busybox tar czf /backup/uptime-kuma-backup-v1-$(date +%F).tar.gz -C /app/data .
   # then copy the tarball off-box (developer-env or nastynas)
   

   docker compose stop uptime-kuma
   docker compose up -d uptime-kuma
   docker compose logs -f uptime-kuma
   

   docker inspect uptime-kuma --format '{{.State.Health.Status}}'   # expect: healthy
   docker exec uptime-kuma cat /app/package.json | grep '"version"'  # expect: "2.x.x"
   # /metrics endpoint auth (Basic, empty username, UPTIME_KUMA_API_KEY as password):
   curl -s -o /dev/null -w "%{http_code}\n" -u ":$UPTIME_KUMA_API_KEY" \
     http://192.168.42.229:3001/metrics                              # expect: 200
   

Tag-pinning decision

Result

Follow-up

Symptoms

• curl to vaultwarden FQDN hangs indefinitely
• curl direct to LXC IP:8000 returns blank
• Vaultwarden shows active/running and ss shows port 8000 bound
• Caddy logs show no error — backend silently non-responsive

Symptom

Root Cause

Fix

pct exec 108 -- sed -i "s|^ROCKET_TLS=.*||" /opt/vaultwarden/.env
pct exec 108 -- systemctl restart vaultwarden

ssh root@192.168.42.25 "curl -sI http://192.168.42.209:8000" | head -3
Expect: HTTP/1.1 200 OK

Note

Symptoms

• VS Code connects briefly then immediately reconnects
• SSH: developer-env shows reconnection spinner
• raw ssh developer-env works fine
• server log shows repeated 'The client has reconnected'

Causes (fix in order)

1. Shell startup output breaks VS Code's env probe

# Guard interactive-only output in ~/.bashrc on developer-env:
if [[ $- == *i* ]] && [[ -z "$VSCODE_INJECTION" ]]; then
    source ~/projects/homelab/start.sh
fi

2. Duplicate `Host developer-env` stanza in `~/.ssh/config` on the Mac

3. `Remote.SSH: Use Local Server` enabled

# Kill the server after changing settings
ssh developer-env 'pkill -f vscode'

Symptoms

• A known, already-diagnosed, self-resolving pattern (e.g. ddns-update — Freshness flapping on Porkbun 503 clustering, or CrowdSec bouncer-restart blips) still triggers ntfy priority-5 'Watchdog: manual intervention required' every single occurrence
• watchdog daemon log shows 'Triage: <a correct, specific, named diagnosis>' immediately followed by 'No playbook matched -- escalating' in the same alert cycle -- the diagnosis and the escalation decision visibly disagree
• A known-fixes writeup describes a triage/playbook fix as shipped and live, but the noisy paging continues unchanged after that date
• watchdog daemon.py 'Unknown check path' WARNING lines for paths like 'probes.caddy_lan_reachable.reachable' or 'headplane.running' alongside the escalation

Symptom

Two independent bugs stacked on top of each other

Bug 1 — the fix was never deployed (clone drift)

Bug 2 — even deployed, it was structurally disconnected (the real bug)

Fix

Related fix bundled in the same pass: mislabeled log timestamps

Diagnostic path (if a similar disagreement shows up again)

Symptom

Root cause

Fix

Prevention / notes for next time

Symptoms

• apt dist-upgrade / apt-get install takes 20-30+ minutes for a modest package count with no errors printed
• upgrade includes a kernel and/or large packages (build-essential, docker-ce) and just seems to hang
• nproc/free -h look fine but the operation still feels absurdly slow

Symptom

Diagnosis — split elapsed time into download phase vs. install phase

# Full download-vs-install timeline with real timestamps
sudo cat /var/log/apt/history.log | tail -60

apt's own term.log shows every individual unpack/setup step with timestamps
sudo grep -E '^(Log started|Log ended|Preparing to unpack|Unpacking|Setting up|Processing triggers)' /var/log/apt/term.log | tail -100

Rule out resource starvation while you're at it
nproc
free -h

Cause

Fix — switch to Ubuntu's geo-mirror redirector

sudo cat /etc/apt/sources.list.d/ubuntu.sources

sudo sed -i 's|http://us.archive.ubuntu.com/ubuntu/|mirror://mirrors.ubuntu.com/mirrors.txt|' /etc/apt/sources.list.d/ubuntu.sources

Verify

Secondary, unrelated finding from the same session

General takeaway — "is it actually stuck, or just slow?"

Related

Symptoms

• ntfy: 'Backup Freshness: ATTENTION' -- pibox/nastynas/wildwood all STALE, newest entry ~2.1d ago (expected within ~26h)
• Alert re-fires on consecutive nights even though backup-music.sh's own log shows every leg completing OK and the Uptime Kuma heartbeat firing successfully each run
• The nightly rsync legs DO transfer a handful of files each night (tens of MB) despite the alert -- but they're not new show folders

Symptom

Cause

Fix

# Verify consistency directly:
ssh watchdog@192.168.42.229 check-backup-freshness --top 5

Prevention / open follow-up

Symptoms

• GitHub Actions 'Validate Caddyfile' workflow fails on the 'Validate Caddyfile syntax' step with exit code 1
• First gap: job log shows Error: setting up default log: opening log writer using &logging.FileWriter{Filename:"/var/log/caddy/access.log"...}: mkdir /var/log/caddy: permission denied
• Second gap (only visible after fixing the first): job log shows Error: loading crowdsec app module: crowdsec: invalid configuration: crowdsec API key must not be empty
• Job annotations show only unrelated warnings (Node.js 20 deprecation notice, go.sum cache-restore miss) -- the actual error is buried in the step's raw log output, not surfaced as an annotation
• Workflow triggers on any push touching bin/** or caddy/Caddyfile (per its paths filter), so it can fire and fail on a commit that never touched the Caddyfile at all

Symptom

Gap 1: log directory permission denied

Error: setting up default log: opening log writer using
&logging.FileWriter{Filename:"/var/log/caddy/access.log", ...}:
mkdir /var/log/caddy: permission denied

- name: Create log directory to match production caddy LXC
  run: sudo mkdir -p /var/log/caddy && sudo chown "$(whoami)" /var/log/caddy

Gap 2: CrowdSec bouncer API key not stubbed

Error: loading crowdsec app module: crowdsec: invalid configuration:
crowdsec API key must not be empty

CROWDSEC_BOUNCER_API_KEY: dummy-crowdsec-key-for-ci-validation

Timeline and why both gaps went unnoticed for three days

Why this needed care to diagnose correctly

Lesson

Symptoms

• collect-homelab: watchdog/raspi4 clone is behind fires repeatedly across multiple days even after running the alert's own `git -C ~/homelab pull` fix each time
• SSHing in and checking confirms the clone IS clean and current right after pulling -- the notification just comes back a few runs later

Symptom

Diagnosis

watchdog: main @ 7ac934b, 0 local-only / 20 origin-only commits, clean
raspi4:   main @ 7ac934b, 0 local-only / 20 origin-only commits, clean

Fix -- auto-sync instead of manual-pull nagging