Incident Response

Incident Response

522 — Cloudflare can’t reach origin

Symptoms: Browser shows Cloudflare 522 error.

Diagnosis:

# 1. Is the container running?
docker ps --filter "name=level147-net"

# 2. Is Next.js responding inside the container?
docker exec level147-net node -e \
  "require('http').get('http://localhost:3000/', r => {console.log(r.statusCode); process.exit(0)}).on('error', e => {console.error(e.message); process.exit(1)})"

# 3. Can proxy-net reach the app?
docker run --rm --network proxy-net alpine:3.21 wget -qO- http://level147-net:3000/ | head -1

# 4. Cloudflared tunnel state
docker logs cloudflared --tail 20

Common causes:

Cause	Fix
Next.js HTTP server crashed (process alive via cron timer)	docker restart level147-net
Cloudflare Access app scoped to wrong domain	Zero Trust → Access → Applications — check domain
Tunnel ingress pointing to wrong service	Zero Trust → Networks → Tunnels → Public Hostnames — verify http://level147-net:3000
cloudflared disconnected	docker restart cloudflared

Container stuck in “starting” (healthcheck failing)

# Check what's failing
docker inspect level147-net | grep -A 20 Health

# Manual healthcheck test
docker exec level147-net node -e \
  "require('http').get('http://localhost:3000/api/health', r => process.exit(r.statusCode === 200 ? 0 : 1)).on('error', () => process.exit(1))"

If the app is responding but the healthcheck still fails, restart the container with --force-recreate:

docker compose -f /home/level147.net/docker-compose.yml \
  --env-file /home/level147.net/.env \
  up -d --force-recreate

Woodpecker deploy not reaching web VPS

The deploy step SSHes to 100.101.148.89 (web VPS Tailscale IP) from the tools VPS. If this fails with a timeout:

1. Check Tailscale status on tools VPS: tailscale status
2. Check SSH key is valid: echo $WEB_SSH_KEY | base64 -d | ssh-keygen -l -f /dev/stdin
3. Verify web VPS SSH allows the key: cat /root/.ssh/authorized_keys