Deployment

Deployment

Normal deploy (push to main)

git push origin main

Woodpecker CI triggers automatically:

1. checks step — lint + typecheck + tests must pass (gates everything below)
2. smoke step — builds the Next.js app and asserts the dashboard shell renders over HTTP
3. build-prod — builds the Docker image, pushes :prod and :<sha8> tags to the Gitea registry
4. deploy-prod — SCPs docker-compose.yml and rollback.sh to the web VPS, pulls :prod, recreates the container, and waits for the healthcheck to go green

Monitor progress in Woodpecker at http://100.81.122.65:8000 (Tailscale access required).

Manual deploy

In Woodpecker, navigate to the level147/level147.net repo and click Run pipeline to trigger the deploy pipeline manually.

Linter warnings

The deploy pipeline shows linter warnings about network_mode: host at the step level. These are cosmetic — the pipeline executes correctly. backend_options.docker.network_mode is not honoured by the installed Woodpecker agent version.

Verifying the deploy

# On web VPS
docker ps --filter "name=level147-net" --format "{{.Names}}\t{{.Status}}"
# Should show: level147-net    Up X minutes (healthy)

# Check the app responds
docker exec level147-net node -e \
  "require('http').get('http://localhost:3000/api/health', r => {console.log(r.statusCode); process.exit(0)}).on('error', e => {console.error(e.message); process.exit(1)})"
# Should print: 200

Rollback

Each build pushes two tags: :prod (mutable, always latest) and :<sha8> (immutable, one per commit). To roll back to a previous build without rebuilding:

# Find available SHA tags:
#   Gitea → level147/level147.net → Packages (or browse http://100.81.122.65:3000)

# Retag the previous SHA as :prod and push (run on tools VPS or any host with registry access)
docker login gitea.level147.net -u level147
docker pull gitea.level147.net/level147/level147.net:<sha8>
docker tag  gitea.level147.net/level147/level147.net:<sha8> \
            gitea.level147.net/level147/level147.net:prod
docker push gitea.level147.net/level147/level147.net:prod

Then trigger a manual deploy pipeline in Woodpecker — no rebuild needed, the image is already in the registry. Total time under 5 minutes.

If you need to revert the source too: git revert HEAD && git push origin main triggers a full rebuild.

Break-glass deploy (Gitea / Woodpecker / registry unavailable)

When the tools VPS (100.81.122.65) is unreachable and you need to ship a fix directly:

# 1. Run the quality gate locally first
npm run check

# 2. Build locally
docker build -t level147.net:recovery .

# 3. Save and stream to the web VPS over Tailscale SSH
docker save level147.net:recovery | ssh root@100.101.148.89 "docker load"

# 4. Tag as :prod and recreate the container
ssh root@100.101.148.89 "
  docker tag level147.net:recovery gitea.level147.net/level147/level147.net:prod &&
  docker compose -f /home/level147.net/docker-compose.yml \
    --env-file /home/level147.net/.env \
    up -d --force-recreate --wait --wait-timeout 60
"

This bypasses CI. Only use it when the normal pipeline is genuinely unavailable.

Secrets inventory

If /home/level147.net/.env on the web VPS or the Woodpecker secret store is lost, re-mint from:

Variable	Where to re-mint
AUTH_SECRET	Generate: openssl rand -base64 32
AUTH_TRUST_HOST	Set to true — not a secret
AUTHENTIK_ID	Authentik admin → Applications → level147-dashboard → OAuth2 provider → Client ID
AUTHENTIK_SECRET	Authentik admin → Applications → level147-dashboard → OAuth2 provider → regenerate secret
AUTHENTIK_ISSUER	Authentik admin → Applications → level147-dashboard → OAuth2 provider → OpenID Configuration URL (strip /.well-known/openid-configuration)
PORTAINER_URL	Config value (e.g. http://100.x.x.x:9000) — not a secret
PORTAINER_TOKEN	Portainer → Account (top-right) → Access tokens → Add token
TAILSCALE_CLIENT_ID / TAILSCALE_CLIENT_SECRET	Tailscale admin console → Settings → OAuth clients → Generate client
CLOUDFLARED_METRICS_URL	Config value (http://cloudflared:2019/metrics) — not a secret
NODE_EXPORTER_URL	Config value — not a secret
PROMETHEUS_URL / PROMETHEUS_NODE_JOB	Config values — not secrets
UPTIME_KUMA_URL / UPTIME_KUMA_SLUG	Config values — not secrets
AUTHENTIK_API_URL	Config value (e.g. http://100.81.122.65:9000) — not a secret
AUTHENTIK_API_TOKEN	Authentik admin → Directory → Tokens → Create token (Intent: API, under a read-only service account)
GITEA_TOKEN (CI secret)	Gitea → User settings → Applications → Generate token (write:packages scope)
WOODPECKER_TOKEN (CI secret)	Woodpecker → User settings → API token
WEB_SSH_KEY (CI secret)	Generate keypair: ssh-keygen -t ed25519; add public key to /root/.ssh/authorized_keys on web VPS; base64-encode private key: base64 -w0 id_ed25519
WEB_SSH_KNOWN_HOSTS (CI secret)	From tools VPS: ssh-keyscan 100.101.148.89 | base64 -w0

Monitoring agents

The security widget (fail2ban metrics) is fed by scripts/fail2ban-exporter.sh running on each monitored host. If a host is missing from the security widget after reinstall, deploy the exporter:

# On the target host (as root)
scp scripts/fail2ban-exporter.sh root@<host>:/usr/local/bin/fail2ban-exporter.sh
ssh root@<host> "chmod +x /usr/local/bin/fail2ban-exporter.sh && \
  echo '* * * * * root /usr/local/bin/fail2ban-exporter.sh' >> /etc/cron.d/fail2ban-exporter"

node_exporter must be running with --collector.textfile.directory=/var/lib/node_exporter/textfile_collector.