Compare commits

...

2 Commits

Author SHA1 Message Date
Nico
a0514f0823 docs: switch active deployment runbooks from dokploy to ssh compose
Some checks failed
Build & Deploy Fiddy (SSH Compose) / build (push) Failing after 1s
Build & Deploy Fiddy (SSH Compose) / deploy (push) Has been skipped
2026-02-22 01:51:44 -08:00
Nico
8495fe5f1f ci: rename ssh workflow and enforce web/scheduler runtime checks 2026-02-22 01:51:31 -08:00
4 changed files with 79 additions and 41 deletions

View File

@ -85,14 +85,45 @@ jobs:
ssh "${{ secrets.DEPLOY_USER }}@${{ secrets.DEPLOY_HOST }}" \ ssh "${{ secrets.DEPLOY_USER }}@${{ secrets.DEPLOY_HOST }}" \
"cd '$DEPLOY_PATH' && IMAGE_TAG='$IMAGE_TAG' docker compose pull && IMAGE_TAG='$IMAGE_TAG' docker compose up -d --remove-orphans && docker image prune -f" "cd '$DEPLOY_PATH' && IMAGE_TAG='$IMAGE_TAG' docker compose pull && IMAGE_TAG='$IMAGE_TAG' docker compose up -d --remove-orphans && docker image prune -f"
- name: Verify Web and Scheduler Are Running
run: |
set -euo pipefail
ssh "${{ secrets.DEPLOY_USER }}@${{ secrets.DEPLOY_HOST }}" "DEPLOY_PATH='$DEPLOY_PATH' bash -s" << 'EOF'
set -euo pipefail
cd "$DEPLOY_PATH"
web_id="$(docker compose ps -q web)"
scheduler_id="$(docker compose ps -q scheduler)"
if [ -z "$web_id" ]; then
echo "web service container not found"
exit 1
fi
if [ -z "$scheduler_id" ]; then
echo "scheduler service container not found"
exit 1
fi
web_running="$(docker inspect -f '{{.State.Running}}' "$web_id")"
scheduler_running="$(docker inspect -f '{{.State.Running}}' "$scheduler_id")"
if [ "$web_running" != "true" ]; then
echo "web service is not running"
exit 1
fi
if [ "$scheduler_running" != "true" ]; then
echo "scheduler service is not running"
exit 1
fi
EOF
- name: Wait for Ready Health Check - name: Wait for Ready Health Check
env: env:
HEALTH_URL: ${{ secrets.DOKPLOY_HEALTHCHECK_URL }} HEALTH_URL: ${{ secrets.DEPLOY_HEALTHCHECK_URL }}
MAX_ATTEMPTS: "30" MAX_ATTEMPTS: "30"
SLEEP_SECONDS: "10" SLEEP_SECONDS: "10"
run: | run: |
if [ -z "$HEALTH_URL" ]; then if [ -z "$HEALTH_URL" ]; then
echo "Missing DOKPLOY_HEALTHCHECK_URL secret" echo "Missing DEPLOY_HEALTHCHECK_URL secret"
exit 1 exit 1
fi fi
bash scripts/wait-for-health.sh bash scripts/wait-for-health.sh

View File

@ -8,19 +8,23 @@
- [ ] Postgres port `5432` is not public. - [ ] Postgres port `5432` is not public.
## B) App and Deployment ## B) App and Deployment
- [ ] Dokploy project connected to Gitea repo. - [ ] SSH deployment host is prepared (`/opt/fiddy`, Docker Engine, Compose plugin).
- [ ] Secrets configured: - [ ] Deploy host runtime env is configured in `/opt/fiddy/.env`:
- [ ] `DATABASE_URL` - [ ] `DATABASE_URL`
- [ ] `DATABASE_SSL` - [ ] `DATABASE_SSL`
- [ ] `ALLOWED_DB_NAMES` - [ ] `ALLOWED_DB_NAMES`
- [ ] `SESSION_COOKIE_NAME` - [ ] `SESSION_COOKIE_NAME`
- [ ] `SESSION_TTL_DAYS` - [ ] `SESSION_TTL_DAYS`
- [ ] `DEBUG_API=0` - [ ] `DEBUG_API`
- [ ] `DOKPLOY_DEPLOY_HOOK` - [ ] Gitea Actions secrets configured:
- [ ] `DOKPLOY_SCHEDULER_DEPLOY_HOOK` - [ ] `REGISTRY_USER`
- [ ] `DOKPLOY_HEALTHCHECK_URL` - [ ] `REGISTRY_PASS`
- [ ] `DEPLOY_KEY`
- [ ] `DEPLOY_HOST`
- [ ] `DEPLOY_USER`
- [ ] `DEPLOY_HEALTHCHECK_URL`
- [ ] Deploy workflow passes build/test/push/deploy. - [ ] Deploy workflow passes build/test/push/deploy.
- [ ] Scheduler deploy workflow step passes. - [ ] Deploy guard confirms `web` and `scheduler` are running.
- [ ] Post-deploy health gate passes (`scripts/wait-for-health.sh`). - [ ] Post-deploy health gate passes (`scripts/wait-for-health.sh`).
- [ ] Manual smoke passes (`scripts/smoke-public-launch.sh`). - [ ] Manual smoke passes (`scripts/smoke-public-launch.sh`).
@ -47,6 +51,6 @@
- [ ] Measured RTO is acceptable. - [ ] Measured RTO is acceptable.
## F) Rollback Readiness ## F) Rollback Readiness
- [ ] Previous stable release retained in Dokploy. - [ ] Previous stable image tags retained in registry (for rollback).
- [ ] Rollback runbook tested once in staging or low-risk window. - [ ] Rollback runbook tested once in staging or low-risk window (SSH Compose deploy by older image tag).
- [ ] Rollback smoke check verified. - [ ] Rollback smoke check verified.

View File

@ -25,28 +25,30 @@ Use these in execution updates for fast scanning:
- [ ] `docs/08_NGINX_PROXY_MANAGER_SETUP.md` - [ ] `docs/08_NGINX_PROXY_MANAGER_SETUP.md`
- [ ] `docs/06_SECURITY_REVIEW.md` - [ ] `docs/06_SECURITY_REVIEW.md`
## Phase 1: Registry + Dokploy Wiring (Operator Needed) ## Phase 1: Registry + SSH Compose Wiring (Operator Needed)
First-time reference: `docs/11_DOKPLOY_FIRST_TIME_WALKTHROUGH.md`.
Hands-on checkpoints: Hands-on checkpoints:
1. Create/verify secrets in Gitea: 1. Create/verify secrets in Gitea:
- `REGISTRY_USER` - `REGISTRY_USER`
- `REGISTRY_PASS` - `REGISTRY_PASS`
- `DOKPLOY_DEPLOY_HOOK` - `DEPLOY_KEY`
- `DOKPLOY_SCHEDULER_DEPLOY_HOOK` - `DEPLOY_HOST`
- `DOKPLOY_HEALTHCHECK_URL` - `DEPLOY_USER`
2. In Dokploy app settings (Web): - `DEPLOY_HEALTHCHECK_URL`
- image source points to `git.nicosaya.com/nalalangan/fiddy/web` 2. Prepare deploy host for SSH Compose:
- health endpoint is `/api/health/ready` - install Docker Engine + Compose plugin
- release history retention is enabled - create `/opt/fiddy/.env` with production variables
3. In Dokploy app settings (Scheduler): - run `docker login git.nicosaya.com` as deploy user
- image source points to `git.nicosaya.com/nalalangan/fiddy/scheduler` 3. Confirm production compose contract:
- no public port exposed - web image source: `git.nicosaya.com/nalalangan/fiddy/web`
- env vars set: `DATABASE_URL`, `DATABASE_SSL`, `ALLOWED_DB_NAMES` - scheduler image source: `git.nicosaya.com/nalalangan/fiddy/scheduler`
- web publishes `3010:3000`
- scheduler has no public port
Validation: Validation:
- [ ] Push-to-main triggers `.gitea/workflows/deploy-dokploy.yml` - [ ] Push-to-main triggers `.gitea/workflows/deploy-ssh-compose.yml`
- [ ] Web and Scheduler deploy hooks fire successfully - [ ] SSH deploy updates both web and scheduler containers
- [ ] Deploy guard confirms web and scheduler are running
- [ ] Health gate completes via `scripts/wait-for-health.sh` - [ ] Health gate completes via `scripts/wait-for-health.sh`
## Phase 2: NPM Edge Setup (Operator Needed) ## Phase 2: NPM Edge Setup (Operator Needed)

View File

@ -1,4 +1,4 @@
# Public Launch Runbook (Self-Hosted + Dokploy) # Public Launch Runbook (Self-Hosted + SSH Compose)
## 1) Goals ## 1) Goals
- Deploy Fiddy publicly without stack rewrite. - Deploy Fiddy publicly without stack rewrite.
@ -6,14 +6,14 @@
- Enable fast rollback and basic operational visibility. - Enable fast rollback and basic operational visibility.
- Keep security baseline enforceable for direct home-IP exposure. - Keep security baseline enforceable for direct home-IP exposure.
## 2) Deploy Control Plane (Dokploy) ## 2) Deploy Host (SSH Compose)
1. Install Dokploy on your Proxmox Docker host. 1. Prepare Linux deploy host with Docker Engine + Compose plugin.
2. Add project in Dokploy and connect Gitea repository. 2. Ensure deploy target directory exists (`/opt/fiddy`).
3. Configure web image source: `git.nicosaya.com/nalalangan/fiddy/web`. 3. Configure web image source: `git.nicosaya.com/nalalangan/fiddy/web`.
4. Configure scheduler image source: `git.nicosaya.com/nalalangan/fiddy/scheduler`. 4. Configure scheduler image source: `git.nicosaya.com/nalalangan/fiddy/scheduler`.
4. Deploy by immutable tag (`github.sha`) and keep `main` as convenience tag. 5. Deploy by immutable tag (`github.sha`) and keep `main` as convenience tag.
5. Configure health check endpoint: `/api/health/ready`. 6. Configure health check endpoint: `/api/health/ready`.
6. Keep previous releases for rollback and verify rollback button path. 7. Keep previous image tags for rollback.
### Required secrets/variables ### Required secrets/variables
- `DATABASE_URL` - `DATABASE_URL`
@ -26,15 +26,16 @@
- `SCHEDULER_BATCH_SIZE` (scheduler app, optional) - `SCHEDULER_BATCH_SIZE` (scheduler app, optional)
## 3) CI/CD (Gitea Actions) ## 3) CI/CD (Gitea Actions)
- Use `.gitea/workflows/deploy-dokploy.yml`. - Use `.gitea/workflows/deploy-ssh-compose.yml`.
- Required secrets: - Required secrets:
- `REGISTRY_USER` - `REGISTRY_USER`
- `REGISTRY_PASS` - `REGISTRY_PASS`
- `DOKPLOY_DEPLOY_HOOK` - `DEPLOY_KEY`
- `DOKPLOY_SCHEDULER_DEPLOY_HOOK` - `DEPLOY_HOST`
- `DOKPLOY_HEALTHCHECK_URL` - `DEPLOY_USER`
- `DEPLOY_HEALTHCHECK_URL`
- Health gate: - Health gate:
- workflow calls `scripts/wait-for-health.sh` against `DOKPLOY_HEALTHCHECK_URL` - workflow calls `scripts/wait-for-health.sh` against `DEPLOY_HEALTHCHECK_URL`
- default retry window: 5 minutes (30 attempts x 10s) - default retry window: 5 minutes (30 attempts x 10s)
## 4) Reverse Proxy + Network Hardening ## 4) Reverse Proxy + Network Hardening
@ -100,12 +101,12 @@
1. Identify failing request and `request_id`. 1. Identify failing request and `request_id`.
2. Correlate application logs (Loki) by `request_id`. 2. Correlate application logs (Loki) by `request_id`.
3. Check `/api/health/ready` status and DB connectivity. 3. Check `/api/health/ready` status and DB connectivity.
4. Roll back to previous known-good Dokploy release if needed. 4. Roll back to previous known-good image tag via SSH Compose if needed.
5. Capture root cause and update this runbook/checklist. 5. Capture root cause and update this runbook/checklist.
## 8) Rollback Checklist ## 8) Rollback Checklist
1. Select previous healthy image in Dokploy release history. 1. Select previous healthy image tag for both `web` and `scheduler`.
2. Trigger rollback and wait for deployment completion. 2. Trigger rollback deploy and wait for completion.
3. Run `scripts/smoke-public-launch.sh https://your-domain`. 3. Run `scripts/smoke-public-launch.sh https://your-domain`.
4. Verify error-rate drop in Grafana/Loki and confirm no DB migration mismatch. 4. Verify error-rate drop in Grafana/Loki and confirm no DB migration mismatch.
5. Log the rolled back version, timestamp, and reason. 5. Log the rolled back version, timestamp, and reason.