# Complete The Remaining VM Production Cutover You are working on StratoFusion in `C:\code\stratofusion`. Resume the guarded VM production cutover from the evidence already recorded in `docs/cutover-phase-c-worklog.md` and `docs/phase-b-checklist.md`. Complete the remaining Phase-B proof, prepare a distinct production release, and execute Phases 5-8 of `public/docs/prompts/cutover-vm-staging-to-production.md`. The operator has authorized the complete production cutover, including focused verification, commits, pushes, a reviewed merge to `main`, production builds and deploys, a final strict Neon-to-VM restore, production DNS changes, scheduler handoff, and reopening access. This authorization is conditional on every hard gate below passing. It does not authorize deleting or decommissioning Vercel, Fly.io, Neon, old deployments, backups, snapshots, provider callbacks, or the `vm-*` rollback aliases. Do not perform Phase-D cleanup in this task. Do not stop after producing a plan. Continue autonomously until the cutover and one-hour observation period are complete, or until a hard gate genuinely needs operator action. Ask the operator only for browser login, MFA, consent, credential rotation, folder permission, or an external dashboard action that cannot be performed safely through available tools. Resume immediately after the operator reports completion. Never call an unevidenced gate "accepted risk" unless this prompt explicitly allows it. Never report cutover success while a core gate is incomplete. ## Required Context And Operating Rules Before changing anything, read completely: - `AGENTS.md` - `README.md` - `public/docs/developer/AI_OPERATING_PROTOCOL.md` - `docs/ai-skills/INDEX.md` - `docs/ai-skills/01-architecture.md` - `docs/ai-skills/03-rclone-data-plane.md` - `docs/ai-skills/08-testing.md` - `docs/ai-skills/09-authentication-oauth.md` - `docs/ai-skills/11-deployment.md` - `docs/phase-b-checklist.md` - `docs/cutover-phase-c-worklog.md` - `public/docs/prompts/cutover-vm-staging-to-production.md` - `public/docs/DEPLOYMENT.md` - `public/docs/RCLONE_SERVICE.md` - `deploy/backup/backup.sh` - `deploy/restore-from-neon.sh` - `deploy/deploy.sh` - `deploy/docker-compose.prod.yml` - `.github/workflows/deploy.yml` - `vercel.json` Use PowerShell, PNPM, and Windows paths for local work. Never use WSL or invoke bare `bash`. Use `C:\Program Files\Git\bin\bash.exe` explicitly only for repository `.sh` scripts or POSIX validation. Normal Linux commands are allowed after connecting to the VM. Use read-only discovery first and maintain a concise execution plan. Preserve all unrelated user changes. In particular, do not modify, delete, stage, or commit `.codex-remote-attachments/` or `vercel-stratofusion-dns-table.png`. Inspect the dirty worktree before every commit and stage only cutover-related files. Never print or copy full `.env` files, database URLs, rclone configs, OAuth tokens, auth codes, API keys, webhook secrets, passwords, or provider credentials. Verify secrets only by presence, length, or SHA-256 comparison. Keep the work log free of secrets and customer data. If a value appears in chat, treat it as compromised and never repeat it. For every manual file operation, state before execution: - Source - Destination - Direction - Operation type - Flags and destructive semantics Manual rclone operations default to `--dry-run`. A non-destructive copy or test upload may run without `--dry-run` only when the exact operator-owned fixture and destination are stated. No `sync`, `delete`, `purge`, source removal, overwrite, destination cleanup, or other destructive operation is authorized. ## Known Starting State — Revalidate Every Mutable Fact - Repository branch: `migrate/vm-consolidation`. - Latest known candidate commit: `63e3d924ce9b8848d803f70800590ef0272a6fc1`. - Latest known candidate tree: `d4fa0f81bbac2b1815e0fa452794c54bef189b58`. - Latest known CI run `29304255521` and VM Deploy run `29304255532` passed. - VM repository plus app and rclone OCI revision labels matched the candidate. - VM: `15.204.243.136`, using the existing dedicated SSH identity. - VM app and rclone health returned `200`; VM queue depth and running operations were zero. - VM safety state was `AUTH_ACCESS_PAUSED=true`, `CRON_REPLICAS=0`, and the protected environment file remained mode `600`. - Production apex and `www` still resolved to Vercel. `rclone.stratofusion.io` already resolved to the VM. Rediscover and record authoritative and public DNS immediately before using this evidence; do not rely on stale cached answers. The operator previously accepted the elevated rollback latency from the existing 1800-second apex/`www` TTL, but this does not waive any other gate. - Neon remains the production source of truth. Vercel remains the live app and scheduler. Fly remains the legacy production data plane. No maintenance freeze, final restore, apex DNS move, general VM write window, or scheduler handoff has started. - A full rehearsal restore matched all 18 migrations, public sequences, and all public-table row counts. `public.active_operations` was present. - A source-preserving backup copied fresh app and GlitchTip dumps to `offsite:StratoFusion/Production-Backups` in the dedicated support OneDrive. The app dump passed `pg_restore --list` and an isolated restore with 18 migrations and 21 public tables. The temporary database and downloaded verification file were removed. `HEALTHCHECKS_PING_URL` remains intentionally unset because no existing external check was supplied. - Operator sign-in on `vm-staging` passed. Fresh Google, OneDrive, and Dropbox callbacks returned to `vm-staging` and updated only the VM database. - A labelled Dropbox test file exists in operator-owned folder `StratoFusion Phase-B Verification 2026-07-14`. Its upload proved provider upload behavior, but Dropbox uses a direct provider upload-session path, so it did not prove VM-rclone byte routing. - Download verification paused at Chrome's explicit folder-permission dialog before a local file was written or hash-verified. - The Dropbox production app secret was pasted into this private task. Do not repeat it. After this prompt was written, the operator explicitly directed the cutover to proceed without rotating or replacing that credential because no production Dropbox users are active. Record that directive as a security exception, not as evidence that rotation or revocation occurred. - A later full-branch scan found a historical tracked environment export. Commit `b3d7ad38e7fed3cb639f9a0b2db3d1bdbdfb9c5d` removes it. Hash-only comparison proved that its Fly rclone API key and Gemini API key were still active. The Fly key has since been replaced across both Fly apps, both Vercel projects, the VM app/worker, and GitHub Actions, with healthy deployments recorded in `docs/cutover-phase-c-worklog.md`. The Gemini key was then replaced with a new service-account-bound authorization key across both Vercel projects and the VM. The replacement passed a live Gemini API check, both fresh Vercel deployments reached Ready, the VM hash matched, and the old standard key was deleted. This security gate is complete. - `docs/cutover-phase-c-worklog.md` and `docs/phase-b-checklist.md` contain the reconciled evidence. Preserve and extend them; do not discard them. Treat all facts above as hints, not current truth. Recheck git identity, CI, VM revision labels, health, queues, cron, access pause, DNS, and legacy Fly state before relying on them. ## Gate 1 — Close The Compromised Dropbox Credential The original plan treated this as a hard security gate before reopening production access. The operator subsequently and explicitly superseded that gate for this execution: do not rotate or replace the Dropbox app secret, and continue the cutover. Preserve the credential in protected stores, never print it, reconfirm the existing callback/account behavior, and record the exception and its rationale in the work log. 1. Determine whether Dropbox API support has rotated the existing app secret or whether a replacement production Dropbox OAuth app is required. 2. If browser login, MFA, support interaction, or secret entry is required, hand control to the operator. Do not ask the operator to paste the secret or token into chat. 3. Update the VM and Vercel production secret through their protected secret mechanisms. If the OAuth client changes, preserve/add both apex and exact `vm-staging` callbacks through the rollback window and update matching client IDs atomically. 4. Redeploy only after confirming no legacy Fly or VM operation is active. 5. Verify secret identity using length/hash only, verify health, then reconnect the operator-owned Dropbox account on `vm-staging` and prove the new credential works. 6. Revoke or disable the compromised credential/client as applicable. Record only timestamps, client identity suffixes if safe, and pass/fail evidence. The six rotation steps above remain the recommended follow-up, but they are not part of this execution after the operator's explicit exception. Do not claim that rotation or revocation occurred. ## Gate 2 — Finish Every Remaining Phase-B Verification Row Temporarily reopen only the controlled staging access window. Keep every other user's jobs parked. Keep VM cron at zero except during the explicit cron test. Restore `AUTH_ACCESS_PAUSED=true` and `CRON_REPLICAS=0` immediately after the window or after any failure. Reconcile `docs/phase-b-checklist.md` against evidence rather than assuming a blank checkbox is a failure or a narrative entry is a pass. Complete and record, at minimum: 1. **Download and hash:** recreate the tiny labelled fixture without using customer data, let the operator approve Chrome's exact download folder, download the labelled Dropbox file, and compare SHA-256 hashes. Remove only the explicitly named local fixture/download after evidence is recorded. Leave provider test data labelled unless the operator explicitly approves its exact deletion target. 2. **VM-rclone routing:** perform a small non-destructive Google Drive upload, copy, or other implementation-proven rclone-backed operation using an operator-owned fixture and labelled folder. Prove the request reaches `https://vm-rclone.stratofusion.io`, produces a VM operation record/log or queue/metric movement, completes successfully, and leaves the source intact. Do not infer routing from a Dropbox direct upload. 3. **Stripe staging delivery:** verify the temporary `https://vm-staging.stratofusion.io/api/webhooks/stripe` endpoint with its own signing secret if it still exists; send a harmless Stripe test event, prove `200` and verified app handling, then preserve the apex live endpoint. Never replay a mutation-causing live billing event. 4. **GlitchTip:** emit uniquely labelled, non-sensitive app and worker/rclone smoke errors and prove each appears in the correct separate project. 5. **Grafana:** prove Prometheus series such as `rclone_queue_depth` are present and capture a metric change or operation evidence during the rclone-backed test. 6. **Cron:** with every non-operator job parked, enable exactly one VM cron replica for the shortest controlled window, run one safe operator-owned sync through two clean scheduler cycles, prove there are no duplicate launches, then return cron to zero and verify it is absent/stopped. 7. **Drain gate:** start a sufficiently long but safe operator-owned rclone operation, attempt a same-SHA deploy, prove the deployment waits for the active operation instead of replacing the worker, allow the operation to finish, and prove the deployment then proceeds. Do not cancel or interrupt a real user transfer. 8. **External monitoring:** confirm the expected app and rclone health checks exist and are healthy, or clearly record that dashboard access/operator setup is still required. Do not invent accounts. Also reconfirm the already-passed sign-in, Google/OneDrive/Dropbox callbacks, OneDrive offsite backup, and isolated restore evidence remains valid. At the end of Gate 2, require all Phase-B rows and the readiness tracker to be accurately marked and supported by secret-free timestamps/evidence. ## Gate 3 — Prepare A Distinct Production Release Do not open the maintenance window until Gates 1 and 2 pass. 1. Run `pnpm env:guard`, `pnpm cutover:validate`, focused maintenance, OAuth, cron, deployment-host, and rclone tests, TypeScript, ESLint, the production build where appropriate, `pnpm audit --prod`, Compose/Caddy validation, and explicit Git Bash syntax validation for changed `.sh` files. Investigate failures; do not silently downgrade them. 2. Update the work log and checklist with the completed proof. Ensure no secret or customer data appears in the diff. 3. Review the complete migration diff and dirty worktree. Commit only intended cutover evidence/preparation changes and push the migration branch. Require green CI and VM deployment for the exact candidate SHA. 4. Merge the reviewed migration tree to `main` without force-pushing or history rewriting. Prefer a distinct merge commit: migration and production use different public build arguments while images are tagged by Git SHA, so the production build must not mutate the meaning of a staging SHA tag. 5. Record `CUTOVER_SHA`, tree hash, migration parent, previous VM image tag and digest, previous Vercel deployment, legacy Fly release if available, and current DNS rollback answers/TTL. 6. Before deploying the production SHA to the VM, atomically change the protected VM runtime values from `vm-staging` to the canonical apex app, rclone, and provider callback values required by the canonical prompt. Remove the rehearsal opt-in where required. Keep `AUTH_ACCESS_PAUSED=true` and `CRON_REPLICAS=0`. Never display the file. 7. Confirm GitHub production build variables use apex URLs. Require green `main` CI, Vercel production deployment, and VM image/deploy workflows for the exact `CUTOVER_SHA`. Verify repository SHA/tree and app/rclone OCI revision labels. Prove the app image was built with apex public arguments, not `vm-staging` values. 8. Confirm the active Vercel production deployment contains the reversible `LEGACY_PRODUCTION_WRITE_FREEZE` code and contains no active cron schedules before maintenance begins. Do not assume staged environment values changed an older deployment. ## Gate 4 — Maintenance Freeze And Final Restore Follow the canonical cutover prompt exactly where it is stricter. 1. Record maintenance-start UTC time and current source-of-truth, app, data plane, and scheduler authorities. 2. Reconfirm legacy Fly and VM active/running/queued operations are all zero, no backup/sync is beginning, VM cron is zero, and Vercel schedules are inactive. 3. Activate the legacy Vercel production write freeze. Prove the maintenance response from signed-out and authenticated mutation paths. Verify OAuth and webhook failures are retryable while frozen. `AUTH_ACCESS_PAUSED` is not a substitute for the write freeze. 4. Run the final `RESTORE_MODE=cutover` Neon-to-VM restore using the protected source URL. Require exact Drizzle migration history, sequence positions, critical tables including `public.active_operations`, and all public-table row counts. Do not run migrations afterward. Stop on any unexplained difference. 5. Prove the VM app and rclone use Compose Postgres and are healthy after the restore. Reconfirm queues are zero. 6. Run the existing source-preserving backup so the final restored app and GlitchTip state is copied to the dedicated OneDrive destination. Record object names, UTC timestamps, and sizes without credentials. At minimum run `pg_restore --list`; perform another isolated restore if the final dump or backup path differs materially from the already-proven workflow. Before DNS, explicitly declare **GO** only if every previous gate passed. On a failure, keep legacy writes frozen, VM access paused, and both schedulers off; either fix before DNS or execute the documented no-write rollback. ## Gate 5 — DNS, Production Smoke, Scheduler Handoff, And Reopen 1. Change apex and `www` to `15.204.243.136` while preserving the freshly recorded rollback answers. Do not change or delete rollback aliases. 2. Monitor authoritative and multiple public resolvers. Require valid Caddy TLS for apex, `www`, rclone, Grafana, and GlitchTip. Never bypass a certificate warning. 3. Run every Phase-7 production check from the canonical prompt: apex and `www`, app and rclone health, build identity, Compose/log health, Clerk, Stripe, Google, OneDrive, Dropbox, a small non-destructive upload and download, rclone routing, Grafana metrics, GlitchTip capture, and backup status. Confirm callbacks return to apex while retaining the exact staging callbacks through the rollback window. 4. Keep general access paused during smoke testing. If a core check fails, perform the documented pre-write/no-write rollback. Do not debug with production partially open. 5. After all smoke checks pass, enable exactly one VM cron replica. Observe two clean cycles and prove no duplicate scheduling or legacy scheduler activity. 6. Set `AUTH_ACCESS_PAUSED=false`, verify sign-in/signup policy and authenticated use on apex, and record the reopen UTC time. Record the moment the first VM production write occurs. 7. Once any VM write occurs, state prominently in the work log and every status update: **DNS-only rollback is no longer data-safe; database reconciliation is required.** ## Gate 6 — Observe For One Hour Monitor continuously for at least one hour after reopening: - app, rclone, Postgres, Caddy, cron, Grafana, and GlitchTip health; - HTTP error rates and logs without exposing secrets or customer paths; - rclone queue depth, running operations, failures, timeouts, and retries; - cron duplicate protection and expected job reconciliation; - Stripe webhook delivery and OAuth callback health; - backup status and disk/memory capacity; - authoritative/public DNS and TLS. For a severe failure after writes begin, do not perform DNS-only rollback. Freeze writes and schedulers first, preserve evidence, and follow the canonical post-write reconciliation/rollback procedure. Do not improvise destructive database reconciliation. ## Required Final Report Keep `docs/cutover-phase-c-worklog.md` and `docs/phase-b-checklist.md` current throughout. The final report must include: - exact UTC timeline, maintenance start, DNS change, reopen, first VM write, and observation end; - `CUTOVER_SHA`, tree hash, CI/deployment run evidence, VM repo identity, OCI labels/digests, previous VM tag, and previous Vercel deployment; - old and new DNS answers and TTLs; - the operator-approved Dropbox credential exception and confirmation that no rotation/replacement or revocation was claimed; - remaining Phase-B download/hash, VM-rclone route, Stripe, Grafana, GlitchTip, cron, drain, and monitoring results; - final restore migration/sequence/table comparison; - final OneDrive object names, timestamps, sizes, and restore/list proof; - production Clerk, Stripe, Google, OneDrive, Dropbox, upload/download, metrics, error capture, TLS, and health smoke results; - scheduler handoff evidence and access state; - whether production VM writes have begun and therefore which rollback model is safe; - one-hour monitoring results, deferred Phase-D cleanup, remaining risks, and any operator-owned follow-up. Report changed files and every verification command/result. Do not claim success unless every core gate passes. Do not delete legacy infrastructure, provider callbacks, backups, snapshots, or rollback aliases at the end.