StratoFusion uses local Windows development, optional local-VM rehearsal, CI
validation, and authoritative production. The Phase-C cutover is complete:
OVHcloud hosts the production application, PostgreSQL database, rclone data
plane, scheduler, observability, and backup source.
Vercel production, Neon production, and both Fly rclone apps are legacy
rollback resources. They are not safe standalone production rollback targets
after the first VM write; any rollback requires explicit database
reconciliation. The former shared-development deployment is retired from
repository runtime policy and must not be replaced by another permanent shared
development target on the production VM. Its remaining external resources are
shutdown candidates tracked in the
shared-development retirement runbook.
All Fly Machines are intentionally stopped.
Environment Matrix
Environment
Git source
App runtime
Primary origin
Database
Rclone runtime
Scheduler
Local Windows
working tree
local Node.js
http://localhost:3000
stratofusion-local
http://127.0.0.1:3001
explicit local replay only
Local VM rehearsal
exact reviewed revision
local Compose stack
https://local-vm.stratofusion.io through the Windows hosts file
local Compose PostgreSQL
https://local-vm-rclone.stratofusion.io
disabled by default
CI validation
pushed revision
ephemeral GitHub runner
none
deterministic fixtures/mocks
deterministic worker tests
none
Temporary staging
explicitly approved revision
approved migration/rehearsal target only
exact approved origin
isolated or restored snapshot per runbook
exact approved worker
disabled unless separately approved
Production
main
OVHcloud VM Compose (app)
https://stratofusion.io
VM Compose PostgreSQL
https://rclone.stratofusion.io
exactly one VM cron replica
The Git dev branch is not a deployment environment. CI validates pull
requests plus long-lived main and dev pushes, but only successful
push-triggered main CI drives the automatic VM deploy workflow.
For API request-integrity checks, preserve Caddy's public Host header and
its sanitized X-Forwarded-Proto scheme (consumed by Next.js). The app compares
browser origins using that Host and request scheme, not the standalone
container's listening address. Do not rewrite Host to app:3000 or enable
arbitrary forwarded-host trust to work around origin failures.
System Audit Logs
System Logs are available at /admin/system-logs with admin and recent MFA
verification. The exact legacy /dev/system-logs URL redirects there; all
other development surfaces retain their production restrictions. These are
application audit logs; use private VM administration for Caddy logs.
Private VM Administration
The September 2026 security baseline uses Next.js and eslint-config-next
15.5.24, Sharp 0.35.4, and worker Morgan 1.12.0. Both production dependency
audits remain required release gates; install from the checked-in lockfiles.
The VM remains a headless Ubuntu server. Operators use VS Code Remote SSH over
the existing key-only deploy account, with Tailscale as the private management
network and optional Cockpit access behind Tailscale Serve. Cockpit is a host
package bound to loopback; it is not part of the application Compose stack and
must not receive a public Caddy hostname or firewall rule.
Public SSH remains available until the GitHub Actions deploy job is migrated to
an independently verified private path. Do not restrict port 22 merely because
interactive operator access works over Tailscale.
Use Private VM Administration for setup,
least-privilege authentication, verification, and rollback. The host helper is
deploy/vm-admin-access.sh.
When production SSH credentials are available only to GitHub Actions, use the
manual CI admin bootstrap. It inspects
the production host before installing Tailscale and registering a separate
operator public key, while preserving existing deployment access.
When inspection finds an authentication mismatch, its separately confirmed
harden-ssh action applies key-only OpenSSH settings with validation,
rollback on apply failure, and a fresh CI key-authenticated connection check.
Read-Only Infrastructure Dashboard Configuration
The admin infrastructure overview queries Prometheus from the Next.js server
over the private Compose network. Production uses
PROMETHEUS_BASE_URL=http://prometheus:9090; this variable is server-only and
must never use a NEXT_PUBLIC_* name. Local Windows development may leave it
unset, in which case the dashboard renders an explicit metrics-unavailable
state while application and local rclone health remain independently visible.
The optional INFRASTRUCTURE_NODE_NAME,
INFRASTRUCTURE_NODE_ENVIRONMENT, INFRASTRUCTURE_NODE_PROVIDER, and
INFRASTRUCTURE_NODE_REGION values are non-secret display labels. Leave an
unknown value blank rather than inserting a hostname, public IP, provider
service ID, or guessed region. GIT_COMMIT_SHA remains the release source in
the production image. Grafana and GlitchTip links are emitted only when their
configured HTTPS origins are within the stratofusion.io domain family; the
private Cockpit tailnet URL is never part of the response.
Compose passes GRAFANA_ROOT_URL and GLITCHTIP_DOMAIN explicitly to the app,
sharing the same defaults as Grafana and GlitchTip. Missing or blank .env
values use the configured production ingress origins
(https://grafana.stratofusion.io and https://glitchtip.stratofusion.io).
The local-VM Compose override sets matching local origins on the app as well
as the tools. Outside Compose, the resolver still requires explicit values.
Environment changes require recreating the app container; restarting it does
not reload .env. Apply tracked Compose changes through the normal exact-SHA
deployment workflow.
After production deployment, GitHub Actions verifies the app container's release
SHA, allowlisted Grafana/GlitchTip runtime destinations, application/database
readiness, and tool HTTP reachability using deploy/verify-management-links.mjs.
The read-only probes have bounded timeouts and retries; they report no credentials.
Private database-tool configuration is reported separately without probing the
operator's tunnel. A final runner-side probe checks public application readiness.
The dashboard associates these destinations with their Compose service rows.
Application, rclone, Caddy, Weaviate, and backup rows use fixed same-origin
operations pages. Prometheus and node-exporter use the allowlisted Grafana
origin, while GlitchTip web and worker use the allowlisted GlitchTip origin.
Rejected or absent external URLs render as not configured. Link configuration
does not grant the application any credentials or mutation capability.
Optional Private Database Tools
pgadmin and redisinsight live behind the optional Compose management
profile. Production leaves COMPOSE_PROFILES blank by default. A local VM may
set COMPOSE_PROFILES=management after replacing the pgAdmin password and
RedisInsight encryption-key placeholders in its protected .env file. The
services use immutable image digests, dedicated persistent volumes, and host
bindings 127.0.0.1:${PGADMIN_HOST_PORT:-5050} and
127.0.0.1:${REDISINSIGHT_HOST_PORT:-5540}. They are not Caddy upstreams and
must not receive public DNS records or firewall rules.
PGADMIN_MANAGEMENT_URL and REDISINSIGHT_MANAGEMENT_URL control whether the
corresponding dashboard row has an action. The server accepts only loopback
HTTP (for SSH forwarding) or private *.ts.net HTTPS origins; leave either
blank when its private access path is not configured. These variables are
server-only and must never use a NEXT_PUBLIC_* prefix. See
Private VM Administration for operator
setup and verification.
This configuration adds no Docker socket, SSH key, OVHcloud credential,
Tailscale credential, or host-command capability to the app container.
The Compose stack adds two isolated operational telemetry services. The
no-network operations-collector performs a fixed read of Docker container
state and writes only allowlisted aggregate counts to operations_telemetry.
The unprivileged operations-exporter has no Docker socket, reads that volume
plus sanitized numeric backup state, and is reachable only by Prometheus on the
internal telemetry_scrape network. The app remains on the existing
server-only Prometheus boundary.
INFRASTRUCTURE_BACKUP_FRESHNESS_HOURS defaults to 36 and may be set from 1
through 168 hours. The nightly backup writes atomic numeric state files for its
latest attempt, latest success, and current/final result; it never exports dump
names, paths, destinations, credentials, or rclone output. A missing first-run
state is unavailable, a failed run or an old success is degraded, and a current
run remains healthy only when a prior successful backup is still fresh.
Production deploy.sh now waits for an exporter health response and a non-empty
sanitized Compose snapshot before succeeding. It then requires exactly one
running cron replica. The local-VM override remains exempt from that deploy
gate because its scheduler is deliberately scaled to zero; its dashboard will
show the scheduler invariant as degraded, making the safety difference
explicit.
Historical VM Naming And Cutover Procedure
The procedure below records the completed rehearsal and cutover model. Do not
repeat its Phase-B or Phase-C DNS steps during normal operation. Current
production state and rollback constraints are recorded in
docs/cutover-phase-c-worklog.md.
Use three distinct hostname sets so local rehearsal, public VM rehearsal, and
production cutover cannot be confused:
Stage
App
Rclone
Grafana
GlitchTip
Local-only VM rehearsal
local-vm.stratofusion.io
local-vm-rclone.stratofusion.io
local-vm-grafana.stratofusion.io
local-vm-glitchtip.stratofusion.io
OVH public rehearsal
vm-staging.stratofusion.io
vm-rclone.stratofusion.io
vm-grafana.stratofusion.io
vm-glitchtip.stratofusion.io
Final production after cutover
stratofusion.io, www.stratofusion.io
rclone.stratofusion.io
grafana.stratofusion.io
glitchtip.stratofusion.io
Required OVH rehearsal DNS records in the Vercel-hosted zone:
Record
Type
Value
vm-staging
A
15.204.243.136
vm-rclone
A
15.204.243.136
vm-grafana
A
15.204.243.136
vm-glitchtip
A
15.204.243.136
Do not change apex, www, or the final production service records during
Phase B. If temporary legacy records such as staging.stratofusion.io still
point at the VM, remove them only after the vm-* records are verified and no
current rehearsal depends on them.
Local-VM releases use the manually dispatched
.github/workflows/deploy-local-vm.yml workflow. It builds both images with
the local public URLs, publishes only commit-SHA image coordinates with
provenance and SBOM attestations, and never runs the production SSH deploy.
Its build uses only the dedicated repository variables
LOCAL_VM_NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY and
LOCAL_VM_NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY; both must be test-mode
pk_test_... values. The workflow fails before building instead of falling
back to production public variables. One immutable SHA image can therefore be
deployed to multiple local-VM guests that share the same local/test public
origin contract.
After the build succeeds, an operator checks out the exact SHA on the private
guest, runs the guarded bootstrap-local-vm-token-encryption.sh command, and
then runs TAG=<sha> ./deploy.sh. The bootstrap requires the local Compose
override and DEPLOY_ENV=local-vm, generates the active key only on the guest,
preserves the legacy local key as decrypt-only material, and refuses production
configuration. See docs/local-vm-rehearsal.md for the complete setup and
verification sequence.
At runtime, the passed TAG is authoritative for both the selected images and
the GIT_COMMIT_SHA exposed by the app and worker. An older value left in an
untracked guest .env cannot replace the release SHA shown by the
infrastructure dashboard.
External callback and webhook dashboards must be updated for the OVH rehearsal
before OAuth or billing tests:
Clerk production instance: add https://vm-staging.stratofusion.io.
Google OAuth production client: add https://vm-staging.stratofusion.io/api/google.
Microsoft Entra production app: add https://vm-staging.stratofusion.io/api/onedrive.
Dropbox production app: add https://vm-staging.stratofusion.io/api/dropbox.
Stripe live dashboard: add a temporary webhook endpoint at https://vm-staging.stratofusion.io/api/webhooks/stripe if webhook validation is part of the rehearsal.
Rollback during Phase B is DNS-safe: leave production DNS untouched, restore
the previous VM image tag with TAG=<previous-good-sha> ./deploy.sh, or remove
the vm-* DNS records if the rehearsal must be abandoned. Keep cron disabled
and access paused while investigating.
Phase C cutover checklist:
Explicit operator approval for cutover.
Prove every Phase-B checklist item, backup, drain gate, and rollback path.
Deploy a SHA-tagged release built with apex production URLs and keep VM cron at zero.
Disable Vercel schedules and enable the Vercel-only
LEGACY_PRODUCTION_WRITE_FREEZE gate. Verify it returns retryable 503
responses for UI/API writes, OAuth callbacks, cron, and webhooks while
leaving GET/HEAD /api/health available.
Drain Fly and VM rclone operations, then run the final restore with
RESTORE_MODE=cutover. Do not run migrations after the restore.
Repoint apex, www, rclone, grafana, and glitchtip DNS records to the VM.
Smoke test app/rclone health, OAuth callbacks, Stripe webhooks, upload and
download, observability, backups, image labels, and compose database use.
Enable exactly one compose cron replica only after legacy schedules are
proven inactive and the production smoke checks pass.
Reopen access only after the no-write validation window. Keep temporary
vm-* aliases and provider callbacks through the initial rollback window.
Offsite database backup
The production backup container creates timestamped custom-format dumps of the
application and GlitchTip databases under /backups, then uses source-preserving
rclone copy to BACKUP_RCLONE_REMOTE. An offsite OneDrive remote uses a
dedicated single-tenant Entra client and is supplied through the protected VM
RCLONE_CONFIG_OFFSITE_* variables: TYPE, CLIENT_ID, CLIENT_SECRET,
TOKEN, DRIVE_ID, DRIVE_TYPE, tenant-specific AUTH_URL/TOKEN_URL, and
optional REGION. Complete OAuth interactively and set ACCESS_SCOPES to
rclone's read/write profile without SharePoint-site browsing. Never commit or
print those values. For a controlled cutover proof, set
BACKUP_PRUNE_LOCAL=false so the run cannot remove older local dumps. Do not use
sync, remote cleanup, or source-removal flags.
The final VM runtime uses apex URLs, an empty OAUTH_REHEARSAL_ORIGIN,
CRON_EXTRA_ALLOWED_HOSTS, and CRON_REPLICAS=0 until scheduler handoff.
deploy/deploy.sh requires an immutable git-SHA tag; latest is never a
deployable rollback reference. Both production images carry the SHA in the
org.opencontainers.image.revision OCI label, and the deploy script refuses
to roll the stack unless both pulled labels match the requested SHA. Production
Compose references pin third-party images by manifest digest. The rclone image
also validates the official archive SHA-256 before installation.
Routine releases must not bypass the rclone drain gate. When an exact active
operation has explicit cancellation approval, use the manually dispatched
Cancel Production Rclone Operation workflow from main, enter the lowercase
UUID twice, and retain its sanitized pre/post evidence. The immutable script is
executed with the production deploy directory explicitly selected, even though
the script itself is extracted to a temporary file. After the operation
leaves the active set, rerun the failed deploy job for the same immutable SHA;
do not rebuild under a different release coordinate merely to retry cutover.
All external GitHub Actions are pinned to reviewed 40-character commits and
CI dependency installs use frozen lockfiles. Dependabot proposes weekly npm,
GitHub Actions, and Docker updates; those proposals still require the normal
review and complete test/build gate before merge. A digest or action pin must
never be refreshed without reviewing the corresponding upstream release.
Source Of Truth
Use these in order of authority:
local development: .env.local.
local-VM rehearsal: the untracked local VM deploy/.env assembled from the
production example plus deploy/local-vm.env.example overrides.
production deployment: /opt/stratofusion/deploy/.env and the immutable
main image built after successful push-triggered main CI.
approved temporary staging: the explicit rehearsal/migration runbook and its
isolated configuration for that one window.
Repository env templates are reference material only. Retired Vercel or Neon
snapshots must not be treated as an active development source of truth.
AUTH_ACCESS_PAUSED controls the public /sign-in and /sign-up migration
guard at runtime. Leave it unset or set it to true to show the pause notice;
set it to false to render the normal Clerk sign-in and sign-up entry points.
Restart the local Next.js server or redeploy the authoritative VM application
after changing this value for the intended environment.
LEGACY_PRODUCTION_WRITE_FREEZE is a separate Phase-C safety gate. It can
activate only when its value is exactly true on a Vercel production runtime
whose configured site is the apex or www. The VM cannot activate it. Keep it
off during ordinary operation and use it only after transfers are drained.
Environment Ownership Inventory
Each environment has separate application, billing, database, provider OAuth,
and rclone ownership. Keep secrets, full OAuth client IDs, and full account
keys out of public documentation; use this table to verify that a deployment is
pointing at the intended environment family.
Environment
Application target
Database
Clerk
Stripe
Provider OAuth callbacks
Rclone
Local
http://localhost:3000 from .env.local
stratofusion-local
local test instance
Stripe sandbox through the CLI listener
localhost callbacks
local service at http://127.0.0.1:3001
Local VM
local hosts-file origin
local Compose PostgreSQL
local/test instance
sandbox through a local forwarder
exact local-VM callbacks in separate test provider apps
local VM gateway
Production
https://stratofusion.io on the OVHcloud VM
VM Compose PostgreSQL
live instance
Stripe live
apex callbacks
VM gateway at https://rclone.stratofusion.io
Approved rehearsal
exact, time-bounded origin only
isolated database or approved snapshot
explicitly selected instance
explicitly selected mode
exact approved callbacks
exact approved worker
The root and Fly rclone environment templates should use the same section order
for OAuth, Clerk, Stripe, Neon, and Fly settings. Template drift is a deployment
risk because it makes cross-environment comparison harder during OAuth,
billing, database, and rclone incidents.
Multi-Environment Clerk OAuth
Clerk social sign-in and direct provider OAuth are environment-sensitive.
Local and local-VM use test resources; production uses live resources. Phase B
is a controlled exception that temporarily reuses production resources from
the exact staging origin.
Required rules:
Local uses its own Clerk test instance and only localhost redirect URIs.
Local VM uses the local/test Clerk instance and callbacks registered in
separate test provider apps for the exact local-VM origin.
Production uses its own Clerk live instance and only stratofusion.io redirect URIs.
Phase B requires OAUTH_REHEARSAL_ORIGIN=https://vm-staging.stratofusion.io,
matching runtime/build-time app URLs, and exact Google, OneDrive, and Dropbox
redirect URIs on that origin.
Arbitrary previews, wildcard origins, and other *.stratofusion.io hosts are
not valid OAuth rehearsal environments.
The environment-specific Google and Microsoft apps may be reused by both direct provider OAuth and the matching Clerk social connection, but they must contain both redirect URIs for that environment.
the StratoFusion callback route such as /api/google or /api/onedrive.
the Clerk callback URI copied from the matching Clerk instance.
The application now validates the following at startup:
NEXT_PUBLIC_SITE_URL.
NEXT_PUBLIC_APP_URL.
OAUTH_REHEARSAL_ORIGIN when Phase B is enabled.
NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY.
CLERK_SECRET_KEY.
GOOGLE_CLIENT_ID and GOOGLE_REDIRECT_URI.
ONEDRIVE_CLIENT_ID and ONEDRIVE_REDIRECT_URI.
DROPBOX_CLIENT_ID and DROPBOX_REDIRECT_URI.
Direct-provider callback state is HMAC signed. Configure a unique 32+ byte
OAUTH_STATE_SECRET per environment when practical. If it is blank, the
server falls back to TOKEN_ENCRYPTION_KEY and then CLERK_SECRET_KEY, so no
production configuration change is required for the code to fail closed.
Rotating the active signing secret invalidates only in-flight OAuth state,
which expires after ten minutes.
Provider tokens require a dedicated TOKEN_ENCRYPTION_KEY of at least 32
bytes and a stable TOKEN_ENCRYPTION_KEY_ID. The application does not derive
this key from Clerk or session secrets. New records use an authenticated
enc:v2:<key-id> envelope. During rotation, put earlier key IDs and secrets in
the server-only TOKEN_ENCRYPTION_PREVIOUS_KEYS JSON object, set a new active
key and ID, deploy, then run pnpm exec tsx scripts/casa/rotate-token-encryption.ts first in its default dry-run mode.
Database writes require both --apply and
TOKEN_ROTATION_APPLY_CONFIRM=reencrypt-user-service-tokens. Retain prior keys
until a second dry run reports zero candidate rows; never put key values in
logs or captured evidence.
Before the first release that enforces the dedicated key, dispatch
Bootstrap Production Token Encryption from main and enter
bootstrap-production-token-encryption exactly. The workflow runs under the
production deployment lock, generates the active key on the VM, retains the
existing Clerk-derived key only as legacy-v1 decrypt-only material, updates
/opt/stratofusion/deploy/.env atomically, and never sends key values through
GitHub Actions output. It is idempotent for an already-valid dedicated key and
fails closed for partial configuration or a local-VM Compose override.
After the matching release deploys, dispatch Audit Production Token Encryption from main and enter audit-production-token-encryption exactly.
The workflow verifies that the running app image has the selected release SHA,
then scans token envelopes inside a repeatable-read, read-only database
transaction. Its JSON result contains only the active key ID and aggregate
scanned, candidateRows, and rewrittenRows counts; rewrittenRows is
always zero. It cannot invoke the write-enabled rotation path. Treat a nonzero
candidate count as migration planning input, not authorization to rewrite
production data.
Expected runtime mapping:
Environment
Site/app URL
Rehearsal opt-in
Clerk key mode
Google redirect
OneDrive redirect
Dropbox redirect
Local
http://localhost:3000
unset
test
http://localhost:3000/api/google
http://localhost:3000/api/onedrive
http://localhost:3000/api/dropbox
Local VM
https://local-vm.stratofusion.io
exact local-VM origin
test
https://local-vm.stratofusion.io/api/google
https://local-vm.stratofusion.io/api/onedrive
https://local-vm.stratofusion.io/api/dropbox
Phase-B rehearsal
https://vm-staging.stratofusion.io
exact staging origin
live
https://vm-staging.stratofusion.io/api/google
https://vm-staging.stratofusion.io/api/onedrive
https://vm-staging.stratofusion.io/api/dropbox
Production
https://stratofusion.io
unset
live
https://stratofusion.io/api/google
https://stratofusion.io/api/onedrive
https://stratofusion.io/api/dropbox
The generic runtime environment classifier may continue to report Phase B as
unknown. The OAuth-specific validator reports staging only when the exact
rehearsal opt-in and all matching URLs are present, while resource-safety
validation still requires the production database name, live Stripe mode, the
configured production rclone host, and a strong cron secret.
The production cutover release intentionally contains no crons entry in
vercel.json. A production Vercel deployment of that
release must be active and its dashboard schedules must be absent before the
compose cron container can scale above zero. Editing the file alone is not
proof of scheduler handoff.
The runtime cron guard allows these jobs only on the stable custom domains:
https://stratofusion.io.
https://www.stratofusion.io.
Preview deployments and unknown hosts still return a skipped 200 response even if the cron route is called manually.
Local development uses scripts/local-cron.ps1 for replay and debugging instead of Vercel cron. Local replay requires ALLOW_LOCAL_CRON=true in .env.local, a local request host such as localhost or 127.0.0.1, and the usual CRON_SECRET bearer token. Restart pnpm dev after changing .env.local so the Next.js server sees the flag.
The AI reconciliation schedule may remain installed while AI indexing is
disabled. After cron authentication and production-host validation, the
reconcile-ai-indexing route returns a successful skipped response when
AI_INDEXING_ENABLED is not true and does not inspect provider accounts.
Legacy Fly.io rclone Service
Use Git Bash on Windows. Do not use WSL.
The OVHcloud VM is the authoritative production rclone runtime. Both legacy Fly
apps are stopped. The production app remains an approval-gated recovery input;
the former development app is a retirement candidate. Do not probe either
public endpoint while stopped because older deployed configuration may still
allow a request to wake a Machine.
The active recovery script supports only fly-rclone/fly.prod.toml from
main. fly-rclone/fly.toml is retained as external-deletion evidence and is
not a deploy target.
Retained rollback policy:
Retained configurations record auto_start=false, auto_stop="stop", and
min_machines_running=0.
The Fly deployment workflow is manual-only and requires an explicit paid
legacy-capacity confirmation plus a rollback/recovery reason.
Active operation polling and remote cancellation requests now dual-write to PostgreSQL inside fly-rclone.
ZIP download endpoints have been removed from both Next.js and fly-rclone.
Retained stopped Machines cannot be started by Fly Proxy; an operator must
intentionally start or redeploy them for an approved recovery.
Queue metrics are now exposed on an internal Fly metrics port so queue depth can be scraped without publishing /metrics on the public app service.
A separate fly-autoscaler app can be added later if backlog-based scaling is needed.
Manually recover the Prod Fly App
cd fly-rclone
./deploy.sh --env prod
Do not create a burst pool during normal VM operation. For an explicitly
approved Fly recovery only:
cd fly-rclone
./deploy.sh --env prod --machine-pool-size 5
Do not run those public checks while the legacy apps are intentionally stopped.
Local metrics verification:
cd fly-rclone
curl http://localhost:9091/metrics
/health now returns 503 with status: "unhealthy" when Fly service-to-service authentication is missing. A successful health check therefore verifies that FLYIO_RCLONE_API_KEY is configured on the Fly app.
Required per-app secrets:
NODE_ENV.
LOG_LEVEL.
FLYIO_RCLONE_API_KEY so the Next.js app can authenticate to protected Fly routes.
DATABASE_URL for shared Fly active operation state.
Configured per-app provider secrets:
GOOGLE_CLIENT_ID.
GOOGLE_CLIENT_SECRET.
ONEDRIVE_CLIENT_ID.
ONEDRIVE_CLIENT_SECRET.
DROPBOX_CLIENT_ID.
DROPBOX_CLIENT_SECRET.
Optional per-app secrets:
SENTRY_DSN.
SENTRY_ENVIRONMENT.
DATABASE_URL.
Current note:
The VM rclone container is authoritative; separate Fly apps are stopped
legacy resources.
The Next.js app generates the rclone config and sends it to the VM rclone
service per request, so provider secrets are not a separate data-plane
credential source today.
Local/test resources and production/live resources remain separated.
The Fly rclone image pins pnpm@9.15.0 and installs a pinned upstream rclone release with retry logic so deployment does not drift onto older distro packages that lack newer flags such as --name-transform.
Release Order
AI Search OCR
OCR for AI indexing is disabled by default and is controlled separately from the
main AI search and indexing rollout flags.
Variable
Default
Purpose
AI_SEARCH_OCR_ENABLED
false
Enables OCR routing for standalone PNG/JPEG AI indexing inputs.
AI_SEARCH_OCR_PROVIDER
tesseract
Selects the OCR provider. Phase 6A implements only local Tesseract.js.
AI_SEARCH_OCR_TIMEOUT_MS
60000
Bounds OCR work per extraction attempt.
AI_SEARCH_OCR_MIN_CONFIDENCE
0
Skips OCR output below this provider confidence when confidence is available.
AI_SEARCH_OCR_MAX_PAGES
10
Reserved page cap for future PDF OCR; PDF rendering is not implemented in Phase 6A.
AI_SEARCH_OCR_MAX_IMAGES
20
Bounds image inputs per OCR extraction attempt.
The implemented local provider uses tesseract.js 7.x and WebAssembly. It does
not require system Tesseract binaries, WSL, or Fly.io rclone changes. PDF page
OCR is intentionally deferred because pdfjs-dist page rendering in Node uses
@napi-rs/canvas, which would add a native dependency and needs separate
Windows, Vercel, and Fly.io validation before scanned PDFs can be enabled.
Tesseract.js may fetch/cache language data on first use unless deployment
pre-seeds or configures that data path; validate this behavior manually before
enabling OCR outside internal environments.
When promoting environment-sensitive changes, keep the verification order strict:
Run local checks and CI for the exact candidate SHA.
Build the candidate with Local VM Release, deploy it through the local
VM's drain-gated deploy.sh, and verify the local Compose/TLS stack.
For an approved rehearsal window, manually dispatch VM Deploy with
target=staging and the same exact release_sha. The workflow builds a
distinct <sha>-staging app image and rclone image, starts isolated app and
worker containers, and routes only vm-staging.stratofusion.io to them.
Worker control and signed browser-stream paths use that same rehearsal
origin, so the shared production worker never needs to admit the Stage
origin. Database-backed worker operation persistence and Stage worker error
reporting are disabled for this read-only rehearsal. The workflow verifies
that both production container IDs remain unchanged and restores the
canonical Caddy route automatically when candidate deployment fails.
Keep staging validation read-only because the temporary candidate uses the
authoritative VM's existing backing services. Do not launch transfers,
billing writes, OAuth reconnects, migrations, or scheduled work from it.
Dispatch target=cleanup-staging after the rehearsal to restore the
canonical production-only route and remove both candidate containers. The
worker is removed only after its active-operation drain gate reaches zero.
Cleanup fails unless production remains healthy and the public staging
endpoint is unavailable (a transport failure, HTTP 404, or HTTP 421). HTTP
5xx is a failure because it can indicate that the staging alias still
targets a removed container. Rollback and cleanup
read the canonical Caddy configuration from the exact candidate SHA without
resetting the production checkout, so a pre-merge rehearsal cannot restore
stale production routing.
Merge the reviewed release to main; successful push-triggered CI then
promotes that exact SHA through the normal production workflow.
Do not skip directly to production for OAuth, billing, database, or rclone changes.
Production Workflow
Use PowerShell for Git and GitHub operations on Windows. Do not use WSL.
Merge the reviewed release to main.
Confirm CI Checks succeeds for the main push. Its completed run triggers
VM Deploy; failed, cancelled, manual, and pull-request CI runs cannot enter
the privileged build job.
Confirm VM Deploy checks out workflow_run.head_sha, uses that exact SHA
for both image tags and OCI revision labels, and passes it to the VM as
TAG=<sha>. The deploy workflow emits maximum provenance plus image SBOMs
and remains non-cancelling.
Verify /opt/stratofusion on the VM and both OCI revision labels equal the
CI-tested commit SHA.
Verify application, rclone, PostgreSQL, public TLS, queue state, backups,
observability, and exactly one cron replica.
Do not deploy stratofusion-prod on Vercel as part of normal production
release work.
Manual workflow_dispatch requires an explicit target. target=production
remains the recovery path and deploys the supplied exact release_sha (or the
selected ref's github.sha). target=staging and target=cleanup-staging are
time-bounded rehearsal operations and never replace the production app
container. Production recovery follows the same exact-SHA checkout, image tag,
OCI label, and VM TAG path without requiring an upstream CI event.
Local And CI Workflow
Develop on Windows with the local app, local PostgreSQL, local rclone worker,
local Stripe listener, and explicit local cron replay. Use the documented
local-VM stack when full Compose/TLS rehearsal is useful. Pull requests and
long-lived branch pushes receive CI validation; a branch is not a deployment
environment. A public staging target may exist only for a separately approved,
time-bounded migration or rehearsal and must be removed afterward.
Domain Notes
stratofusion.io, www.stratofusion.io, rclone.stratofusion.io,
grafana.stratofusion.io, and glitchtip.stratofusion.io point to the
OVHcloud VM.
The retired dev.stratofusion.io assignment is external cleanup state, not
an application environment. Follow the retirement runbook; do not repoint it
to the production VM or another permanent shared deployment.
DNS for vm-staging.stratofusion.io may remain pointed at the self-hosted VM
for a controlled Phase-B rehearsal, but the canonical Caddy configuration
does not serve it. The deployment workflow adds the route only for an
explicitly approved window and cleanup must make the endpoint unavailable.
The domain zone is delegated to Vercel nameservers.
Preview Deployments
Automatic Vercel preview deployments are disabled. Use local app and
Storybook review instead.
Do not use arbitrary *.vercel.app previews for OAuth validation, Clerk validation, or provider callback testing.
Provider redirect URIs and popup return flows are intentionally pinned to
localhost and stratofusion.io, plus the exact local-VM or
vm-staging.stratofusion.io rehearsal origin only while
OAUTH_REHEARSAL_ORIGIN=https://vm-staging.stratofusion.io is set for Phase B.
The rehearsal opt-in accepts no wildcard, alternate subdomain, path, port, or
non-HTTPS value.
OAuth routes treat the direct Host/request URL as authoritative and use
X-Forwarded-Host only as a same-authority consistency check. Conflicting
hosts or non-default ports are rejected before provider or Clerk work.
OAuth callback returns use the validated configured site origin rather than
forwarded host or protocol values.
Local Storybook review remains useful because it does not depend on the full OAuth callback chain.
Verification Checklist
VM repository HEAD and app/rclone OCI labels equal the intended main SHA.
VM application, rclone, PostgreSQL, observability, backups, and one cron
replica are healthy.
Local .env.local points to stratofusion-local.
https://stratofusion.io and the production service hostnames resolve to the
OVHcloud VM.
The retired shared-development host is rejected by OAuth, CORS, and cron
policy. Its external domain state matches the approved stage of the
retirement runbook (410 Gone or no DNS after removal).
Both Fly apps report all Machines stopped; do not probe their public health
endpoints during normal operation.
Stable-host cron routes return 200 when called with CRON_SECRET.
Preview or unknown-host cron calls return a skipped 200.
Local and local-VM testing use test Clerk, provider, and Stripe resources.
Production uses live Clerk and live Stripe.
Phase B uses the exact staging origin for both NEXT_PUBLIC_SITE_URL and
NEXT_PUBLIC_APP_URL, and for the Google, OneDrive, and Dropbox callbacks.
Clerk, Google, Microsoft, Dropbox, the Next.js runtime, and the rclone CORS
policy all agree on the exact rehearsal origin before OAuth testing starts.
Arbitrary preview hosts remain OAuth-disabled.
OAUTH_REHEARSAL_ORIGIN is absent from normal production and the apex
app/callback URLs are authoritative.
next.config.js and vercel.json security headers are aligned before release.
Public docs are available under /docs; all /dev, /api/dev,
/api/debug, debug, demo, and test surfaces return a non-cacheable 404 on
every non-loopback host.
pnpm audit --prod reports no known production vulnerabilities before CASA submission.