# Comprehensive Multi-Machine Validation Plan for StratoFusion Fly Operations

## Purpose

Validate that Fly.io multi-machine routing is safe for the operations that actually depend on shared active state:

- Fly-backed copy and move operations.
- Folder and batch transfers.
- Sync and backup runs that execute through Fly.
- Remote cancellation via PostgreSQL.
- Stale-operation cleanup via heartbeat timeout.

This plan also includes a small smoke set for non-Fly toolbar actions so the overall UX is checked, but it does **not** confuse those actions with multi-machine behavior.

---

## Runtime Inputs

Pass these into the agent at execution time instead of hardcoding them in this file:

- `APP_URL`.
- `APP_EMAIL`.
- `APP_PASSWORD`.
- `FLY_APP_NAME=stratofusion-rclone-prod`.
- `DATABASE_URL`.

Example values for the operator to provide at runtime:

- `APP_URL=https://stratofusion.io`.
- `FLY_APP_NAME=stratofusion-rclone-prod`.

Reason: this prompt should stay safe to commit. Credentials should not live in repo content.

---

## Current Architecture Reality Check

This prompt assumes the current repository state, not the older ZIP-era design.

- `active_operations` is the shared Fly state table.
- Relevant columns are.
  - `operation_id`.
  - `status`.
  - `type`.
  - `machine_id`.
  - `snapshot`.
  - `cancel_requested_at`.
  - `cancel_reason`.
  - `heartbeat_at`.
  - `started_at`.
  - `completed_at`.
  - `created_at`.
  - `updated_at`.
- Remote cancel is DB-mediated through `cancel_requested_at` and `cancel_reason`.
- ZIP downloads are removed from the live code path.
- `download_sessions` is no longer used by the application.
  - If the table still exists in production, treat that as schema drift, not active behavior.
- Browser-direct download is the primary path.
- Some Google and Dropbox download fallbacks still hit Fly stream endpoints, but they are stateless token flows, not DB-backed ZIP sessions.
- Production Fly config keeps one warm machine by default.
  - The stress portion of this plan must **manually** scale the app to 2 machines or more.

---

## What Actually Proves Multi-Machine Safety

High-value tests:

1. Start a Fly-backed operation on one machine and poll it successfully through the app.
2. Start enough concurrent Fly-backed work to ensure 2+ Fly machines are active.
3. Cancel a running operation and verify the owning machine honors the DB cancellation request.
4. Verify no stale `running` rows remain after the cleanup window.
5. Verify operations complete correctly across provider boundaries.

Low-value for multi-machine, but still useful as smoke tests:

- Search.
- Rename.
- Delete.
- Create folder.
- Refresh.
- Basic browser-direct download.

These mostly exercise Next.js and provider APIs, not shared Fly operation state.

---

## Safety Rules

1. Do not mutate real user data outside disposable test fixtures.
2. Create all test content under a dedicated root per provider/account:
   - `StratoFusion Multi-Machine Test/<timestamp>/...`.
3. Only delete files and folders created during this test run.
4. Prefer small and medium fixture files first.
5. Only run large-file and concurrency stress tests after the small-path tests pass.
6. Scale production back to one warm machine after the stress suite unless the operator explicitly wants to keep it higher.

---

## Connected Accounts Under Test

### Google Drive

- `rickster.dev@gmail.com` (Personal).
- `richard@stratofusion.tech` (Business).

### Dropbox

- An isolated Google personal-drive test identity supplied by the test owner.
- `rickster.dev@gmail.com` (Business).

### OneDrive

- `stratofusion002@outlook.com` (Personal).
- `richard@stratofusion.io` (Business).

---

## Required Tools

- Browser automation for UI flows.
- `flyctl` for machine status, scale, and logs.
- `psql` for DB verification.
- Optional: a second browser session or multiple tabs/windows for concurrency.

---

## Pre-Test Setup

### Step 1: Verify Fly state and machine count

```bash
fly status --app stratofusion-rclone-prod
fly machine list --app stratofusion-rclone-prod
```

Record:

- current machine count.
- machine IDs.
- current regions and states.

### Step 2: Scale up for cross-machine validation

If only one machine is running, scale to at least two before the concurrency suite:

```bash
fly scale count 2 --app stratofusion-rclone-prod
fly machine list --app stratofusion-rclone-prod
```

If `fly scale count` is not supported for the current Fly app shape, use the Fly dashboard or the appropriate machine clone/start command. The important condition is: **2 running machines at the same time**.

### Step 3: Start log monitoring

```bash
fly logs --app stratofusion-rclone-prod
```

Keep this running through the test.

### Step 4: Verify DB access and current schema

```bash
psql "$DATABASE_URL" -c "\d active_operations"
psql "$DATABASE_URL" -c "SELECT COUNT(*) AS stale_running_rows FROM active_operations WHERE status = 'running' AND heartbeat_at < NOW() - INTERVAL '10 minutes';"
```

Optional drift check:

```bash
psql "$DATABASE_URL" -c "SELECT to_regclass('public.download_sessions') AS legacy_download_sessions_table;"
```

Expected:

- `active_operations` exists.
- stale running row count is `0`.
- `download_sessions` may or may not exist, but it should not be required for any current test.

### Step 5: Create disposable fixtures

In each provider/account pair you plan to touch, create:

- one small text file.
- one medium binary or PDF-like file.
- one test folder with 5-10 files.
- one large-file candidate only if needed for cancellation/stress.

Use a shared naming scheme, for example:

- `StratoFusion Multi-Machine Test/2026-03-31/small-google-personal.txt`.
- `StratoFusion Multi-Machine Test/2026-03-31/folder-one/`.

### Step 6: Log into the production app

Use runtime-supplied credentials for:

- `APP_URL`.
- `APP_EMAIL`.
- `APP_PASSWORD`.

Verify that all expected connected accounts appear in the UI before testing operations.

---