## Test Suites

## Suite 1: Non-Destructive Smoke Checks

These are useful baseline checks, but they are **not** evidence of Fly multi-machine correctness.

### 1.1 Search smoke test

Steps:

1. Search for a known test term across all services.
2. Confirm results appear from multiple connected accounts.

Expected:

- results render correctly.
- no obvious UI or provider errors.
- no requirement for `active_operations` evidence.

### 1.2 Rename smoke test

Steps:

1. Rename a disposable test file.
2. Refresh and verify the new name persists.

Expected:

- rename succeeds.
- no Fly-specific DB row required.

### 1.3 Create folder and delete smoke test

Steps:

1. Create a disposable folder.
2. Delete only that disposable folder or file.

Expected:

- operation succeeds.
- do not treat lack of `active_operations` evidence as a failure.

---

## Suite 2: Core Fly-Backed Transfer Operations

These are the primary multi-machine tests.

For every test in this suite, collect:

```sql
SELECT
  operation_id,
  status,
  type,
  machine_id,
  cancel_requested_at,
  cancel_reason,
  heartbeat_at,
  started_at,
  completed_at
FROM active_operations
ORDER BY created_at DESC
LIMIT 10;
```

### 2.1 Single-file copy across accounts on the same provider

Suggested path:

- Google Drive Personal -> Google Drive Business.

Expected:

- copy starts successfully.
- destination file appears.
- source file remains.
- recent `active_operations` row shows.
  - `type` aligned with copy behavior.
  - terminal `status` of `completed`.
  - non-null `machine_id`.

### 2.2 Single-file copy across providers

Run at least these combinations:

1. Google Drive Personal -> Dropbox Personal
2. Dropbox Business -> OneDrive Personal
3. OneDrive Business -> Google Drive Business

Expected:

- each transfer completes successfully.
- file size and name are correct at destination.
- DB row is present with terminal success.

### 2.3 Folder copy across providers

Run one recursive folder copy with 5-10 files.

Expected:

- folder structure is preserved.
- progress updates appear during transfer.
- destination contents match source fixture.
- recent `snapshot` JSON reflects the operation details.

Useful query:

```sql
SELECT
  operation_id,
  status,
  type,
  machine_id,
  snapshot
FROM active_operations
ORDER BY created_at DESC
LIMIT 5;
```

### 2.4 Move within the same provider

Use a disposable file only.

Suggested path:

- Dropbox Personal -> different folder in the same account.

Expected:

- source entry disappears.
- destination entry appears.
- no duplicate remains behind.
- DB row reaches terminal success.

### 2.5 Batch copy or multi-select copy

Use 5-10 files in a disposable folder.

Expected:

- operation progresses cleanly.
- no silent partial failures.
- no duplicate or missing files at destination.

---

## Suite 3: Sync and Backup Runs

These are important because they are long-running and operationally sensitive.

### 3.1 One-time sync

Suggested path:

- Google Drive Personal -> Dropbox Personal.

Expected:

- sync job starts and completes.
- destination matches source fixture.
- recent `active_operations` row shows `type` of `sync` or `bisync` as appropriate.

### 3.2 Scheduled sync smoke validation

Create or use a disposable scheduled sync job, then trigger it manually if the UI supports that.

Expected:

- job runs.
- `sync_jobs.last_run_at` updates.
- `next_run_at` is populated when scheduling applies.

Query:

```sql
SELECT
  id,
  status,
  schedule,
  last_run_at,
  next_run_at
FROM sync_jobs
ORDER BY created_at DESC
LIMIT 10;
```

### 3.3 Backup run

Create or use a disposable backup path and run it once.

Expected:

- backup executes successfully.
- `backup_jobs.last_run_at` updates.
- Fly operation state remains consistent throughout execution.

Query:

```sql
SELECT
  id,
  status,
  schedule,
  last_run_at,
  next_run_at
FROM backup_jobs
ORDER BY created_at DESC
LIMIT 10;
```

---

## Suite 4: Cross-Machine Stress and Cancellation

This suite is the actual horizontal-scaling proof.

### 4.1 Confirm 2+ running Fly machines

```bash
fly machine list --app stratofusion-rclone-prod
```

Expected:

- at least two machines in a running state.

### 4.2 Launch concurrent Fly-backed operations

Open multiple tabs or sessions and start at least 4 Fly-backed operations close together:

1. cross-provider copy
2. same-provider copy
3. folder copy
4. sync or backup run

Expected:

- all operations start successfully.
- logs show activity from multiple machine IDs.
- DB shows recent operations spread across 2+ `machine_id` values.

Query:

```sql
SELECT
  machine_id,
  COUNT(*) AS operation_count
FROM active_operations
WHERE created_at > NOW() - INTERVAL '15 minutes'
GROUP BY machine_id
ORDER BY operation_count DESC;
```

Success condition:

- more than one non-null `machine_id` appears during the stress window.

### 4.3 Remote cancellation

Start one large enough folder copy or backup that it remains active for at least 15-30 seconds, then cancel it from the UI.

Expected:

- cancellation request is accepted by the UI.
- `cancel_requested_at` becomes non-null in DB.
- the operation reaches a terminal non-running state quickly.
- no indefinite `running` row remains.

Query:

```sql
SELECT
  operation_id,
  status,
  machine_id,
  cancel_requested_at,
  cancel_reason,
  heartbeat_at,
  completed_at,
  snapshot
FROM active_operations
WHERE cancel_requested_at IS NOT NULL
ORDER BY created_at DESC
LIMIT 10;
```

Success condition:

- `cancel_reason` reflects user cancellation semantics.
- final status is no longer `running`, `started`, `pending`, or `retrying`.

### 4.4 Heartbeat cleanup check

Do **not** intentionally kill random production machines unless the operator explicitly approves it.

Safer production validation:

1. After stress and cancellation tests, wait long enough to exceed the stale threshold for any abandoned row.
2. Query for stale active rows.

Query:

```sql
SELECT
  operation_id,
  status,
  machine_id,
  heartbeat_at
FROM active_operations
WHERE status IN ('pending', 'started', 'running', 'retrying')
  AND heartbeat_at < NOW() - INTERVAL '10 minutes';
```

Expected:

- zero rows.

If the operator explicitly approves a machine-stop simulation, record that separately as a high-risk production test.

---

## Suite 5: Download Regression Checks

Downloads are still worth checking, but they are no longer the multi-machine blocker.

### 5.1 Browser-direct single-file download

Run one download from each provider where practical.

Expected:

- download starts successfully.
- no requests hit removed ZIP endpoints.

Negative check:

- no network request to `/api/rclone/download`.
- no network request to `/api/download-zip`.

### 5.2 Provider fallback stream smoke

Where applicable, verify that Google Workspace export or Dropbox restricted download fallback still works.

Expected:

- download succeeds.
- if Fly is involved, it uses current stream endpoints, not ZIP sessions.

---

## Suite 6: Database Consistency Checks

Run these at the end of the session.

### 6.1 No orphaned active operations

```sql
SELECT COUNT(*) AS orphaned_count
FROM active_operations
WHERE status IN ('pending', 'started', 'running', 'retrying')
  AND heartbeat_at < NOW() - INTERVAL '10 minutes';
```

Expected:

- `orphaned_count = 0`.

### 6.2 Recent terminal operations have machine ownership

```sql
SELECT
  operation_id,
  status,
  type,
  machine_id,
  started_at,
  completed_at
FROM active_operations
WHERE created_at > NOW() - INTERVAL '1 hour'
ORDER BY created_at DESC
LIMIT 20;
```

Expected:

- Fly-backed operations have non-null `machine_id`.
- completed or cancelled operations show sensible timestamps.

### 6.3 Optional legacy table drift check

```sql
SELECT to_regclass('public.download_sessions') AS legacy_download_sessions_table;
```

Interpretation:

- if null: good, schema is fully current.
- if non-null: note it as cleanup drift only, not an active runtime failure.

---

## Post-Test Cleanup

1. Delete only disposable test fixtures created during this session.
2. Scale Fly back to the intended baseline if you raised it for testing:

```bash
fly scale count 1 --app stratofusion-rclone-prod
fly machine list --app stratofusion-rclone-prod
```

3. Re-run the orphaned-operation query.
4. Summarize pass/fail by suite and by provider combination.

---

## Success Criteria

- All Fly-backed copy, move, folder, batch, sync, and backup tests pass.
- At least one stress window shows operations owned by 2+ Fly machines.
- Remote cancellation updates DB state and stops the operation cleanly.
- No stale active-operation rows remain after the cleanup window.
- Browser downloads do not hit removed ZIP endpoints.
- Non-Fly toolbar smoke tests show no obvious regressions.

---

## Failure Report Template

Use this format for every failure:

- Test Case.
- Provider / Account Pair.
- Fixture Path.
- Expected Result.
- Actual Result.
- UI Error.
- DB Evidence.
- Fly Log Evidence.
- Reproducible.
- Severity.

---

## Quick Start Command for the Agent

```text
Execute the multi-machine validation plan in `prompts/test-multi-machine-scaling.md`.

Runtime inputs:
- APP_URL=https://stratofusion.io
- APP_EMAIL=<provided securely at runtime>
- APP_PASSWORD=<provided securely at runtime>
- FLY_APP_NAME=stratofusion-rclone-prod
- DATABASE_URL=<provided securely at runtime>

Rules:
1. Create disposable test fixtures first.
2. Treat search/rename/delete/create-folder as smoke tests only.
3. Focus multi-machine validation on Fly-backed copy, move, folder, batch, sync, and backup operations.
4. Record SQL evidence from `active_operations` for every Fly-backed test.
5. Report results suite by suite, with explicit pass/fail and evidence.

Begin with Pre-Test Setup and Suite 1.
```