StratoFusion
Documentation / Multi-Machine Test Suites And Reporting Test Suites
Suite 1: Non-Destructive Smoke Checks
These are useful baseline checks, but they are not evidence of Fly multi-machine correctness.
1.1 Search smoke test
Steps:
Search for a known test term across all services.
Confirm results appear from multiple connected accounts.
Expected:
results render correctly.
no obvious UI or provider errors.
no requirement for active_operations evidence.
1.2 Rename smoke test
Steps:
Rename a disposable test file.
Refresh and verify the new name persists.
Expected:
rename succeeds.
no Fly-specific DB row required.
1.3 Create folder and delete smoke test
Steps:
Create a disposable folder.
Delete only that disposable folder or file.
Expected:
operation succeeds.
do not treat lack of active_operations evidence as a failure.
Suite 2: Core Fly-Backed Transfer Operations
These are the primary multi-machine tests.
For every test in this suite, collect:
SELECT
operation_id ,
status ,
type ,
machine_id ,
cancel_requested_at ,
cancel_reason ,
heartbeat_at ,
started_at ,
completed_at
FROM active_operations
ORDER BY created_at DESC
LIMIT 10 ;
2.1 Single-file copy across accounts on the same provider
Suggested path:
Google Drive Personal -> Google Drive Business.
copy starts successfully.
destination file appears.
source file remains.
recent active_operations row shows.
type aligned with copy behavior.terminal status of completed.
non-null machine_id.
2.2 Single-file copy across providers Run at least these combinations:
Google Drive Personal -> Dropbox Personal
Dropbox Business -> OneDrive Personal
OneDrive Business -> Google Drive Business
each transfer completes successfully.
file size and name are correct at destination.
DB row is present with terminal success.
2.3 Folder copy across providers Run one recursive folder copy with 5-10 files.
folder structure is preserved.
progress updates appear during transfer.
destination contents match source fixture.
recent snapshot JSON reflects the operation details.
SELECT
operation_id ,
status ,
type ,
machine_id ,
snapshot
FROM active_operations
ORDER BY created_at DESC
LIMIT 5 ;
2.4 Move within the same provider Use a disposable file only.
Dropbox Personal -> different folder in the same account.
source entry disappears.
destination entry appears.
no duplicate remains behind.
DB row reaches terminal success.
2.5 Batch copy or multi-select copy Use 5-10 files in a disposable folder.
operation progresses cleanly.
no silent partial failures.
no duplicate or missing files at destination.
Suite 3: Sync and Backup Runs These are important because they are long-running and operationally sensitive.
3.1 One-time sync
Google Drive Personal -> Dropbox Personal.
sync job starts and completes.
destination matches source fixture.
recent active_operations row shows type of sync or bisync as appropriate.
3.2 Scheduled sync smoke validation Create or use a disposable scheduled sync job, then trigger it manually if the UI supports that.
job runs.
sync_jobs.last_run_at updates.next_run_at is populated when scheduling applies. SELECT
id ,
status ,
schedule ,
last_run_at ,
next_run_at
FROM sync_jobs
ORDER BY created_at DESC
LIMIT 10 ;
3.3 Backup run Create or use a disposable backup path and run it once.
backup executes successfully.
backup_jobs.last_run_at updates.Fly operation state remains consistent throughout execution.
SELECT
id ,
status ,
schedule ,
last_run_at ,
next_run_at
FROM backup_jobs
ORDER BY created_at DESC
LIMIT 10 ;
Suite 4: Cross-Machine Stress and Cancellation This suite is the actual horizontal-scaling proof.
4.1 Confirm 2+ running Fly machines fly machine list --app stratofusion-rclone-prod
at least two machines in a running state.
4.2 Launch concurrent Fly-backed operations Open multiple tabs or sessions and start at least 4 Fly-backed operations close together:
cross-provider copy
same-provider copy
folder copy
sync or backup run
all operations start successfully.
logs show activity from multiple machine IDs.
DB shows recent operations spread across 2+ machine_id values.
SELECT
machine_id ,
COUNT ( * ) AS operation_count
FROM active_operations
WHERE created_at > NOW ( ) - INTERVAL '15 minutes'
GROUP BY machine_id
ORDER BY operation_count DESC ;
more than one non-null machine_id appears during the stress window.
4.3 Remote cancellation Start one large enough folder copy or backup that it remains active for at least 15-30 seconds, then cancel it from the UI.
cancellation request is accepted by the UI.
cancel_requested_at becomes non-null in DB.the operation reaches a terminal non-running state quickly.
no indefinite running row remains.
SELECT
operation_id ,
status ,
machine_id ,
cancel_requested_at ,
cancel_reason ,
heartbeat_at ,
completed_at ,
snapshot
FROM active_operations
WHERE cancel_requested_at IS NOT NULL
ORDER BY created_at DESC
LIMIT 10 ;
cancel_reason reflects user cancellation semantics.final status is no longer running, started, pending, or retrying.
4.4 Heartbeat cleanup check Do not intentionally kill random production machines unless the operator explicitly approves it.
Safer production validation:
After stress and cancellation tests, wait long enough to exceed the stale threshold for any abandoned row.
Query for stale active rows.
SELECT
operation_id ,
status ,
machine_id ,
heartbeat_at
FROM active_operations
WHERE status IN ( 'pending' , 'started' , 'running' , 'retrying' )
AND heartbeat_at < NOW ( ) - INTERVAL '10 minutes' ;
If the operator explicitly approves a machine-stop simulation, record that separately as a high-risk production test.
Suite 5: Download Regression Checks Downloads are still worth checking, but they are no longer the multi-machine blocker.
5.1 Browser-direct single-file download Run one download from each provider where practical.
download starts successfully.
no requests hit removed ZIP endpoints.
no network request to /api/rclone/download.
no network request to /api/download-zip.
5.2 Provider fallback stream smoke Where applicable, verify that Google Workspace export or Dropbox restricted download fallback still works.
download succeeds.
if Fly is involved, it uses current stream endpoints, not ZIP sessions.
Suite 6: Database Consistency Checks Run these at the end of the session.
6.1 No orphaned active operations SELECT COUNT ( * ) AS orphaned_count
FROM active_operations
WHERE status IN ( 'pending' , 'started' , 'running' , 'retrying' )
AND heartbeat_at < NOW ( ) - INTERVAL '10 minutes' ;
6.2 Recent terminal operations have machine ownership SELECT
operation_id ,
status ,
type ,
machine_id ,
started_at ,
completed_at
FROM active_operations
WHERE created_at > NOW ( ) - INTERVAL '1 hour'
ORDER BY created_at DESC
LIMIT 20 ;
Fly-backed operations have non-null machine_id.
completed or cancelled operations show sensible timestamps.
6.3 Optional legacy table drift check SELECT to_regclass ( 'public.download_sessions' ) AS legacy_download_sessions_table ;
if null: good, schema is fully current.
if non-null: note it as cleanup drift only, not an active runtime failure.
Post-Test Cleanup
Delete only disposable test fixtures created during this session.
Scale Fly back to the intended baseline if you raised it for testing:
fly scale count 1 --app stratofusion-rclone-prod
fly machine list --app stratofusion-rclone-prod
Re-run the orphaned-operation query.
Summarize pass/fail by suite and by provider combination.
Success Criteria
All Fly-backed copy, move, folder, batch, sync, and backup tests pass.
At least one stress window shows operations owned by 2+ Fly machines.
Remote cancellation updates DB state and stops the operation cleanly.
No stale active-operation rows remain after the cleanup window.
Browser downloads do not hit removed ZIP endpoints.
Non-Fly toolbar smoke tests show no obvious regressions.
Failure Report Template Use this format for every failure:
Test Case.
Provider / Account Pair.
Fixture Path.
Expected Result.
Actual Result.
UI Error.
DB Evidence.
Fly Log Evidence.
Reproducible.
Severity.
Quick Start Command for the Agent Execute the multi-machine validation plan in `prompts/test-multi-machine-scaling.md`.
Runtime inputs:
- APP_URL=https://stratofusion.io
- APP_EMAIL=<provided securely at runtime>
- APP_PASSWORD=<provided securely at runtime>
- FLY_APP_NAME=stratofusion-rclone-prod
- DATABASE_URL=<provided securely at runtime>
Rules:
1. Create disposable test fixtures first.
2. Treat search/rename/delete/create-folder as smoke tests only.
3. Focus multi-machine validation on Fly-backed copy, move, folder, batch, sync, and backup operations.
4. Record SQL evidence from `active_operations` for every Fly-backed test.
5. Report results suite by suite, with explicit pass/fail and evidence.
Begin with Pre-Test Setup and Suite 1.