If the sync dialog shows Request origin could not be verified while waiting
to start, middleware rejected the API request before the sync handler ran.
Check the request's Origin against its public Host and scheme. Standalone
Next.js can expose the internal container address in nextUrl.origin; the
request-integrity policy must use the Host preserved by Caddy instead.
Deploy the middleware origin fix through the normal CI/release workflow.
Resetting bisync state or reconnecting storage accounts does not repair this
request-integrity failure. Cross-site requests and mismatched public origins
must continue to receive 403 responses.
Issue: Access Test Failed Error
Symptom
When executing a two-way sync (bisync) operation, the sync fails with the following error:
2025/11/12 02:46:46 NOTICE: --check-access: Failed to find any files named RCLONE_TEST
2025/11/12 02:46:46 ERROR : Access test failed: Path1 count 0, Path2 count 0 - RCLONE_TEST
2025/11/12 02:46:46 ERROR : Bisync critical error: check file check failed
2025/11/12 02:46:46 ERROR : Bisync aborted. Error is retryable without --resync due to --resilient mode.
Root Cause
The error occurs when rclone bisync runs with --check-access but cannot find access-check marker files (RCLONE_TEST) in both sync paths.
Why this happens:
First-time bisync requires --resync: New two-way jobs must run a first initialization with --resync.
--check-access is explicit opt-in: The sync stack does not enable this by default on subsequent runs.
If opted in, marker files must exist: When is enabled without marker files, rclone aborts with .
checkAccess: true
Path1 count 0, Path2 count 0 - RCLONE_TEST
Possible Causes
checkAccess was explicitly enabled on a job/request
No RCLONE_TEST marker files exist in one or both paths
Database state issue: bisyncInitialized is wrong, so run mode is wrong
Folder path resolution failure: source/destination path is not what you expected
Issue: Prior Lock File Found
Symptom
A scheduled two-way sync fails with output similar to:
NOTICE: Failed to bisync: prior lock file found: /root/.cache/rclone/bisync/<name>.lck
Root Cause
Rclone creates a bisync lock file while it is comparing and reconciling the two paths. If a Fly.io rclone process is interrupted before rclone clears that file, a later run can fail before doing any data-plane work.
Current Handling
Fly rclone sync launches now:
Pass --color NEVER so terminal ANSI color codes do not appear in job failure messages.
Use --workdir <isolated temp dir> for bisync even when no persisted state exists, so new runs do not fall back to the shared /root/.cache/rclone/bisync default lock location.
Continue uploading successful bisync state to Neon and cleaning the temporary workdir after success, failure, or cancellation.
The Jobs UI maps prior lock file found into a safe retry message instead of showing the raw rclone cache path or its suggested deletefile command.
Issue: Token Expired During Bisync
Symptom
Rclone reports that a provider token expired during an active sync and that the remote has no refresh token, often followed by:
Bisync aborted. Must run --resync to recover.
Current Handling
The reconciliation cron now treats this as recoverable when the StratoFusion account can refresh credentials outside rclone. It force-refreshes the source and destination accounts, clears the stale rclone operation IDs, resets sync items to pending, and schedules the next two-way run with --resync.
If the provider grant itself requires interactive consent, the job still surfaces a reconnect action because the platform cannot mint a new provider grant without the user.
Issue: Transfer Quota Hit During Sync
If a sync launch or active rclone operation reaches the daily/monthly transfer quota, the job is paused until the parsed reset time instead of staying running or failing permanently. Two-way syncs are marked for --resync on the next run because the previous bisync baseline may be incomplete.
Diagnostic Steps
1. Check Bisync Initialization Status
Use the provided script to check the status of your sync job:
node scripts/check-bisync-status.mjs <jobId>
This will show:
Current value of bisyncInitialized.
Job status and mode.
Last run time.
Source and destination summaries.
2. Check Application Logs
Look for the following log entries to understand what's happening:
--check-access is used only if checkAccess: true is explicitly passed.
Auto-Recovery Behavior
When bisync encounters conflicts that require --resync (e.g., bisync aborted, bisync critical error, Path1 and Path2 are not in sync, missing bisync listings, or --check-access gating), the reconciler now:
Detects the error via isBisyncConflict() and increments the recovery_attempts counter (default limit: 3; configurable via SYNC_MAX_RECOVERY_ATTEMPTS).
Resets bisyncInitialized to false, clears the job error, and schedules the job to rerun immediately with next_run_at = NOW().
Sets metadata.requiresResync = true (plus resyncReason, recoveryAttempt, originalError) so the launch phase always sends --resync to the Fly.io rclone service.
Logs each recovery attempt through writeSyncReconciliationActivityLog with override messages so the Activity Log shows Sync incomplete / recovering instead of a generic failed message while auto-recovery is queued.
Exposes recovering jobs in the Jobs UI with a recovering badge so users know an automatic retry is already scheduled and do not need to press Retry immediately.
Use this query to inspect recovery state:
SELECT id,status, bisync_initialized, recovery_attempts, metadata ->>'requiresResync'AS requires_resync
FROM sync_jobs
WHERE id ='<jobId>';
Watch for the following log patterns:
[reconcile] Sync job <id>: Bisync conflict detected, attempting auto-recovery
[executeSyncJob] Using --resync flag for job <id> (reason: first-time initialization|recovery from bisync conflict)
[SYNC] Bisync using --resync flag (reason: <first_time|recovery>)
[reconcile] Sync job <id> → scheduled (auto-recovery)
[reconcile] Sync job <id>: Cleared recovery metadata after successful resync
If recovery attempts exceed the configured limit, the job is left failed, the user-facing message changes to Sync incomplete ... Manual retry is required, and the activity log notes recoveryAttemptsExhausted.
Fly machine shutdown or heartbeat-loss recovery is tracked separately from bisync conflict recovery. Machine interruptions default to 6 attempts and can be configured with SYNC_MAX_MACHINE_RECOVERY_ATTEMPTS; bisync/auth recovery still uses SYNC_MAX_RECOVERY_ATTEMPTS. The latest machine recovery attempt is stored under metadata.recovery with reason = 'fly_machine_shutdown_or_heartbeat_loss', so a long first-time bisync can survive more Fly ownership-loss events without consuming the lower bisync-conflict budget.