# Sync Bisync Troubleshooting Guide

## Issue: Request Origin Could Not Be Verified

If the sync dialog shows `Request origin could not be verified` while waiting
to start, middleware rejected the API request before the sync handler ran.
Check the request's Origin against its public Host and scheme. Standalone
Next.js can expose the internal container address in `nextUrl.origin`; the
request-integrity policy must use the Host preserved by Caddy instead.

Deploy the middleware origin fix through the normal CI/release workflow.
Resetting bisync state or reconnecting storage accounts does not repair this
request-integrity failure. Cross-site requests and mismatched public origins
must continue to receive 403 responses.

## Issue: Access Test Failed Error

### Symptom

When executing a two-way sync (bisync) operation, the sync fails with the following error:

```
2025/11/12 02:46:46 NOTICE: --check-access: Failed to find any files named RCLONE_TEST
2025/11/12 02:46:46 ERROR : Access test failed: Path1 count 0, Path2 count 0 - RCLONE_TEST
2025/11/12 02:46:46 ERROR : Bisync critical error: check file check failed
2025/11/12 02:46:46 ERROR : Bisync aborted. Error is retryable without --resync due to --resilient mode.
```

### Root Cause

The error occurs when rclone bisync runs with `--check-access` but cannot find access-check marker files (`RCLONE_TEST`) in both sync paths.

**Why this happens:**

1. **First-time bisync requires `--resync`**: New two-way jobs must run a first initialization with `--resync`.

2. **`--check-access` is explicit opt-in**: The sync stack does not enable this by default on subsequent runs.

3. **If opted in, marker files must exist**: When `checkAccess: true` is enabled without marker files, rclone aborts with `Path1 count 0, Path2 count 0 - RCLONE_TEST`.

### Possible Causes

1. **`checkAccess` was explicitly enabled** on a job/request
2. **No `RCLONE_TEST` marker files exist** in one or both paths
3. **Database state issue**: `bisyncInitialized` is wrong, so run mode is wrong
4. **Folder path resolution failure**: source/destination path is not what you expected

## Issue: Prior Lock File Found

### Symptom

A scheduled two-way sync fails with output similar to:

```
NOTICE: Failed to bisync: prior lock file found: /root/.cache/rclone/bisync/<name>.lck
```

### Root Cause

Rclone creates a bisync lock file while it is comparing and reconciling the two paths. If a Fly.io rclone process is interrupted before rclone clears that file, a later run can fail before doing any data-plane work.

### Current Handling

Fly rclone sync launches now:

1. Pass `--color NEVER` so terminal ANSI color codes do not appear in job failure messages.
2. Use `--workdir <isolated temp dir>` for bisync even when no persisted state exists, so new runs do not fall back to the shared `/root/.cache/rclone/bisync` default lock location.
3. Continue uploading successful bisync state to Neon and cleaning the temporary workdir after success, failure, or cancellation.

The Jobs UI maps `prior lock file found` into a safe retry message instead of showing the raw rclone cache path or its suggested `deletefile` command.

## Issue: Token Expired During Bisync

### Symptom

Rclone reports that a provider token expired during an active sync and that the remote has no refresh token, often followed by:

```text
Bisync aborted. Must run --resync to recover.
```

### Current Handling

The reconciliation cron now treats this as recoverable when the StratoFusion account can refresh credentials outside rclone. It force-refreshes the source and destination accounts, clears the stale rclone operation IDs, resets sync items to pending, and schedules the next two-way run with `--resync`.

If the provider grant itself requires interactive consent, the job still surfaces a reconnect action because the platform cannot mint a new provider grant without the user.

## Issue: Transfer Quota Hit During Sync

If a sync launch or active rclone operation reaches the daily/monthly transfer quota, the job is paused until the parsed reset time instead of staying `running` or failing permanently. Two-way syncs are marked for `--resync` on the next run because the previous bisync baseline may be incomplete.

## Diagnostic Steps

### 1. Check Bisync Initialization Status

Use the provided script to check the status of your sync job:

```bash
node scripts/check-bisync-status.mjs <jobId>
```

This will show:

- Current value of `bisyncInitialized`.
- Job status and mode.
- Last run time.
- Source and destination summaries.

### 2. Check Application Logs

Look for the following log entries to understand what's happening:

**In Next.js application logs:**

```
[executeSyncJob] Bisync initialization check for item <itemId>
  - bisyncInitialized: true/false
  - needsResync: true/false
  - willUseResyncFlag: true/false
```

**In rclone service logs:**

```
[SYNC-SERVICE] Starting bisync (two-way) operation
  - resync: true/false
  - resyncFlag: true/false
  - checkAccess: true/false
  - willUseResyncFlag: true/false
  - willUseCheckAccessFlag: true/false
```

**In Fly.io rclone logs:**

```
[SYNC] Bisync flag configuration
  - resyncOption: true/false
  - checkAccessOption: true/false
  - willUseResync: true/false
  - willUseCheckAccess: true/false
```

### 3. Verify Folder Path Resolution

Check the logs for Google Drive folder path resolution:

```
[GOOGLE-DRIVE-RESOLVER] ✅ Resolved folder ID <folderId> to path: "<path>"
  - originalPath: <path>
  - filteredPath: <path>
  - isRootLevel: true/false
```

And destination folder resolution:

```
[TRANSFER-UTILS] ✅ Destination folder resolved
  - originalId: <folderId>
  - resolvedPath: <path>
  - wasResolved: true/false
```

## Solutions

### Solution 1: Reset Bisync Initialization Flag

If the `bisyncInitialized` flag is incorrectly set to `true`, reset it using the script:

```bash
node scripts/check-bisync-status.mjs <jobId> --reset
```

This will:

1. Set `bisyncInitialized` to `false`
2. Update the `updated_at` timestamp
3. Ensure the next sync uses `--resync` for initialization

### Solution 2: Verify Folder Paths

Ensure that both source and destination folder IDs are valid and accessible:

1. **OneDrive source**: Verify the folder ID format (should be `driveId!itemId` or simple item ID)
2. **Google Drive destination**: Verify the folder ID is a valid 33-character alphanumeric string
3. Check that you have access to both folders with the connected accounts

### Solution 3: Delete and Recreate Sync Job

If the issue persists, delete the sync job and create a new one:

1. Delete the existing sync job through the UI or API
2. Create a new sync job with the same configuration
3. The new job will have `bisyncInitialized` set to `false` by default
4. Execute the sync job

## Prevention

### Best Practices

1. **Don't manually modify database**: Avoid manually setting the `bisyncInitialized` flag in the database
2. **Monitor first sync**: Always check logs for the first sync execution to ensure it completes successfully
3. **Handle errors properly**: If a first-time sync fails, the `bisyncInitialized` flag should remain `false` until a successful completion

### Code Flow

The correct flow for bisync initialization:

1. **Job Creation**: `bisyncInitialized` defaults to `false` (database default)
2. **First Execution**:
   - `needsResync = mode === "two-way" && !bisyncInitialized` -> `true`.
   - `resync: needsResync` -> `true`.
   - Rclone uses `--resync` flag, skips `--check-access`.
3. **Successful Completion**: `bisyncInitialized` set to `true`
4. **Subsequent Executions**:
   - `needsResync = mode === "two-way" && !bisyncInitialized` -> `false`.
   - `resync: needsResync` -> `false`.
   - Rclone skips `--resync`.
   - `--check-access` is used only if `checkAccess: true` is explicitly passed.

## Auto-Recovery Behavior

When bisync encounters conflicts that require `--resync` (e.g., `bisync aborted`, `bisync critical error`, `Path1 and Path2 are not in sync`, missing bisync listings, or `--check-access` gating), the reconciler now:

1. Detects the error via `isBisyncConflict()` and increments the `recovery_attempts` counter (default limit: 3; configurable via `SYNC_MAX_RECOVERY_ATTEMPTS`).
2. Resets `bisyncInitialized` to `false`, clears the job error, and schedules the job to rerun immediately with `next_run_at = NOW()`.
3. Sets `metadata.requiresResync = true` (plus `resyncReason`, `recoveryAttempt`, `originalError`) so the launch phase always sends `--resync` to the Fly.io rclone service.
4. Logs each recovery attempt through `writeSyncReconciliationActivityLog` with override messages so the Activity Log shows `Sync incomplete` / `recovering` instead of a generic failed message while auto-recovery is queued.
5. Exposes recovering jobs in the Jobs UI with a `recovering` badge so users know an automatic retry is already scheduled and do not need to press Retry immediately.

Use this query to inspect recovery state:

```sql
SELECT id, status, bisync_initialized, recovery_attempts, metadata ->> 'requiresResync' AS requires_resync
FROM sync_jobs
WHERE id = '<jobId>';
```

Watch for the following log patterns:

```
[reconcile] Sync job <id>: Bisync conflict detected, attempting auto-recovery
[executeSyncJob] Using --resync flag for job <id> (reason: first-time initialization|recovery from bisync conflict)
[SYNC] Bisync using --resync flag (reason: <first_time|recovery>)
[reconcile] Sync job <id> → scheduled (auto-recovery)
[reconcile] Sync job <id>: Cleared recovery metadata after successful resync
```

If recovery attempts exceed the configured limit, the job is left failed, the user-facing message changes to `Sync incomplete ... Manual retry is required`, and the activity log notes `recoveryAttemptsExhausted`.

Fly machine shutdown or heartbeat-loss recovery is tracked separately from bisync conflict recovery. Machine interruptions default to 6 attempts and can be configured with `SYNC_MAX_MACHINE_RECOVERY_ATTEMPTS`; bisync/auth recovery still uses `SYNC_MAX_RECOVERY_ATTEMPTS`. The latest machine recovery attempt is stored under `metadata.recovery` with `reason = 'fly_machine_shutdown_or_heartbeat_loss'`, so a long first-time bisync can survive more Fly ownership-loss events without consuming the lower bisync-conflict budget.

## Technical Details

### Database Schema

```sql
CREATE TABLE sync_jobs (
  ...
  bisync_initialized BOOLEAN DEFAULT false NOT NULL,
  ...
);
```

### Code References

- **Initialization check**: `src/lib/sync/execute-sync-job.ts` line 253.
- **Resync flag setting**: `src/lib/sync/execute-sync-job.ts` line 265.
- **Bisync flag update**: `src/lib/sync/execute-sync-job.ts` line 339.
- **Rclone flag logic**: `fly-rclone/src/routes/syncRoutes.js` line 95.

### Rclone Bisync Flags

- `--resync`: Initialize bisync state (first-time only).
- `--check-access`: Verify both folders are accessible by checking for RCLONE_TEST files (explicit opt-in only).
- `--resilient`: Continue on errors where possible.
- `--recover`: Recover from interrupted sync.

## Additional Resources

- [Rclone Bisync Documentation](https://rclone.org/bisync/).
- [Sync Job Execution Fixes](./SYNC_JOB_EXECUTION_FIXES.md).
- [Database Schema](../../src/lib/database/schema.ts).

## Support

If you continue to experience issues after following this guide:

1. Collect the full logs from both Next.js application and Fly.io rclone service
2. Run the bisync status check script and include the output
3. Verify folder IDs and paths are correct
4. Check that authentication tokens are valid for both services