# Sync Job Execution Architecture

**Last Updated:** 2026-04-20
**Status:** Active

## Summary

Sync jobs now use the same **launch and reconcile** architecture as backup jobs:

1. Launch rclone operations on Fly.io and return immediately.
2. Persist operation IDs in the database.
3. Let `reconcile-stale-jobs` detect terminal completion/failure asynchronously.

This avoids blocking API requests on long-running sync operations (including bisync) and prevents serverless timeout failures.

## Why This Pattern

Vercel serverless functions have strict execution limits, while sync operations can run for many minutes. Waiting for terminal completion inside request handlers is not reliable for production workloads.

## Current Flow

### One-Time Sync (`schedule = "none"`)

- Route: `src/app/api/jobs/sync/route.ts`.
- Creates the job and returns `201` immediately.
- Uses `after()` to launch `executeSyncJob()` in background.
- Uses a 300-second function budget for the background launch window, matching.
  cron and transfer routes so broad selected-item syncs can record operation
  IDs for every selected source before Vercel stops the function.
- Does not recursively enumerate source or destination folders during the create request; quota and file-size checks run in the launch phase before Fly.io rclone starts.
- UI behavior: the sync dialog closes after queueing, and users monitor progress/failures from **Jobs** and **Activity Logs**.

### Scheduled Sync

- Route: `src/app/api/cron/execute-sync-jobs/route.ts`.
- Marks due jobs as `running`.
- Uses `after()` to launch `executeSyncJob()` for each due job.
- Returns cron response immediately after queueing launch work.
- Uses the same 300-second launch budget as run-now sync and transfer routes so.
  broad selected-item syncs can record every operation ID before reconciliation
  checks completeness.

### Launch Phase

- Module: `src/lib/sync/execute-sync-job.ts`.
- Validates job state and payload.
- Resolves/validates source and destination paths.
- Uses OneDrive provider metadata as a fast-launch signal for large one-way.
  sync folders. When the estimate is reliable and above the deferred-manifest
  threshold, recursive manifest validation is skipped for the launch so the
  Fly.io rclone operation ID is persisted before Vercel can time out.
- Preserves two-way bisync logic.
  - `resync` flag for first-time bisync initialization.
  - `--check-access` is explicit opt-in only (`checkAccess: true`).
    - Reason: avoid false `RCLONE_TEST` aborts on valid runs (for example, after deletions that leave both sides empty).
- Launches operations via `executeSyncOperation()`.
- Stores launched operation IDs incrementally on the job record.
- Does **not** call `waitForCompletion()`.

### Reconciliation Phase

- Cron route: `src/app/api/cron/reconcile-stale-jobs/route.ts`.
- Reconciler: `src/lib/jobs/reconcile-stale-jobs.ts`.
- Polls Fly.io for stored operation IDs.
- Finalizes job/item status and activity logs when operations reach terminal state.

## Bisync Auto-Recovery Enhancements

- **Conflict detection + metadata**: `reconcile-stale-jobs.ts` now uses `isBisyncConflict()` to identify bisync-specific failures. Failed jobs reset `bisyncInitialized`, update a `recovery_attempts` counter (default 3 attempts, configurable via `SYNC_MAX_RECOVERY_ATTEMPTS`), and store `metadata.requiresResync = true`/`resyncReason` so the next launch uses `--resync`.
- **Executor awareness**: `execute-sync-job.ts` inspects `metadata.requiresResync` alongside `bisyncInitialized` to set `needsResync`, logs the resync reason (`first-time initialization` vs `recovery from bisync conflict`), and forwards metadata to the Fly.io sync service.
- **Fly.io rclone logging**: `fly-rclone/src/routes/syncRoutes.js` now prefers `--resync` over `--check-access`, logs whether `metadata.resyncReason` drove the decision, and surfaces the flag configuration for every bisync operation.
- **Metadata cleanup**: After a successful reconciliation run that used resync, the reconciler clears `requiresResync` so subsequent launches revert to normal two-way behavior (only `--resync` when `bisyncInitialized`=false again).

## Large Sync Watchdog Profiles

- Sync launch now classifies each operation as one of.
  - `normal-sync`.
  - `large-one-way-sync`.
  - `large-two-way-sync`.
  - `large-bisync-initialization`.
- Classification uses estimated size/file/folder thresholds and unknown-root fallback. This protects very large provider folders where listing/checking can stay byte-idle for long periods.
- First-time bisync initialization (`--resync`) is handled conservatively and receives the longest timeout profile.
- Sync metadata now forwards watchdog hints (`operationTimeoutMs`, `stallTimeoutMs`, profile label/reason, sync mode, and recovery/init flags) to Fly rclone so queue defaults do not incorrectly cancel valid long-running operations.
- Timeout cancellations remain `cancelled` at operation level with `cancelReason = timeout`; reconciliation treats those as failed timeout outcomes rather than user cancellations.

## API Behavior

### `/api/jobs/sync`

Returns success immediately after creating/queueing launch work. It does not block on sync completion.

The create request also avoids recursive provider manifest resolution. Large OneDrive sync launches can defer recursive manifest validation when provider metadata supplies a reliable size estimate, preventing `No operation IDs recorded; job orphaned` failures caused by pre-launch enumeration exceeding the serverless window.

### `/api/rclone/sync`

Launches a sync operation and returns the operation ID immediately. It does not poll for terminal completion in the request lifecycle.

## Google->Google Cross-Account Guard

- File: `fly-rclone/src/routes/syncRoutes.js`.
- One-way sync no longer forces `--drive-server-side-across-configs`.
- The flag is now explicit opt-in only (`options.enableDriveServerSideAcrossConfigs === true`).
- Reason: production logs showed repeated `googleapi: Error 404: File not found` failures for cross-account Google sync jobs when this flag was forced.

## Related Files

- `src/lib/sync/execute-sync-job.ts`.
- `src/app/api/jobs/sync/route.ts`.
- `src/app/api/rclone/sync/route.ts`.
- `src/app/api/cron/execute-sync-jobs/route.ts`.
- `src/app/api/cron/reconcile-stale-jobs/route.ts`.
- `src/lib/jobs/reconcile-stale-jobs.ts`.