# Cron Job Deployment Infrastructure

> **Runtime status:** production scheduling now runs from exactly one `cron`
> container in the OVHcloud VM Compose stack. The Vercel Cron setup and commands
> retained below document the previous architecture and a legacy recovery path;
> they are not normal production deployment instructions. See
> [Deployment](./DEPLOYMENT.md) and the
> [Production VM Runbook](./operations/VM_PRODUCTION_RUNBOOK.md).

## Overview

The same authenticated Next.js cron routes execute scheduled sync, backup, and
reconciliation work in every supported runtime. Current production invokes
them from the VM cron container; the previous architecture invoked them with
Vercel Cron Jobs.

## Architecture

### Cron Job Endpoints

#### 1. Backup Jobs Cron (`/api/cron/execute-backups`)
- **Schedule**: Every 5 minutes (`*/5 * * * *`).
- **Location**: `src/app/api/cron/execute-backups/route.ts`.
- **Purpose**: Executes scheduled backup jobs.
- **Supported Schedules**.
  - Time-based: daily, weekly, monthly.
  - Interval-based (testing): minutely-5, minutely-10, minutely-15, minutely-20, minutely-30.
- **Max Duration**: 300 seconds (Vercel function override).

#### 2. Sync Jobs Cron (`/api/cron/execute-sync-jobs`)
- **Schedule**: Every minute (`* * * * *`).
- **Location**: `src/app/api/cron/execute-sync-jobs/route.ts`.
- **Purpose**: Finds due sync jobs and launches them.
- **Supported Schedules**.
  - Time-based: daily, weekly, monthly.
  - Interval-based: hourly-1, hourly-2, hourly-3, hourly-4, hourly-6, hourly-12.
  - Interval-based: minutely-5, minutely-10, minutely-15, minutely-20, minutely-30.
- **Max Duration**: 300 seconds (Vercel function override).

#### 3. Reconciliation Cron (`/api/cron/reconcile-stale-jobs`)
- **Schedule**: Every minute (`* * * * *`).
- **Location**: `src/app/api/cron/reconcile-stale-jobs/route.ts`.
- **Purpose**: Polls Fly.io for running operation IDs and finalizes job/activity-log terminal state.
- **Max Duration**: 300 seconds (Vercel function override).

### Configuration

#### vercel.json

```json
{
  "crons": [
    {
      "path": "/api/cron/execute-backups",
      "schedule": "*/5 * * * *"
    },
    {
      "path": "/api/cron/execute-sync-jobs",
      "schedule": "* * * * *"
    },
    {
      "path": "/api/cron/reconcile-stale-jobs",
      "schedule": "* * * * *"
    }
  ],
  "functions": {
    "src/app/api/cron/execute-backups/route.ts": {
      "maxDuration": 300
    },
    "src/app/api/cron/execute-sync-jobs/route.ts": {
      "maxDuration": 300
    },
    "src/app/api/cron/reconcile-stale-jobs/route.ts": {
      "maxDuration": 300
    }
  }
}
```

The cron route modules also export `maxDuration = 300`. Keep the route-level
exports and `vercel.json` aligned; scheduled launch work runs in `after()`, and
reconciliation treats mature jobs with missing operation IDs as incomplete
launch failures.

## Implementation Details

### Sync Job Execution Flow

1. **Cron Trigger**: Vercel triggers the cron endpoint every minute
2. **Authentication**: Request is authenticated via `Authorization: Bearer ${CRON_SECRET}`
3. **Job Discovery**: Query database for jobs where `status = 'scheduled'` and `nextRunAt <= now()`
4. **Queue Launch Work**:
   - Update due jobs to `running`.
   - Queue `executeSyncJob()` in `after()` callback.
5. **Launch Phase (`executeSyncJob`)**:
   - Validate payload + resolve paths.
   - Launch rclone sync/bisync operations on Fly.io.
   - Persist operation IDs to `sync_jobs.operationIds`.
   - Return without waiting for terminal completion.
6. **Reconciliation Phase**:
   - `reconcile-stale-jobs` polls Fly.io for each operation ID.
   - Marks jobs/items `completed` or `failed` when terminal state is reached.
   - Updates activity logs with terminal outcomes.
7. **Response**: Cron launch endpoint returns summary immediately after queueing launch work

### Key Functions

#### `executeSyncJob(jobId: string, userId: string)`
Located in `src/lib/sync/execute-sync-job.ts`

Executes a single sync job:
- Retrieves job and items from database.
- Parses job payload.
- Launches sync operation(s) for each item on Fly.io.
- Stores operation IDs for reconciliation.
- Does not poll for completion.
- Handles errors gracefully.

#### `computeNextRunAt(schedule: SyncSchedule, scheduledTime?: string, timezone?: string)`
Located in `src/lib/database/sync-jobs.ts`

Calculates the next run time based on schedule:
- **Time-based**: Uses scheduled time and timezone.
- **Interval-based**: Adds interval to current time.

#### `generateCronExpression(schedule: SyncSchedule, scheduledTime?: string)`
Located in `src/lib/database/sync-jobs.ts`

Generates cron expressions for all schedule types:
- Daily: `30 14 * * *` (14:30).
- Weekly: `30 14 * * 1` (Monday 14:30).
- Monthly: `30 14 1 * *` (1st of month 14:30).
- Hourly: `0 */2 * * *` (every 2 hours).
- Minutely: `*/15 * * * *` (every 15 minutes).

## Database Schema

### sync_jobs Table

```sql
CREATE TABLE sync_jobs (
  id TEXT PRIMARY KEY,
  userId TEXT NOT NULL,
  status TEXT NOT NULL,  -- scheduled | running | completed | failed | cancelled
  schedule TEXT NOT NULL,  -- daily | weekly | monthly | hourly-N | minutely-N
  scheduledTime TEXT,  -- HH:mm format
  timezone TEXT DEFAULT 'UTC',
  nextRunAt TIMESTAMP,  -- When job should run next
  lastRunAt TIMESTAMP,  -- When job last ran
  startedAt TIMESTAMP,  -- When current execution started
  finishedAt TIMESTAMP,  -- When current execution finished
  lastError TEXT,  -- Error message if failed
  ...
);

CREATE INDEX sync_jobs_next_run_idx ON sync_jobs(nextRunAt);
CREATE INDEX sync_jobs_status_idx ON sync_jobs(userId, status);
```

## Security

### Authentication
- All cron endpoints require `CRON_SECRET` environment variable.
- Request must include `Authorization: Bearer ${CRON_SECRET}` header.
- Unauthorized requests return 401 Unauthorized.

### Environment Variables
```bash
CRON_SECRET=your-secret-key-here
```

## Monitoring & Troubleshooting

### Vercel Logs
```bash
# View production logs
vercel logs --prod --follow

# Filter for cron jobs
vercel logs --prod | grep "execute-sync-jobs"

# View logs from last hour
vercel logs --prod --since 1h
```

### Manual Testing
```bash
# Test sync jobs cron
curl -X GET https://stratofusion.io/api/cron/execute-sync-jobs \
  -H "Authorization: Bearer your-cron-secret"
```

### Local Windows Testing

Use PowerShell on Windows 11. Vercel cron does not run automatically on `localhost`.

```powershell
$env:ALLOW_LOCAL_CRON = "true"
$env:CRON_SECRET = "local-dev-secret-123"
.\scripts\local-cron.ps1
```

For persistent local replay, set both values in `.env.local` and restart `pnpm dev` before running the script:

```env
ALLOW_LOCAL_CRON=true
CRON_SECRET=your-local-secret
```

Useful variants:

```powershell
.\scripts\local-cron.ps1 -RunOnce
.\scripts\local-cron.ps1 -SkipBackups
.\scripts\local-cron.ps1 -BaseUrl "http://localhost:3000" -CronSecret "local-dev-secret-123"
```

### Common Issues

**Issue**: Sync jobs not running
- Check `nextRunAt` is in the past.
- Verify job status is 'scheduled'.
- Check database connection.
- Review cron logs for errors.

**Issue**: Jobs marked as failed
- Check `lastError` field in database.
- Verify sync operation parameters.
- Check rclone service availability.

**Issue**: Unauthorized errors
- Verify `CRON_SECRET` is set in production.
- Check secret matches in code and environment.
- Redeploy after setting environment variable.

## Performance Considerations

### Job Limits
- Maximum 10 jobs processed per cron run.
- Reduces timeout risk by bounding per-run launch volume.
- Remaining jobs processed in next cron run.

### Database Indexes
- `nextRunAt` index for efficient job discovery.
- `userId + status` index for filtering.

### Optimization Tips
- Monitor job execution times.
- Adjust cron frequency if needed.
- Consider batch processing for large operations.
- Use database connection pooling.

## Future Enhancements

1. **Adaptive Scheduling**: Adjust cron frequency based on job volume
2. **Job Prioritization**: Process high-priority jobs first
3. **Retry Logic**: Automatic retry for failed jobs
4. **Metrics**: Track execution times and success rates
5. **Alerts**: Notify admins of failed jobs
6. **Dead Letter Queue**: Handle permanently failed jobs

## Related Documentation

- [Sync Scheduling Intervals](./SYNC_SCHEDULING_INTERVALS.md).
- [Deployment Guide](./DEPLOYMENT.md).
- [Backup Feature](./BACKUP_FEATURE.md).