# Fix: Scheduled Backups Creating Empty Directories

**Date:** October 18, 2025
**Status:** ✅ FIXED
**Branch:** `fix/backups-fail-on-vercel`
**Commit:** `ee9a0302`

---

## 🔴 Problem Summary

After deploying to Vercel Production environment (following the resolution of the Preview vs Production cron job issue), scheduled backup cron jobs were executing successfully but creating empty directories instead of copying files.

### Observed Behavior

- ✅ Vercel cron jobs triggering on schedule (every 5 minutes).
- ✅ `/api/cron/execute-backups` endpoint being called.
- ✅ Backup jobs marked as "running" in database.
- ✅ Backup directories created in destination service.
- ❌ **Backup directories were empty - no files copied**.
- ❌ Jobs marked as "failed" with error: `Unexpected token '<', "<!DOCTYPE "... is not valid JSON`.

---

## 🔍 Root Cause Analysis

### Investigation Steps

1. **Ran investigation script:**
   ```bash
   node scripts/investigate-failed-backup.mjs
   ```
   Result: Found failed jobs with error `Unexpected token '<', "<!DOCTYPE "... is not valid JSON`

2. **Analyzed error pattern:**
   - This error occurs when code expects JSON but receives HTML.
   - Typically happens when fetching a non-existent endpoint (404 page).

3. **Examined backup execution code:**
   - File: `src/app/api/jobs/[id]/execute/route.ts`.
   - Line 157: `const batchRes = await fetch(\`${process.env.NEXT_PUBLIC_SITE_URL}${batchCopyPath}\`, ...)`.
   - Line 114: Similar pattern for create-folder endpoint.

4. **Identified the issue:**
   - `process.env.NEXT_PUBLIC_SITE_URL` is **undefined** in server-side code.
   - The `NEXT_PUBLIC_` prefix means the variable is for **client-side code only**.
   - In server-side code, `NEXT_PUBLIC_*` variables are not available via `process.env`.
   - This caused the fetch URL to be `undefined/api/rclone/batch-copy` (invalid URL).
   - Invalid URL resulted in a 404 HTML error page instead of JSON response.

### Why This Wasn't Caught Earlier

- One-time backups worked because they were triggered from the client-side UI.
- Client-side code has access to `NEXT_PUBLIC_SITE_URL`.
- Scheduled backups run entirely server-side (cron → execute endpoint → batch-copy).
- Server-side code doesn't have access to `NEXT_PUBLIC_*` variables.

---

## ✅ Solution

### Created Server-Side URL Utility

**File:** `src/lib/server-url.ts`

**Key Functions:**
- `getServerApiUrl(path)` - Async function to construct full API URLs for server-side fetch calls.
- `getServerBaseUrl()` - Determines the correct base URL for the current environment.

**URL Construction Priority:**
1. **VERCEL_URL** environment variable (automatically set by Vercel in all deployments)
2. **x-forwarded-host** header (from incoming request)
3. **host** header (from incoming request)
4. **Fallback:** `http://localhost:3000` (for local development)

### Updated Affected Files

**1. `src/app/api/jobs/[id]/execute/route.ts`**
- Added import: `import { getServerApiUrl } from "~/lib/server-url";`.
- Line 115: Changed from `${process.env.NEXT_PUBLIC_SITE_URL}${createFolderPath}` to `await getServerApiUrl(createFolderPath)`.
- Line 159: Changed from `${process.env.NEXT_PUBLIC_SITE_URL}${batchCopyPath}` to `await getServerApiUrl(batchCopyPath)`.

**2. `src/app/api/cron/execute-backups/route.ts`**
- Added import: `import { getServerApiUrl } from "~/lib/server-url";`.
- Line 77: Changed from `${process.env.NEXT_PUBLIC_SITE_URL}/api/jobs/${job.id}/execute` to `await getServerApiUrl(\`/api/jobs/${job.id}/execute\`)`.

---

## 🧪 Testing & Verification

### Before Deploying

1. **Type Check:**
   ```bash
   pnpm typecheck
   ```
   Result: ✅ No errors

2. **Lint Check:**
   ```bash
   pnpm lint
   ```
   Result: ✅ No errors

### After Deploying to Production

1. **Wait for next cron execution** (within 5 minutes)

2. **Check Vercel logs:**
   ```bash
   vercel logs https://your-production-url.vercel.app
   ```
   Look for successful backup execution logs

3. **Verify in database:**
   ```bash
   node scripts/check-backup-jobs.mjs
   ```
   Check that jobs are marked as "completed" instead of "failed"

4. **Check destination service:**
   - Navigate to the backup destination folder.
   - Verify that timestamped backup folders contain files (not empty).

5. **Monitor for errors:**
   ```bash
   node scripts/investigate-failed-backup.mjs
   ```
   Should show no new failures with the "Unexpected token" error

---

## 📊 Technical Details

### Environment Variables in Next.js

**Client-Side Variables (`NEXT_PUBLIC_*`):**
- Available in browser JavaScript.
- Bundled into client-side code at build time.
- Accessible via `process.env.NEXT_PUBLIC_*` in client components.
- **NOT available in server-side code via `process.env`**.

**Server-Side Variables:**
- Available only in server-side code (API routes, server components).
- Accessible via `process.env.*` (without `NEXT_PUBLIC_` prefix).
- **NOT bundled into client-side code**.

**Vercel-Specific Variables:**
- `VERCEL_URL` - Automatically set by Vercel for all deployments.
- Format: `project-name-hash.vercel.app` (without protocol).
- Always use HTTPS: `https://${process.env.VERCEL_URL}`.

### Why VERCEL_URL is Better Than NEXT_PUBLIC_SITE_URL

1. **Automatically set** - No manual configuration needed
2. **Always correct** - Matches the actual deployment URL
3. **Works for all deployments** - Production, Preview, and Development
4. **Server-side accessible** - Available in `process.env` for server code

---

## 🚀 Deployment Steps

1. **Commit the fix:**
   ```bash
   git add -A
   git commit -m "fix: use proper server-side URL construction for internal API calls"
   ```

2. **Push to remote:**
   ```bash
   git push origin fix/backups-fail-on-vercel
   ```

3. **Deploy to Production:**
   ```bash
   vercel --prod
   ```
   Or merge to main branch if using automatic deployments

4. **Verify deployment:**
   ```bash
   vercel inspect your-domain.com
   ```
   Should show `target: production`

5. **Monitor first cron execution:**
   - Wait up to 5 minutes for next cron run.
   - Check logs for successful execution.
   - Verify backup files are copied.

---

## 📝 Lessons Learned

1. **Never use `NEXT_PUBLIC_*` variables in server-side code**
   - They are for client-side code only.
   - Use regular environment variables or `VERCEL_URL` instead.

2. **Test scheduled jobs in production-like environment**
   - Cron jobs behave differently than manual API calls.
   - Server-side execution has different variable access.

3. **Use proper URL construction utilities**
   - Don't hardcode URLs or assume environment variables are available.
   - Create helper functions that work in all contexts.

4. **Vercel-specific best practices:**
   - Use `VERCEL_URL` for server-side internal API calls.
   - It's automatically set and always correct.
   - No manual configuration needed.

---

## 🔗 Related Documentation

- [Cron Job Deployment Guide](../CRON_JOB_DEPLOYMENT.md).
- [Backup Failure Investigation Guide](../INVESTIGATIONS.md).
- [Next.js Environment Variables](https://nextjs.org/docs/app/building-your-application/configuring/environment-variables).
- [Vercel Environment Variables](https://vercel.com/docs/projects/environment-variables/system-environment-variables).

---

## ✅ Resolution Status

**Status:** FIXED
**Commit:** `ee9a0302`
**Files Changed:** 3 files, 116 insertions(+), 3 deletions(-)
**Ready for Deployment:** YES

**Next Steps:**
1. Deploy to production with `vercel --prod`
2. Monitor first cron execution
3. Verify backup files are copied successfully
4. Close related issues/tickets