# Transfer Filter Configuration

This document describes the file filtering system that prevents problematic files from being transferred between cloud storage services using rclone.

## Overview

The transfer filter system automatically identifies and skips files that rclone cannot properly transfer between cloud storage services. This prevents transfer failures, data corruption, and unexpected behavior.

## Configuration File

The filter rules are defined in `src/config/transfer-filters.ts`. This file contains:

- **Filter Rule Definitions**: Structured rules that define which files to skip.
- **Categories**: Logical groupings of related filter rules.
- **Utility Functions**: Helper functions for working with filter rules.

## Filter Rule Types

### 1. Extension-based Filters
Skip files based on their file extension:
```typescript
{
  type: 'extension',
  value: 'lnk',
  reason: 'Windows .lnk shortcut files are not preserved as shortcuts during transfer',
  category: 'Symbolic Links & Shortcuts',
}
```

### 2. MIME Type Filters
Skip files based on their MIME type:
```typescript
{
  type: 'mimeType',
  value: 'application/vnd.google-apps.form',
  reason: 'Google Forms cannot be exported or transferred to other services',
  category: 'Google Workspace Non-Exportable',
}
```

### 3. Filename Filters
Skip files with specific names:
```typescript
{
  type: 'filename',
  value: '.DS_Store',
  reason: 'macOS metadata files are not useful on other platforms',
  category: 'Platform-Specific Files',
}
```

### 4. Pattern Filters
Skip files matching regex patterns:
```typescript
{
  type: 'pattern',
  value: '.*\\._.*',
  reason: 'macOS resource fork files (._filename) lose their special properties during transfer',
  category: 'Platform-Specific Files',
}
```

## Filter Categories

### Symbolic Links & Shortcuts
- Windows `.lnk` shortcut files.
- macOS `.alias` files.
- Internet `.url` shortcut files.
- Rclone `.rclonelink` placeholder files.

### Platform-Specific Files
- iCloud Drive `.icloud` placeholder files.
- macOS `.DS_Store` metadata files.
- Windows `Thumbs.db` and `desktop.ini` files.
- macOS resource fork files (`._filename`).

### Google Workspace Non-Exportable
- Google Forms (`.gform`).
- Google My Maps (`.gmap`).
- Google Sites (`.gsite`).
- Google Drive shortcuts.

### System & Temporary Files
- Hidden files starting with `.`.
- Temporary files (`.tmp`, `.temp`).
- Backup files ending with `~`.

### Special Cases
- Files with empty names.
- Files with control characters in names.

## Usage

### Checking Individual Files

```typescript
import { shouldSkipFile } from '~/lib/rclone/utils/transfer-filter';

const fileInfo = {
  fileName: 'document.lnk',
  mimeType: 'application/x-ms-shortcut',
};

const result = shouldSkipFile(fileInfo);
if (result.shouldSkip) {
  console.log(`Skipping file: ${result.reason}`);
}
```

### Filtering File Arrays

```typescript
import { filterTransferableFiles } from '~/lib/rclone/utils/transfer-filter';

const files = [
  { fileName: 'document.pdf' },
  { fileName: 'shortcut.lnk' },
  { fileName: 'image.jpg' },
];

const { transferable, skipped } = filterTransferableFiles(files);
console.log(`${transferable.length} files can be transferred`);
console.log(`${skipped.length} files will be skipped`);
```

### Getting Filter Summary

```typescript
import { getFilterSummary } from '~/lib/rclone/utils/transfer-filter';

const summary = getFilterSummary(files);
console.log(`Total: ${summary.totalFiles}`);
console.log(`Transferable: ${summary.transferableFiles}`);
console.log(`Skipped: ${summary.skippedFiles}`);
console.log('Skipped by category:', summary.skippedByCategory);
```

## Integration with Transfer Service

The filter system is automatically integrated into the rclone transfer service:

### Single File Transfers
- Files are validated before transfer.
- Unsupported files throw an error with detailed reason.

### Batch File Transfers
- Unsupported files are automatically filtered out.
- Transfer continues with supported files.
- Skipped files are reported in the result.

### Folder Transfers
- Individual files within folders are filtered during transfer.
- Folder structure is preserved for supported files.

## Adding New Filter Rules

To add new filter rules, edit `src/config/transfer-filters.ts`:

1. Add the new rule to the `TRANSFER_FILTER_RULES` array
2. Choose the appropriate rule type (`extension`, `mimeType`, `filename`, or `pattern`)
3. Provide a clear, descriptive reason
4. Assign to an appropriate category
5. Add unit tests in `src/lib/rclone/utils/__tests__/transfer-filter.test.ts`

Example:
```typescript
{
  type: 'extension',
  value: 'newext',
  reason: 'Files with .newext extension cannot be properly transferred',
  category: 'Special Cases',
}
```

## Testing

Run the filter tests:
```bash
pnpm test src/lib/rclone/utils/__tests__/transfer-filter.test.ts
```

## Troubleshooting

### File Unexpectedly Skipped
1. Check the filter rules in `transfer-filters.ts`
2. Verify the file extension, MIME type, or filename
3. Check if any pattern rules match the filename

### File Should Be Skipped But Isn't
1. Verify the filter rule is correctly defined
2. Check the rule type matches the file property
3. Ensure case sensitivity settings are correct

### Performance Issues
- Filter rules are checked in order - put most common rules first.
- Avoid complex regex patterns when possible.
- Consider caching filter results for large file sets.

## References

- [Rclone Documentation](https://rclone.org/overview/).
- [Rclone Limitations](https://rclone.org/overview/#limitations).
- [Cloud Storage Service Limitations](https://rclone.org/overview/#supported-providers).