Production currently runs on the OVHcloud VM Compose stack. The previous
Vercel control plane, Neon production database, and Fly.io rclone deployment
remain documented as legacy rollback and migration context; they are not the
authoritative production runtime. Rehearsal hostnames, cutover guards, and
restore scripts are transitional compatibility mechanisms, not a fourth
environment.
See Architecture Evolution And Runtime Status
for the verified current VM model, the previous Fly.io container model, a
side-by-side operational comparison, migration constraints, and explicitly
planned or unimplemented architecture.
The product boundary is unchanged by the hosting migration: Next.js owns the
control plane, provider services and adapters normalize provider behavior, and
the hosting-neutral rclone worker owns byte-heavy data-plane execution. Some
code and environment names retain Fly/FLYIO for compatibility.
Transfer Module Boundaries
BatchFileTransferDialog is a render-only entry point. src/components/batch-transfer/BatchTransfer*.tsx render the dialog sections; src/hooks/batch-transfer/useBatchTransferController.ts composes state, navigation, cancellation, recovery, completion, and scheduled-job hooks. File and folder execution and compatibility checks live under src/lib/transfers/batch/, behind typed progress/tracking callbacks. The existing transfer service remains the external API boundary.
The worker sync route validates requests and starts operations. syncCommand.js prepares the existing flags and bisync workdir, syncLifecycle.js preserves terminal status and watchdog behavior, and syncProcessEvents.js owns stream/event handling and state persistence. The route retains its existing exports for callers and tests. This separation does not change copy, move, backup, sync, or bisync semantics.
System Audit Log Access
/admin/system-logs is the production system audit viewer. Middleware and
the admin server layout require an admin role and recent MFA verification.
The existing /api/logs/audit endpoint retains its server-side admin check.
The exact legacy /dev/system-logs URL redirects to the admin route before
the nonproduction guard; all other development surfaces remain restricted.
The viewer delegates client state to useSystemLogs and renders its table
and details through focused log components.
Service Abstraction Layer
Overview
Stratofusion implements a comprehensive Service Abstraction Layer that provides a unified interface for all cloud storage providers (Google Drive, OneDrive, Dropbox, Box, pCloud, Jupiter).
Core Components
1. Type Definitions (src/types/cloud-storage.ts)
CloudStorageService Interface: Unified interface for all cloud storage operations.
ServiceCapabilities Interface: Defines what features each service supports.
Response Types: Standardized response formats for all operations.
Delete Outcome Model: src/domain/models/DeleteOutcome.ts requires provider delete adapters to report the operation actually completed (trash or permanent); API and UI layers must not infer disposition from the requested mode.
Error Handling: Comprehensive error codes and error class.
Configuration Types: Service configuration and OAuth setup types.
2. Base Abstract Class (src/services/base/BaseCloudStorageService.ts)
Common OAuth Logic: Handles authentication flow for all services.
Token Management: Automatic token refresh and validation.
Error Handling: Consistent error handling and retry mechanisms.
Request Utilities: Authenticated API request helpers.
Abstract Methods: Enforces implementation of required methods.
3. Service Registry System (src/services/ServiceRegistry.ts)
ServiceFactory: Standardized creation of service instances using a loader pattern.
Dynamic Registration: configuration-driven service registration that avoids static circular dependencies.
ServiceRegistry: Manages service instance lifecycle and discovery via getService().
Health Monitoring: Integrated ServiceHealthMonitor for real-time status tracking.
Global Access: Lazy-initialized singleton via getServiceRegistry().
Delete-capable providers are registered through an exhaustive typed handler map in
src/app/api/delete/delete-executor.ts. Extending the supported delete-service
union therefore requires a handler that returns the canonical actual outcome.
The executor derives legacy compatibility fields such as trashedOnly from that
outcome so provider fallbacks cannot be reported as permanent deletion.
Service Implementations
Fully Implemented Services:
GoogleDriveService: Complete Google Drive API v3 integration.
OneDriveService: Complete Microsoft Graph API integration.
DropboxService: Complete Dropbox API v2 integration.
Service Capabilities Matrix:
Feature
Google Drive
OneDrive
Dropbox
Box
pCloud
Jupiter
File Operations
✅
✅
✅
🚧
🚧
🚧
Folder Operations
✅
✅
✅
🚧
🚧
🚧
Search
✅
✅
✅
🚧
🚧
🚧
Upload
✅
✅
✅
🚧
🚧
🚧
Download
✅
✅
✅
🚧
🚧
🚧
Standardized API Response Format
Core Response Interface
All API endpoints return responses following the ApiResponse<T> interface:
AuthenticationAdapter: Unified OAuth flows across all services.
Token refresh and validation handling.
Multi-account authentication state management.
Conflict Resolution Architecture
Purpose
Provide consistent transfer conflict handling across providers and operations (copy/move/batch/backup launchers), while keeping UI policy choices provider-agnostic.
Building Blocks
Policy Types: src/types/transfer-options.ts.
skip | overwrite | rename | ask plus concrete normalization for backend execution.
Name Resolution Utilities: src/lib/utils/file-naming.ts.
Case-insensitive unique naming (name (1).ext, name (2).ext) and batch-safe planning.
Uses ServiceRegistry + canonical list APIs to inspect destination contents.
Execution Layers.
src/lib/rclone/services/copy-service.ts.
src/lib/rclone/services/move-service.ts.
src/lib/rclone/services/folder-copy-service.ts (supports skip via ignore-existing).
Execution Flow
UI/API selects a policy (rename default for copy/move).
Service layer normalizes policy (ask -> rename for non-interactive APIs).
For rename, destination names are pre-resolved in batch per (service, account, folder).
For skip, rclone requests include --ignore-existing.
The rclone worker executes copy/move with resolved destination naming. In
production this worker runs in the VM Compose stack; Fly.io is the stopped
legacy recovery deployment.
Notes
Backup transfer launcher applies skip by default.
Move command pathing now supports explicit target filenames (moveto semantics when fileName is supplied), enabling rename-on-conflict behavior.
Ports & Adapters (Hexagonal) Architecture
Stratofusion is transitioning to a Ports & Adapters architecture to enhance modularity and testability.
Core Ports (src/domain/ports/)
CloudStoragePort: The primary entry point for storage operations.
AuthPort: Interface for authentication and token management.
FilePort / FolderPort: Specialized interfaces for filesystem operations.
HierarchyPort: Interface for retrieving organizational structures (Shared Drives, Sites, Namespaces).
SearchPort: Interface for cross-provider search operations.
Unified Hierarchy Port
Phase 12 consolidated provider-specific hierarchy routes into a unified /api/hierarchy endpoint.
Purpose: Provides a single interface to retrieve the top-level organizational structure (e.g., Google Shared Drives, OneDrive Sites, Dropbox Team Folders).
Consolidation: Replaces /api/google/folder-hierarchy, /api/onedrive/folder-hierarchy, etc.
Capability Check: Uses service.getAccountHierarchy to determine if a service supports organizational structure exploration.
AI Semantic Search Cleanup
For the full AI Search architecture, rollout gates, and source consolidation
record, see AI Search Architecture.
AI indexing stores durable state in both Neon and the configured semantic vector
backend. Admin cleanup uses src/lib/admin/reset-user-ai-search.ts as the
single service for manual reset and hard-delete integration.
Active in-process indexing executions are aborted before cleanup.
Vector entries are deleted through the semantic index port with a user-scoped.
filter before any Neon AI rows are removed.
Neon cleanup removes AI indexing job items, jobs, indexed-file metadata, dirty.
scopes, and reconciliation cursors for the target user.
If vector deletion fails or the vector backend is disabled, Neon cleanup and.
user hard delete abort so stale vectors cannot outlive their database owner.
Successful manual resets emit a system audit log; hard delete includes the AI.
reset result in its existing admin deletion audit payload.
User-facing AI indexing control is intentionally narrower than admin cleanup.
/user/ai-indexing starts from a REST snapshot, then uses
/api/search/indexing/jobs/events for fresh progress while active jobs are
visible. The browser falls back to recent-job polling if SSE is unavailable and
stops live updates once the visible jobs are terminal. User controls can cancel
active jobs, retry failed/cancelled/partial jobs by creating a new scoped job,
and dismiss terminal job history with a dismissed_at marker. These controls
never delete job rows, semantic vectors, or indexed-file metadata; broad AI
search reset remains an admin-only operation.
AI indexing observability is built as a service layer under
src/lib/search/ai/indexing. The runner persists item telemetry through the
same progress-writer/database path that updates item status, so status and
diagnostics do not drift apart. Run-specific facts live on
ai_indexing_job_items; the latest file-facing facts needed for status and
coverage diagnostics live on ai_indexed_files. API routes under
/api/search/indexing stay thin: authenticate, validate connected account
ownership and resource identity, call diagnostics services, and map responses.
Client components remain render-focused; hooks fetch diagnostics and health
payloads, while aggregation and taxonomy mapping stay in indexing services.
The runner also performs a semantic-index readiness preflight before provider
discovery so a missing or misrouted vector backend fails the job with a single
sanitized infrastructure reason instead of producing hundreds of misleading
item failures.
Dashboard and status reads load only indexing metadata. The execution runner
and PDF parser are deferred until an indexing job or PDF extraction actually
runs, so optional native document-processing globals cannot break unrelated
server-rendered routes.
State Management Architecture
Context Providers:
ServiceManagerContext: Global service state and multi-service operations.
FileSelectionContext: File selection and bulk operations with selection limits.
DriveNavigationContext: Folder navigation, breadcrumbs, and URL-based navigation.
User Operations Dashboard (/user/dashboard)
Server-rendered route with Clerk auth guard and section-level graceful degradation.
Keeps SubscriptionUsageDashboard on the main dashboard and adds a server-rendered QuotaWarningBanner when enforced provider limits reach 80%+.
Renders the full StorageAvailabilityDashboard as the first primary /user/dashboard card after quota warnings, including Bars, Donut, and Table views over canonical account quota data. The Drive route keeps only a compact summary link so file browsing stays operationally focused.
New dashboard cards (ServiceHealthSummary, RecentActivityCard, ActiveJobsCard, QuickActionsCard) consume normalized server data from src/lib/dashboard/dashboard-data.ts.
ServiceHealthSummary keeps healthy accounts passive and only surfaces reconnect actions when an account is expired or offline. Dropbox accounts also run a live account-access check through src/lib/storage/account-access-health.ts, so cancelled, disabled, downgraded, or token-bound-to-a-different-account Dropbox connections appear as account problems instead of healthy provider uptime.
Storage overview cards consume canonical quota and account-health data. Aggregate usage percentages are derived from totalUsed / totalAvailable at render time so stale or inconsistent cached percentage fields cannot contradict the displayed byte totals.
Dropbox space usage is normalized in src/lib/storage/dropbox-space-usage.ts. Individual allocations use the account's usage and capacity; team allocations pair member-specific usage with member limits, or team-wide usage with shared team capacity when no member limit exists.
Data loading limits: activity and jobs are capped at 10 records per section to protect render and query cost.
Transfer Quotas View (/user/quotas)
Dedicated authenticated route for power users who need the complete provider transfer quota breakdown.
Reuses TransferQuotaDashboard so the detailed quota card logic remains centralized in one client component.
Linked from the dashboard warning banner and the user settings menu to reduce dashboard noise without removing discoverability.
Purpose: Sticky cancellation guard with optional toasts.
Features.
Prevents accidental job cancellation.
Confirmation dialogs.
Toast notifications for user feedback.
Auto-Connection Attempt Tracking: Migration to Database
Overview
Auto-connection attempt tracking moved from session storage to the database to ensure cross-device persistence and per-service behavior.
Key Changes
Database: Added fields in user preferences to record whether auto-connection was attempted and when.
Database service: Introduced helper functions to check, mark, and reset attempt state.
State management: ServiceManagerContext now queries the database (with lightweight caching) instead of session storage.
Logic: Auto-connection now runs per unconnected service rather than being blocked globally by any existing connection.
Benefits
Cross-device/session persistence of attempt state.
Respect for user intent with per-service checks.
Better UX for users adding additional OAuth providers later.
Implementation Notes
Backward compatible for existing users (defaults handled safely).
Async calls wrapped with robust error handling and cache invalidation on user change.
Works alongside manual connection/disconnection flows without regressions.
Authentication & OAuth
OAuth Flow:
Server-side authentication through API routes (src/app/api/auth/).
Token management and refresh handled by BaseCloudStorageService.
Multi-account support with accountId pattern (default, custom IDs).
Dynamic redirect URIs based on NEXT_PUBLIC_SITE_URL environment variable.
Clerk Integration:
User authentication and management using Clerk.
UserButton component for integrated user profile management.
Session management and user metadata storage.
Performance Optimizations
Client-Side Features:
Client-side pagination with configurable page sizes.
Virtualization support for large file lists.
Memory usage tracking and progressive loading.
Folder structure preservation with race condition prevention.
Caching Strategy:
Service response caching with TTL.
File metadata caching.
API response caching with invalidation.
Memory-efficient data structures.
Admin Infrastructure Overview
/admin/infrastructure is a read-only, admin-authorized projection over the
authoritative VM's existing observability sources. Canonical fleet-shaped
models live under src/domain/models, while infrastructure metrics and health
ports isolate collection from aggregation. The Prometheus adapter is
server-only and uses the internal Compose address; the browser receives only
sanitized metrics, status, freshness, warnings, release metadata, and a
server-resolved management-link catalog. Each link is attached to one
allowlisted service and classified as same-origin, public, or private before a
render-only component displays it. Phase-one actions lead only to existing
StratoFusion operations pages or allowlisted Grafana and GlitchTip origins.
The overview aggregates Prometheus queries, application/database readiness,
the VM rclone worker health check, and operational telemetry independently.
Missing, malformed, stale, or timed-out observations become explicit
unavailable or degraded sections; they are never converted to zero or allowed
to erase other successful data. The API has no mutation methods and has no
access to the Docker socket, SSH, systemd, OVHcloud lifecycle APIs, private
Cockpit URL, or host commands.
Management links are navigation affordances, not control-plane capabilities.
Their resolver accepts only fixed same-origin routes and explicitly configured,
credential-free HTTPS origins in the stratofusion.io domain family. The UI
opens them in a new tab with opener isolation. A missing or rejected URL is
rendered as not configured; it never falls back to a guessed hostname.
pgAdmin and RedisInsight are optional Compose services in the management
profile. Their host ports bind to 127.0.0.1 only, their state uses dedicated
volumes, and neither production nor local Caddy has a route for them. When the
profile is running, the allowlisted telemetry catalog adds their aggregate
replica state. When it is disabled, the optional rows are omitted rather than
reported as failures. PostgreSQL and pgAdmin rows share the pgAdmin action;
Redis and RedisInsight rows share the RedisInsight action. Operators reach the
tools through an SSH local forward or an explicitly configured private
Tailscale HTTPS endpoint.
Operational telemetry uses a separate two-process boundary:
operations-collector is the only container that mounts the Docker socket.
It has network_mode: none, accepts no requests, performs one fixed read of
/containers/json, filters to the stratofusion Compose project and an
explicit service allowlist, and writes only replica/health counts plus a
collection timestamp to a dedicated volume.
operations-exporter runs unprivileged on an internal scrape-only network.
It has no Docker socket and exposes the sanitized Compose counts plus numeric
backup attempt/success/status timestamps from the shared telemetry volume.
Prometheus scrapes the exporter. Next.js continues to query only Prometheus
and maps the facts through InfrastructureOperationsPort before rendering.
The Docker socket remains a privileged host capability even when bind-mounted
read-only, because HTTP method permissions are not constrained by the mount
flag. The collector therefore has no network or input surface, a read-only root
filesystem, all Linux capabilities dropped, no-new-privileges, a PID limit,
fixed request code, and an allowlisted output schema. Container IDs, names,
images, environment variables, logs, mounts, raw backup paths, and Docker API
responses never cross the collector boundary.
Deployment, rollback, restart, terminal, and VM power controls remain outside
this read-only boundary. OVHcloud inventory/lifecycle visibility still requires
a separate least-privilege adapter; any action requires an independently
approved authorization and audit design.
Testing Strategy
Test Coverage:
Unit Tests: Hooks, utilities, service logic (Vitest + React Testing Library).
Component Tests: UI components with user interaction testing.
Integration Tests: API route testing with service mocking.
Storybook: Visual component documentation and testing.
270+ Tests: Comprehensive test suite covering all major functionality.
Test Organization:
Tests live relatively near code, mirror main app structure.
Service-specific test suites for each cloud storage provider.
Mock providers for complex contexts in Storybook.
E2E tests with real authentication credentials.
Development Guidelines
Code Organization:
Never create files longer than 500 lines - refactor by splitting into modules.
Organize code into clearly separated modules grouped by feature/responsibility.
Mirror test structure to main app structure.
Type Safety:
TypeScript strict mode for type safety.
Reject 'any' types in TypeScript code and tests.
Comprehensive interfaces and types in dedicated types folders.
Quality Assurance:
ESLint + Prettier for code quality.
Pre-commit hooks with lint, typecheck, and test.
Storybook for component documentation and visual testing.
UI Components & User Experience
OneDrive Warning Component
Environment-controlled warning component that informs users about known Microsoft OneDrive search issues:
Features:
Environment-controlled: Enabled/disabled via NEXT_PUBLIC_SHOW_ONEDRIVE_WARNING.
User-dismissible: Users can close the warning message.
Responsive design: Works on mobile and desktop.
Dark mode support: Automatically adapts to themes.
External link: Links to Microsoft's official documentation.
Display Conditions:
Environment variable set to "true".
User is authenticated.
OneDrive service is connected.
User has performed a search.
User hasn't dismissed the warning in current session.
The job management system implements a clean layered architecture that separates HTTP handling, business logic, and data transformation. This architecture supports both backup jobs and sync jobs through a unified interface.
Unified service layer that abstracts away differences between backup and sync jobs.
Key Functions:
/**
* Get a job by ID (tries both backup and sync tables)
*/exportasyncfunctiongetJobById( userId:string, jobId:string,):Promise<JobResult>;/**
* Update a job (routes to appropriate update function based on job type)
*/exportasyncfunctionupdateJob( userId:string, jobId:string, input:UpdateBackupJobInput|UpdateSyncJobInput,):Promise<JobResult>;/**
* Cancel a job (tries both backup and sync tables)
*/exportasyncfunctioncancelJob( userId:string, jobId:string,):Promise<JobResult>;/**
* Delete a job (tries both backup and sync tables)
*/exportasyncfunctiondeleteJob( userId:string, jobId:string,):Promise<boolean>;
Features:
Automatic job type detection.
Unified error handling.
Security checks enforced (userId validation).
Type-safe interfaces.
Consistent return types.
Job Type Detection Strategy:
GET/DELETE/CANCEL operations: Try backup table first, then sync table.
PATCH operations: Detect by presence of mode field in request body.
2. Job Mappers (src/lib/mappers/job-mappers.ts)
Centralized response transformation logic for converting database records to API responses.
Key Functions:
/**
* Map a backup job to list response format
*/exportfunctionmapBackupJobToListItem(job:BackupJob):ListJobsResponseItem;/**
* Map a sync job to list response format
*/exportfunctionmapSyncJobToListItem(job:SyncJob):ListJobsResponseItem;/**
* Map a backup job to detailed response format
*/exportfunctionmapBackupJobToResponse(job:BackupJob):GetBackupJobResponse;/**
* Map a sync job to detailed response format
*/exportfunctionmapSyncJobToResponse(job:SyncJob):GetSyncJobResponse;
Features:
Consistent date formatting (ISO 8601).
Null safety (handles missing fields gracefully).
Type-safe transformations.
Reusable across all endpoints.
Single source of truth for response formats.
3. API Routes (HTTP Layer)
Thin HTTP handlers that delegate to the service layer.
Example: GET /api/jobs/[id]
exportasyncfunctionGET( request:NextRequest,{ params }:{ params:{ id:string}},){const userId =awaitgetUserId();const jobId = params.id;// Delegate to service layerconst result =awaitgetJobById(userId, jobId);if(!result){returnNextResponse.json({ success:false, error:"Job not found"},{ status:404},);}// Use mapper to transform responseconst response = result.type==="backup"?mapBackupJobToResponse(result):mapSyncJobToResponse(result);returnNextResponse.json({ success:true, data: response });}
Benefits of Layered Architecture
Separation of Concerns ✅
HTTP handling separated from business logic.
Business logic separated from data transformation.
Each layer has a single, clear responsibility.
Code Reusability ✅
Service functions used by multiple API routes.
Mappers used by multiple endpoints.
No code duplication.
Testability ✅
Each layer can be tested independently.
Service layer can be unit tested without HTTP.
Mappers can be tested with mock data.
Maintainability ✅
Changes to business logic in one place.
Changes to response format in one place.
Easy to understand and debug.
Extensibility ✅
Easy to add new job types.
Easy to add new operations.
Open/Closed Principle (open for extension, closed for modification).
Code Quality Improvements
Before Refactoring:
350 lines across 3 API routes.
~60% code duplication.
Business logic mixed with HTTP handling.
Hard to test and extend.
After Refactoring:
174 lines across 3 API routes.
280 lines in service/mapper layers.
~5% code duplication.
Each layer independently testable.
Easy to extend with new job types.
Total Code Reduction: 176 lines (-42%)
Usage Example
// In API routeimport{ getJobById, updateJob, cancelJob, deleteJob,}from"@/lib/services/job-service";import{ mapBackupJobToResponse, mapSyncJobToResponse,}from"@/lib/mappers/job-mappers";// Get jobconst result =awaitgetJobById(userId, jobId);// Update jobconst updated =awaitupdateJob(userId, jobId,{ schedule:"daily", scheduledTime:"08:00", timezone:"Australia/Sydney",});// Cancel jobconst cancelled =awaitcancelJob(userId, jobId);// Delete jobconst deleted =awaitdeleteJob(userId, jobId);
Client-side inactivity monitoring with warning and forced logout handoff.
Cross-tab synchronization via BroadcastChannel/localStorage fallback.
Operation-aware timeout gating to avoid logout while transfers/uploads are active.
The browser logout use case clears service/search state, localStorage, and
sessionStorage, attempts authenticated server-side provider-token cleanup,
and then invalidates the Clerk session. A signed-in-to-signed-out observer is
a second browser-storage cleanup boundary for Clerk-hosted sign-out controls.
Server logout deletes only encrypted provider-token rows. Connected-account
metadata and user preferences are durable configuration and remain intact;
explicit disconnect and hard-delete flows own those destructive mutations.
Administrative MFA boundary
src/lib/auth/admin-access-policy.ts is the canonical admin role and MFA
decision. It treats a signed Clerk v2 fva[1] >= 0 claim as second-factor
evidence and fails closed on absent, malformed, or -1 values.
src/app/admin/layout.tsx protects the complete admin page subtree at the
resource boundary. Each /api/admin handler independently applies the same
policy, while middleware rejects non-compliant requests early.
Clerk dashboard enrollment and production evidence remain operator-owned;
the application does not silently weaken access when that configuration is
incomplete.
API request-integrity and ownership boundaries
Middleware evaluates every mutating /api/* request through
src/lib/auth/request-integrity.ts before public-route or Clerk routing.
Browser calls require exact same-origin evidence. Cookie-authenticated calls
without Origin or Fetch Metadata fail closed; sibling origins are not
trusted merely because they are same-site.
The target origin uses the direct Host authority and the request URL's
scheme. Caddy preserves the public Host and supplies the scheme consumed by
Next.js; the standalone middleware URL's internal listening host/port is not
the browser's origin. Malformed Host values fail origin comparison, and
X-Forwarded-Host cannot override the target authority.
Credentialless machine calls proceed to their route-specific webhook
signature, cron bearer, or rclone service authentication. Request integrity
supplements those controls and never replaces them.
Resource services and persistence calls take the authenticated userId as a
boundary. Route tests use two identities with reused resource identifiers to
verify account, job, transfer-operation, and AI-indexing isolation.