# AI Search Architecture This page is the canonical developer overview for StratoFusion AI Search, semantic indexing, and grounded Ask AI. It consolidates the root AI Search planning files into one maintained architecture reference and links out to the narrow module guides for implementation details. ## Consolidated Sources This page supersedes the root-level planning files: - `1.AI_Search_Plan_Architecture_Mapping.md`. - `2.AI_Search_Plan_AI_Semantic_Search_RAG_Gap Analysis.md`. - `3.AI_Search_Plan_AI_Semantic_Search_RAG_Reuse_Analysis.md`. - `4.AI_Search_Plan_Codex_Implementation_Plan.md`. - `5.AI_Search_Plan_Recommended_Sequence_12_PRs.md`. - `6.AI_Search_Plan_Codex_Engineering Spec.md`. - `7.AI_Search_Plan-Augment-AI_Search_Implementation_Plan-Codebase Validation.md`. - `8.AI_Search_Plan-Augment-Architecture_Coding_Review.md`. - `9.AI_Search_Plan_Consolidated_Implementation_Spec.md`. The current implementation details are maintained in: - [AI Search Infra Ports](./AI_SEARCH_INFRA_PORTS.md). - [AI Search Indexing Adapters](./AI_SEARCH_INDEXING_ADAPTERS.md). - [AI Search Grounded Answers](./AI_SEARCH_RAG.md). - [Search Features](../../SEARCH_FEATURES.md). - [Testing](../../TESTING.md). ## Purpose AI Search adds meaning-aware retrieval and grounded question answering over indexed user files without replacing provider-native `basic` or `fulltext` search. The design keeps provider quirks behind adapters, keeps all indexed content scoped by user and connected account, and treats answer generation as a fail-soft layer that can only answer from retrieved citations. ## Current Status AI Search is implemented behind runtime rollout flags and remains disabled by default. The current backend includes: - provider-neutral indexing adapters for Google Drive, OneDrive, and Dropbox. - deterministic extraction for the explicit supported-file allow-list. - chunking, embedding, semantic upsert, and semantic retrieval behind ports. - durable indexing job state, diagnostics, retry, cancel, dismiss, and health. APIs under `/api/search/indexing` - runtime-gated semantic search with keyword fallback. - grounded `POST /api/search/ask` responses that return `no_answer` instead of. fabricating when retrieval, grounding, or generation is insufficient The user-facing AI indexing surface is `/user/ai-indexing`. AI indexing is separate from transfer jobs; backup and sync stay under `/user/jobs`. ## Diagrams These Mermaid diagrams show the implementation from system context down to the critical runtime flows. If a renderer does not support Mermaid, read each block as structured pseudocode for the same architecture. ### System Context AI Search is a retrieval subsystem inside the Next.js app. It reuses the current provider service layer for account-scoped access and keeps semantic infrastructure behind ports. ```mermaid flowchart LR User["Authenticated user"] Browser["Browser UI\n/drive and /user/ai-indexing"] SearchApi["Next.js search APIs\n/api/search/*"] IndexingApi["Next.js indexing APIs\n/api/search/indexing/*"] AskApi["Ask AI API\n/api/search/ask"] Auth["Clerk auth and connected accounts"] Services["Provider service layer\nGoogle Drive, OneDrive, Dropbox"] Providers["Cloud provider APIs"] AiCore["AI search domain\nadapters, extraction, chunking, retrieval"] Database["PostgreSQL\nCompose prod; retained dev DB\njobs, items, indexed files, dirty scopes"] Embeddings["EmbeddingProvider port"] Vector["SemanticIndexPort\nvector backend"] Answer["GroundedAnswerProvider port"] User --> Browser Browser --> SearchApi Browser --> IndexingApi Browser --> AskApi SearchApi --> Auth IndexingApi --> Auth AskApi --> Auth IndexingApi --> AiCore SearchApi --> AiCore AskApi --> AiCore AiCore --> Services Services --> Providers AiCore --> Database AiCore --> Embeddings AiCore --> Vector AiCore --> Answer ``` ### Indexing Pipeline Indexing converts account-scoped provider resources into canonical semantic chunks. Unsupported or failed files produce explicit item outcomes. ```mermaid flowchart TD Start["User or dirty-scope trigger\nrequests indexing"] Gate["Check auth, subscription,\nfeature flags, provider eligibility"] Job["Create ai_indexing_jobs row\nand initial item records"] Ready["Semantic index readiness preflight"] Discover["Provider adapter discovers\ncanonical resources"] Eligible{"Supported by\npolicy?"} Skip["Persist skipped item\nwith sanitized reason"] Download["Download provider bytes\nor export Google Workspace file"] Extract{"Extractable\ntext?"} ExtractFail["Persist failed or skipped item\nwith taxonomy code"] Chunk["Chunk text with token-aware\nchunking strategy"] Embed["Generate embeddings through\nEmbeddingProvider"] Upsert["Upsert vectors through\nSemanticIndexPort"] Indexed["Update ai_indexed_files\nand job-item telemetry"] Complete{"More items?"} Finish["Finalize job status\ncomplete, partial, failed, cancelled"] Start --> Gate --> Job --> Ready --> Discover --> Eligible Eligible -- "No" --> Skip --> Complete Eligible -- "Yes" --> Download --> Extract Extract -- "No" --> ExtractFail --> Complete Extract -- "Yes" --> Chunk --> Embed --> Upsert --> Indexed --> Complete Complete -- "Yes" --> Discover Complete -- "No" --> Finish ``` ### Indexing Job Sequence The UI never indexes directly. It asks the indexing route to create durable work, then watches snapshots or SSE-style job events. ```mermaid sequenceDiagram autonumber participant UI as Browser UI participant Route as Indexing API route participant JobSvc as Job service participant Runner as Backend indexing runner participant Adapter as Provider indexing adapter participant Extractor as Extraction and chunking participant Infra as Embedding and vector ports participant DB as PostgreSQL (Compose in production) UI->>Route: POST /api/search/indexing/jobs Route->>Route: Validate Clerk user and connected account scope Route->>JobSvc: Create scoped indexing job JobSvc->>DB: Insert job and initial status JobSvc->>Runner: Kick resumable in-process execution UI->>Route: GET recent job snapshot or open events stream Runner->>Infra: Preflight semantic index readiness Runner->>Adapter: Discover canonical provider resources Adapter-->>Runner: IndexingDiscoveredItem records Runner->>DB: Persist item status and telemetry Runner->>Adapter: Download or export eligible content Runner->>Extractor: Extract, classify, and chunk text Runner->>Infra: Embed chunks and upsert semantic vectors Runner->>DB: Update indexed-file metadata and item outcome Route-->>UI: Progress, diagnostics, terminal state ``` ### Semantic Retrieval Path AI Search mode reuses the normal search surface and scope filters. If semantic infrastructure is unavailable, the route falls back instead of failing the whole search. ```mermaid sequenceDiagram autonumber participant UI as Search UI participant Route as /api/search/enhanced or stream participant Auth as Auth and scope validation participant Avail as AI availability gate participant Embed as EmbeddingProvider participant Vector as SemanticIndexPort participant Keyword as Provider keyword search participant Merge as Hybrid merge and ranking UI->>Route: Search request with mode ai, query, service, accountId Route->>Auth: Resolve user and connected-account scope Auth-->>Route: Canonical user/service/account filters Route->>Avail: Check subscription, flags, providers, infra config alt AI unavailable Route->>Keyword: Run allowed keyword/full-text fallback Keyword-->>Route: Provider-native results Route-->>UI: Fallback results with AI unavailable context else AI available Route->>Embed: Embed query with timeout and circuit breaker Embed-->>Route: Query embedding or unavailable Route->>Vector: Query vectors scoped by user, service, account Vector-->>Route: Semantic hits or empty/unavailable result Route->>Keyword: Run keyword context when configured Keyword-->>Route: Keyword/provider hits Route->>Merge: Normalize, dedupe, score, and rank Merge-->>Route: Canonical search results with snippets Route-->>UI: AI search results or fallback response end ``` ### Grounded Ask AI Path Ask AI is stricter than semantic search. It only returns an answer when the retrieved evidence is usable and citations can be returned to the user. ```mermaid flowchart TD Question["Question from Ask AI UI"] Scope["Validate user, service,\nand account filters"] RAGGate{"RAG enabled and\ninfra configured?"} Retrieve["Retrieve semantic hits\ninside canonical scope"] Coverage{"Enough grounded\nevidence?"} Generate["Generate answer from retrieved\nexcerpts only"] Cites{"Citations present\nand in scope?"} Answer["Return status answered\nwith citations and snippets"] NoAnswer["Return status no_answer\nwith safe reason"] Question --> Scope --> RAGGate RAGGate -- "No" --> NoAnswer RAGGate -- "Yes" --> Retrieve --> Coverage Coverage -- "No" --> NoAnswer Coverage -- "Yes" --> Generate --> Cites Cites -- "No" --> NoAnswer Cites -- "Yes" --> Answer ``` ### Data Isolation And Storage The stable identity boundary is `(userId, service, accountId, resourceId)`. Vector entries and PostgreSQL rows must agree on that scope. ```mermaid flowchart LR User["userId"] Account["Connected account\nservice + accountId"] Resource["Provider resource\nresourceId"] Job["ai_indexing_jobs"] Item["ai_indexing_job_items"] File["ai_indexed_files"] Dirty["ai_dirty_scopes"] Vector["Semantic vectors\nchunk metadata"] Admin["Admin AI reset\nand hard delete"] User --> Account --> Resource User --> Job Job --> Item Resource --> Item Resource --> File Resource --> Dirty File --> Vector Item --> Vector Admin --> Vector Admin --> Job Admin --> Item Admin --> File Admin --> Dirty ``` ### Fail-Soft Decision Flow AI infrastructure must degrade safely. Search can fall back to keyword results; Ask AI returns `no_answer` rather than an ungrounded response. ```mermaid flowchart TD Request["AI search or Ask AI request"] Flags{"Flags and subscription\nallow AI?"} Providers{"Requested providers\neligible and connected?"} Embed{"Embedding provider\navailable?"} Vector{"Semantic index\navailable?"} Retrieval{"Scoped retrieval\nreturns usable hits?"} Failure{"Failure or\ninsufficient evidence"} SearchFallback["Search response uses\nkeyword/full-text fallback"] NoAnswer["Ask AI returns no_answer\nwith safe reason"] Semantic["Return semantic or hybrid\nsearch results"] Grounded["Generate grounded answer\nwith citations"] Request --> Flags Flags -- "No" --> Failure Flags -- "Yes" --> Providers Providers -- "No" --> Failure Providers -- "Yes" --> Embed Embed -- "No" --> Failure Embed -- "Yes" --> Vector Vector -- "No" --> Failure Vector -- "Yes" --> Retrieval Retrieval -- "No" --> Failure Failure -- "Search" --> SearchFallback Failure -- "Ask AI" --> NoAnswer Retrieval -- "Search + hits" --> Semantic Retrieval -- "Ask AI + hits" --> Grounded ``` ## Architectural Decisions ### Keep Search APIs Under `/api/search` AI Search is an additive search mode, not a separate product silo. Existing search routes keep ownership of search orchestration and reuse the same auth, subscription, quota, service, account, and current-scope filters as the other search modes. ### Reuse The Existing Service Layer The AI pipeline must not introduce a third provider abstraction. Provider discovery and content retrieval flow through narrow AI-specific adapters that wrap the existing provider services and return canonical indexing records. Shared AI code sees canonical fields such as `userId`, `service`, `accountId`, `resourceId`, `parentResourceId`, `mimeType`, `modifiedAt`, `sizeBytes`, `contentHash`, `eligible`, and `skipReason`. ### Preserve The Account Boundary Every durable record and vector operation is scoped by: ```text (userId, service, accountId, resourceId) ``` Shared UI models must not expose provider-native paths, namespace headers, drive IDs, vector backend IDs, or provider-specific pagination details. ### Separate Discovery, Extraction, Chunking, And Retrieval The pipeline is intentionally layered: ```text adapter discovery -> persisted job items -> content download or export -> deterministic extraction -> chunking -> embedding -> semantic upsert -> retrieval -> grounded answer generation ``` Extraction decides whether bytes contain supported text. Chunking decides how that text is prepared for embeddings. Retrieval and RAG only consume indexed chunks and canonical metadata. ### Use Ports For AI Infrastructure Embedding, semantic index, and grounded-answer generation are behind ports. The current concrete adapters support OpenAI embeddings, Weaviate semantic indexing, and OpenAI grounded answers, but shared callers resolve them through factories and fail-soft wrappers. Infrastructure failure returns disabled, unavailable, empty-hit, or `no_answer` results instead of leaking provider exceptions into routes or UI. ### Model Indexing As A State Machine Indexing is long-running, resumable work. Durable job and item state is required for page reloads, stale-run recovery, retry, cancellation, diagnostics, and external mutation reconciliation. Active user controls can cancel, retry, and dismiss visible job history. Broad AI state reset remains admin-only because it deletes semantic vectors and PostgreSQL metadata. ## Module Layout Primary implementation areas: - `src/lib/search/ai/adapters/` - provider-specific indexing adapters and. user-scoped service bridges - `src/lib/search/ai/extraction/` - deterministic file extraction, Google. Workspace export routing, OCR support, and supported-file policy - `src/lib/search/ai/chunking/` - token-aware chunking contracts and utilities. - `src/lib/search/ai/indexing/` - job lifecycle, runner, progress writing. diagnostics, reconciliation, and mutation dirty-scope handling - `src/lib/search/ai/providers/` - embedding and semantic index ports. factories, fail-soft wrappers, and concrete adapters - `src/lib/search/ai/retrieval/` - grounded answer service and index coverage. - `src/app/api/search/indexing/` - thin indexing API routes. - `src/app/api/search/ask/route.ts` - grounded Ask AI API route. - `src/hooks/useAiSearchAvailability.ts`, `src/hooks/useAiIndexingJob.ts`. `src/hooks/useAskAi.ts` - UI data hooks - `src/components/search/` - AI search, indexing, and Ask AI rendering. ## Rollout Gates AI Search has independent runtime gates: - subscription access to AI Search. - global search/indexing flags such as `AI_SEARCH_ENABLED`. - provider-specific flags for Google Drive, OneDrive, and Dropbox. - embedding provider configuration. - semantic index backend configuration. - grounded-answer provider configuration. - `AI_RAG_ENABLED` for Ask AI. - OCR-specific flags for standalone PNG/JPEG OCR. Keep indexing, semantic retrieval, OCR, and grounded answers independently disableable. Internal rollout order remains: The scheduled external-change reconciliation path obeys the same indexing gate. When `AI_INDEXING_ENABLED` is disabled, its authenticated production cron route is a successful no-op and must not refresh provider tokens or enumerate provider accounts. 1. Enable indexing for internal accounts and supported providers. 2. Validate semantic retrieval quality and fallback behavior. 3. Enable Ask AI only after citation quality is acceptable. 4. Expand file types or provider scopes behind targeted tests and rollback flags. ## Supported Content Policy Indexing is intentionally limited to deterministic formats: - plain text, Markdown/MDX, source code, JSON/YAML/XML/CSV/TSV/HTML/SVG, and. similar text-like formats - PDFs with extractable embedded text. - DOCX files with extractable plain text. - XLSX visible worksheet values. - PPTX visible slide text and speaker notes. - Google Docs, Sheets, Slides, and Drawings exported by Google Drive before. extraction - standalone PNG/JPEG images only when OCR is enabled. Unsupported, encrypted, corrupt, empty, oversized, unknown, or policy-skipped files must be explicit item outcomes, not silent successes. OpenXML extraction validates package metadata before parser dispatch. The guard bounds entry count, per-entry and aggregate expansion, and compression ratio, and rejects encrypted entries, ZIP64 size markers, absolute/traversal entry names, and inconsistent central-directory metadata. Parser errors remain sanitized and document contents are never logged. ## Risk Controls The highest-risk areas are: - multi-tenant vector isolation. - stale vectors after file updates, moves, deletes, account disconnects, or user. deletion - provider token expiry during background work. - provider rate limits and partial provider outages. - embedding and vector query cost growth. - unsupported or low-quality extraction being misrepresented as searchable. - answers without citations being mistaken for grounded results. Required mitigations: - scope all vector filters by `userId`, `service`, and `accountId`. - clear existing vectors for a file before replacement upserts. - reconcile dirty scopes after uploads, copies, moves, renames, deletes, backup. sync, and provider change polling - fail fast when semantic infrastructure is unavailable. - return keyword fallback or `no_answer` instead of fabricating semantic results. - sanitize provider, parser, vector, and model errors before user display. - never log document content, OAuth tokens, raw provider errors, or raw extracted. text ## Test Focus Unit tests should cover extraction policies, chunking, state transitions, adapter metadata normalization, score merging, citation formatting, and fail-soft adapter behavior. Integration tests should cover indexing job lifecycle, provider adapter discovery/download, user and account isolation, stale-index replacement, semantic fallback, Ask AI `no_answer`, and route authorization. Frontend tests should cover mode gating, indexing status rendering, diagnostics, retry/cancel/dismiss controls, citation cards, mobile search controls, and upgrade/paywall states. Regression tests must continue to prove that `basic`, `fulltext`, direct downloads, uploads, backup, sync, and rclone transfer flows are unaffected by AI Search rollout.