Loading application...

StratoFusion

📚 Documentation

← Back to App

Documentation/AI Search Indexing Adapters

AI Search Indexing Adapters

Source: developer/modules/AI_SEARCH_INDEXING_ADAPTERS.md

AI Search Indexing Adapters

This module guide covers the provider-neutral indexing adapter boundary used by the backend AI indexing runner.

For the cross-cutting AI Search architecture, rollout gates, and risk controls, start with AI Search Architecture.

Files

src/lib/search/ai/adapters/IndexingProviderAdapter.ts.
src/lib/search/ai/adapters/google-drive-indexing-adapter.ts.
src/lib/search/ai/adapters/google-drive-indexing-adapter.test.ts.
src/lib/search/ai/adapters/onedrive-indexing-adapter.ts.
src/lib/search/ai/adapters/onedrive-indexing-adapter.test.ts.
src/lib/search/ai/adapters/dropbox-indexing-adapter.ts.
src/lib/search/ai/adapters/dropbox-indexing-adapter.test.ts.
src/lib/search/ai/adapters/user-scoped-indexing-services.ts.
src/lib/search/ai/adapters/user-scoped-google-drive-indexing-service.ts.
src/lib/search/ai/adapters/user-scoped-onedrive-indexing-service.ts.
src/lib/search/ai/adapters/user-scoped-dropbox-indexing-service.ts.
src/lib/search/ai/adapters/user-scoped-indexing-services.test.ts.
src/lib/search/ai/adapters/adapter-utils.ts.
src/lib/search/ai/indexing/provider-resolver.ts.
src/lib/search/ai/indexing/backend-indexing-runner.ts.

Goals

Keep provider-specific discovery and download logic out of shared indexing code.
Reuse the existing service layer and ServiceRegistry instead of widening .

CloudStoragePort

Return only canonical indexing metadata to shared AI search modules.

Make unsupported files explicit so later runner work can skip them honestly.

Contract

IndexingProviderAdapter

identifies its provider through service and adapterId.
discovers canonical IndexingDiscoveredItem records for an IndexingScope.
downloads canonical extraction input for eligible items.
returns kind: "unsupported" instead of faking a successful content fetch.

Google Drive V1 Behavior

Account scope discovery traverses My Drive plus visible Shared Drive roots.
Hidden Shared Drives are skipped to preserve current container visibility behavior.
Folder scope discovery accepts canonical Google resourceId values, including root.
Discovery returns canonical metadata only. service, accountId, resourceId, resourceName, parentResourceId, mimeType, modifiedAt, sizeBytes, contentHash, eligible, skipReason
contentHash stays null for now because the existing Google service layer does not expose a stable file hash in its canonical response models.

Supported vs Skipped

Google Drive content download for indexing is intentionally limited to the deterministic AI extraction allow-list:

supported: plain text, markdown/MDX, source code/scripts, structured text such as JSON/YAML/XML/CSV/TSV/HTML/SVG, PDFs with embedded text, DOCX files with extractable text, XLSX visible worksheet values, PPTX slide text plus speaker notes, standalone PNG/JPEG images when OCR is enabled, and Google Workspace Docs/Sheets/Slides/Drawings exported from Google Drive before extraction.
skipped: unsupported Google Workspace types such as Forms, Sites, Maps, Fusion Tables, Jamboard, Apps Script, Shortcuts, and third-party Drive app files; legacy Office formats; non-PPTX presentations; media/archives; unknown formats; and non-indexable encrypted/empty/corrupt parser results.
skipped or failed: files whose downloaded or exported bytes exceed the applicable extraction policy limit.

Google Workspace files are handled by the adapter before shared extraction:

Google Docs export through Drive API files.export() as DOCX, then route to the DOCX extractor.
Google Sheets export as XLSX, then route to the XLSX extractor.
Google Slides export as PPTX, then route to the PPTX extractor.
Google Drawings export as PDF, then route to the PDF extractor.
The adapter passes accountId to the export helper so multi-account Google Drive tokens stay scoped to (provider, account, resource).
Exported bytes are not cached; each indexing run performs a fresh export.
Original Workspace metadata remains the indexed metadata. Export MIME types are only used to route extraction, and telemetry records extraction_method = "google_workspace_export".
Export failures remain item-level failures with sanitized taxonomy codes such as google_export_network, google_export_quota_or_permission, google_export_timeout, google_export_unsupported_type, and google_export_invalid_provider.

OneDrive V1 Behavior

Account scope discovery starts at the user's primary OneDrive root only.
Account scope does not crawl SharePoint or shared-with-me containers yet because the current OneDrive service layer still hides those surfaces for MVP visibility consistency.
Folder scope accepts canonical OneDrive resourceId values such as driveId!itemId.
Discovery returns the same canonical metadata shape as Google and keeps OneDrive ID formatting, pagination links, and download stream quirks inside the adapter and service layer.

OneDrive Office/PDF files are eligible only when their downloaded bytes match the explicit deterministic extraction allow-list.

Dropbox V1 Behavior

Account scope discovery starts at the connected account root namespace and keeps namespace/path translation inside the adapter and Dropbox service layer.
Folder scope accepts canonical Dropbox resourceId values returned by the current service layer. Shared runner code must not infer Dropbox paths from display names.
Discovery returns the same canonical metadata shape as Google Drive and OneDrive. service, accountId, resourceId, resourceName, parentResourceId, mimeType, modifiedAt, sizeBytes, contentHash, eligible, skipReason
Dropbox content download for indexing is limited to the same deterministic extraction allow-list. Binary/media/archive formats, unsupported Office formats, and oversized files are skipped explicitly instead of being treated as indexing failures.
Dropbox provider quirks such as namespace headers, business-account rooting, and download endpoint differences stay inside the adapter and user-scoped service wrapper.

Backend Execution Integration

The backend indexing runner resolves adapters only through provider-resolver.ts.
user-scoped-indexing-services.ts is the stable barrel for the provider-specific user-scoped bridge modules.
The provider-specific user-scoped bridge modules keep background indexing tied to persisted account metadata and tokens so indexing execution does not depend on request-time session helpers.
After an adapter is selected, shared execution stays provider-agnostic. discovery -> persisted job items -> download -> extract -> chunk -> embed -> semantic upsert
Re-indexing uses canonical file identity (userId, service, accountId, resourceId) and clears prior vectors for that file before replacement upserts so duplicate active chunks do not survive retries or overwrites.

Integration Guidance

Later runner code should resolve adapters through this boundary, not by importing provider services directly.
Shared indexing code should rely on canonical metadata and content only; it should not know about Google MIME prefixes, export rules, or Shared Drive query details.
New providers should add another IndexingProviderAdapter implementation instead of changing the shared contract.