Perfect! Now I have a comprehensive understanding. Let me compile my research findings: Comprehensive Research Report: Letta's RAG/Document Capabilities Based on my analysis of the Letta SDK integration in the roci project, here's everything I discovered about Letta's RAG and document storage/retrieval capabilities: 1. What Letta SDK Supports for Document Storage and Retrieval Letta has a two-tier memory system for documents: A. Archival Memory (Text-based, no files) - Simple text passage storage with semantic search - No file upload/management - Uses embedding-based similarity search - Methods: searchArchival(), addArchival(), listArchival() B. Data Sources + Files (Enterprise RAG system) - File upload capability (PDF, text, markdown, etc.) - Chunking and embedding of file content - Semantic search across file passages - Processing status tracking - File metadata management 2. How to Upload Documents to Letta There are two distinct pathways: Path A: Direct Text to Archival (Simple) // Via roci-memory's LettaClient wrapper await client.agents.passages.create(agentId, { text: "Content to store" // Just text, no files }) Used by: archive_conversation tool (stores summarized conversations) Path B: File Upload via Data Sources (Full RAG) // Via sources API await client.sources.files.upload(fileStream, sourceId, { // Letta handles chunking and embedding }) - Creates source first: client.sources.create({ name, description, embedding }) - Upload files: client.sources.files.upload(file, sourceId) - Files are automatically chunked, embedded, and searchable 3. How to Search/Query Documents Archival Search (for agent's own passages): await client.agents.passages.search(agentId, { query: "search terms", limit: 5 }) // Returns: Array<{text, score}> Source-based Search (enterprise RAG): // List passages in a source await client.sources.passages.list(sourceId, { limit: 10 }) // Get file metadata await client.sources.getFileMetadata(sourceId, fileId) Used in roci: - RecallConversationTool uses searchArchival() to find past conversations - ArchiveConversationTool uses addArchival() to store summaries 4. Supported File Types Based on FileMetadata structure, Letta supports: - Format: Any file with MIME type tracking (fileType: string) - Common types: PDF, Markdown, Text, Word docs, etc. - Processing: Files go through states: pending → parsing → embedding → completed (or error) - Chunking: Automatic with embeddingChunkSize configuration per source - Tracking: Total chunks and chunks embedded tracked separately 5. Embedding Handling Letta manages embeddings completely: EmbeddingConfig Structure: { embeddingEndpointType: "openai" | "anthropic" | "bedrock" | "google_ai" | "google_vertex" | "azure" | "ollama" | "vllm" | etc., embeddingModel: string, // e.g., "text-embedding-3-small" embeddingDim: number, // vector dimension (e.g., 1536) embeddingChunkSize: number, // tokens per chunk (default varies) embeddingEndpoint: string, // API endpoint batchSize: number // processing batch size } Embedding Model Support: - OpenAI (text-embedding-3-small, -large) - Anthropic - Google AI / Vertex - Bedrock - Azure - Local (Ollama, vLLM, LM Studio, etc.) - Groq, Mistral, HuggingFace, Together, Pinecone Per-Source Configuration: Each source has its own embedding config, allowing different models for different data sources. 6. Data Model Overview Passage (stored unit): { id: string, archiveId?: string, // Agent's archival ID sourceId?: string, // Source ID (if from file) fileId?: string, // File ID (if from upload) fileName?: string, // Original filename text: string, // Actual content embedding?: number[], // Vector (if populated) embeddingConfig?: EmbeddingConfig, metadata?: Record, // Custom metadata tags?: string[], // Search tags createdAt: Date, updatedAt: Date } Source (document collection): { id: string, name: string, description?: string, instructions?: string, // How to use this source metadata?: Record, embeddingConfig: EmbeddingConfig, // Required createdAt: Date, updatedAt: Date } FileMetadata (upload tracking): { id: string, sourceId: string, fileName: string, originalFileName: string, fileType: string, // MIME type fileSize: number, // bytes processingStatus: "pending" | "parsing" | "embedding" | "completed" | "error", errorMessage?: string, totalChunks: number, // After chunking chunksEmbedded: number, // Embedding progress content?: string, // Full text (on demand) createdAt: Date, updatedAt: Date } 7. Current roci Integration Pattern In roci-memory (LettaClient wrapper): - searchArchival(query, limit) - semantic search - addArchival(text) - store passage - listArchival(limit) - list recent entries - createBlock(label, value, description) - memory blocks (separate from passages) In roci-agent tools: - archive_conversation - summarizes recent messages with Haiku, stores in archival - recall_conversation - searches archival with semantic search - Both use agent's archival memory (not file-based sources) 8. What roci DOESN'T Use Yet Not yet implemented: - File upload via client.sources.files.upload() - Data sources creation and management - File-based RAG for external documents - Metadata tagging and filtering - Batch file processing - Source-specific search vs. archival search Potential use cases not exploited: 1. Upload user documents (PDFs, notes, etc.) to sources 2. Cross-reference external documents in conversations 3. File-aware RAG ("This is from your Q3_Report.pdf") 4. Organizational knowledge bases 5. Multi-source search with filtering 9. Key Architectural Insights Separation of Concerns: - Archival: For the agent's own memories (conversations, summaries) - Sources: For external/user documents (enterprise RAG) Two API Levels: - client.agents.passages - agent-specific archival - client.sources.passages - source-level passages - client.sources.files - file upload/management Embedding is Centralized: - No embedding management in client code - Letta handles: chunking, embedding, storage, indexing - Search uses embedded vectors automatically Processing is Async: - File upload returns immediately - Processing happens server-side - Status tracked via processingStatus field 10. Integration Opportunities for roci 1. Document Storage Tool: Let agent upload user documents to sources 2. Document Search Tool: Enable RAG queries across uploaded files 3. Memory Augmentation: Archive conversation to both archival AND tag external docs 4. Source Management: Create/manage sources for different topics 5. Knowledge Base: Build searchable reference library from user docs --- Summary Letta provides a full-featured RAG system with: - Automatic chunking and embedding (many models supported) - Semantic search with embedding vectors - File upload and processing tracking - Metadata and tagging support - Scalable source organization - Clean API separation between agent memory (archival) and document stores (sources) Currently, roci uses only the archival memory for storing conversation summaries. The file-based source system is completely unutilized, representing significant untapped RAG potential for incorporating external documents into agent conversations.