James BruntonandGitHub
5541dd666c
# Description of Changes
Flesh out the RAG system and connect it to the PDF Question Agent so it
can respond to questions about PDFs of an extremely large size.
I'd expect lots more work will need to be done to finish off the RAG
system to really be what we need, but this should be a reasonable start
which will let us connect it to tools and have the ingestion mostly
handled automatically. I'm leaving file deletion and proper file ID
management to be done in a future PR. We also need to consider whether
all tools should retrieve content exclusively via RAG, or whether it's
beneficial to have tools sometimes fetch the direct content and other
times fetch it from RAG.
A diagram of the expected interaction is as follows:
```mermaid
sequenceDiagram
autonumber
actor U as User
participant FE as Frontend<br/>(ChatPanel)
participant J as Java<br/>(AiWorkflowService)
participant O as Engine:<br/>OrchestratorAgent
participant QA as Engine:<br/>PdfQuestionAgent
participant RAG as Engine:<br/>RagService + SqliteVecStore
participant V as VoyageAI<br/>(embeddings)
participant L as LLM<br/>(Claude / etc.)
U->>FE: types "Summarise this PDF"<br/>(PDF already uploaded)
FE->>J: POST /api/v1/ai/orchestrate/stream<br/>multipart: fileInputs[], userMessage
Note over J: ByteHashFileIdStrategy<br/>id = sha256(bytes)[:16]
J->>O: POST /api/v1/orchestrator<br/>{ files:[{id,name}], userMessage }
O->>L: route via fast model
L-->>O: delegate_pdf_question
O->>QA: PdfQuestionRequest
loop for each file
QA->>RAG: has_collection(file.id)
RAG-->>QA: false
end
QA-->>O: NeedIngestResponse(files_to_ingest)
O-->>J: { outcome:"need_ingest", filesToIngest:[...] }
Note over J: onNeedIngest
loop per file
J->>J: PDFBox: extract page text
J->>O: POST /api/v1/rag/documents<br/>(long-running timeout)
O->>RAG: chunk + stage documents
O->>V: embed_documents (batches of 256)
V-->>O: embeddings
O->>RAG: add_documents
O-->>J: { chunks_indexed: N }
end
Note over J: retry with resumeWith=pdf_question
J->>O: POST /api/v1/orchestrator
Note over O: fast-path to PdfQuestionAgent
O->>QA: PdfQuestionRequest
Note over QA: build RagCapability<br/>pinned to file IDs
QA->>L: run(prompt) with search_knowledge tool
loop up to max_searches
L->>QA: search_knowledge(query)
QA->>V: embed_query
V-->>QA: query vector
QA->>RAG: search(vector, collections=[file.id])
RAG-->>QA: top-k chunks
QA-->>L: formatted chunks
end
Note over QA: once budget spent,<br/>prepare() hides the tool
L-->>QA: PdfQuestionAnswerResponse
QA-->>O: answer
O-->>J: { outcome:"answer", answer, evidence }
J-->>FE: SSE "result"
FE->>U: assistant bubble
```
RAG Integration Guide
Adding RAG to an Agent
from pydantic_ai import Agent
from stirling.services import AppRuntime
class MyAgent:
def __init__(self, runtime: AppRuntime) -> None:
rag = runtime.rag_capability
self.agent = Agent(
model=runtime.smart_model,
system_prompt="Your prompt here...",
instructions=rag.instructions,
toolsets=[rag.toolset],
)
That's it. The agent gets a search_knowledge tool it can call autonomously.
Scoping to Specific Collections
Collections are named buckets of indexed documents — think folders. By default an agent searches everything in the store. Pass collections= to restrict it to only the docs indexed under those names.
from stirling.rag import RagCapability
# Only searches docs indexed under "company-docs" — ignores everything else
scoped = RagCapability(runtime.rag_service, collections=["company-docs"], top_k=3)
# Searches multiple collections
multi = RagCapability(runtime.rag_service, collections=["company-docs", "product-specs"])
# No collections arg = searches all collections in the store
everything = RagCapability(runtime.rag_service)
Config
Non-secret defaults live in the committed engine/.env:
STIRLING_RAG_BACKEND=sqlite # or "pgvector"
STIRLING_RAG_EMBEDDING_MODEL=voyageai:voyage-4
STIRLING_RAG_STORE_PATH=data/rag.db # used when backend=sqlite
STIRLING_RAG_PGVECTOR_DSN= # used when backend=pgvector
STIRLING_RAG_CHUNK_SIZE=512
STIRLING_RAG_CHUNK_OVERLAP=64
STIRLING_RAG_TOP_K=5
Provider credentials (and any local overrides) go in the uncommitted engine/.env.local:
VOYAGE_API_KEY=your-key
Backends
sqlite — Embedded sqlite-vec. Single .db file, zero ops. Ideal for dev and self-hosted deployments.
pgvector — External PostgreSQL with the vector extension. Point STIRLING_RAG_PGVECTOR_DSN at your Postgres instance.
Both backends implement the same VectorStore interface, so agents and the RAG service work identically regardless of which you pick.
For a self-hosted embedding server (e.g. Ollama, TEI, vLLM) set the model string accordingly and point at the server via its native env var:
# Ollama running on another machine
STIRLING_RAG_EMBEDDING_MODEL=ollama:nomic-embed-text
OLLAMA_HOST=http://192.168.1.50:11434
# Any OpenAI-compatible embedding server
STIRLING_RAG_EMBEDDING_MODEL=openai:my-model
OPENAI_BASE_URL=http://192.168.1.50:8080/v1
API Endpoints
| Method | Endpoint | Purpose |
|---|---|---|
| GET | /api/v1/rag/status |
Report embedding model and existing collections |
| POST | /api/v1/rag/index |
Index text into a collection |
| POST | /api/v1/rag/search |
Search a collection |
| GET | /api/v1/rag/collections |
List collections |
| DELETE | /api/v1/rag/collections/{name} |
Delete a collection |